SSH VPS

How to Use Wget Command

How to use wget command with examples

This guide will show you how to use the wget command in Linux. We’ll also provide 12 useful wget command examples. By the end, you’ll know all about the wget command and will be able to use it to download files from the World Wide Web.

What is the Wget Command?

Wget is a computer tool created by the GNU Project. You can use it to retrieve content and files from various web servers. The name is a combination of World Wide Web and the word get. It supports downloads via FTP, SFTP, HTTP, and HTTPS.

Wget is created in portable C and usable on any Unix system. It’s also possible to implement on Mac OS X, Microsoft Windows, AmigaOS and other popular platforms.

How to Install Wget?

For this wget command demonstration, we’ll be using Ubuntu 16.04. But the syntax will work on any other Linux distribution too.

Firstly, access your server via SSH. Execute the following command to install wget on your system:

apt-get install wget

Once the setup finishes, you’ll be ready to use it. Also, the knowledge of basic SSH commands can make things easier.

Wget Command Examples

To get you started, we’ll provide 12 wget command examples that you can use for everyday tasks. Keep in mind that you may also call this function from scripts and cron jobs!

Using Wget Command to Download Single Files

One of the most basic wget command examples is downloading a single file and storing it on your current working directory. For example, you may get the latest version of WordPress by using the following:

wget https://wordpress.org/latest.zip

Here is the output that you will see:

--2018-02-23 12:53:10-- https://wordpress.org/latest.zip
Resolving wordpress.org (wordpress.org)... 198.143.164.252
Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9332728 (8,9M) [application/zip]
Saving to: 'latest.zip'

latest.zip 100%[===================>] 8,90M 6,88MB/s in 1,3s

2018-02-23 12:53:14 (6,88 MB/s) - 'latest.zip' saved [9332728/9332728]

In this example, a file named latest.zip will be downloaded in the current working directory. You’ll also see extra information, such the download progress, speed, size, time, and date.

Using Wget Command to Download Multiple Files

We can take wget usage one step further and download multiple files at once. To do that, we will need to create a text document and place the download URLs there. In this example, we will retrieve the latest versions of WordPress, Joomla, and Drupal. Enter the following:

nano example.txt

This will create an example.txt file and open a text editor interface. Paste these links there:

https://wordpress.org/latest.zip

https://downloads.joomla.org/cms/joomla3/3-8-5/Joomla_3-8-5-Stable-Full_Package.zip

https://ftp.drupal.org/files/projects/drupal-8.4.5.zip

Once done, you may use -i to get all the files stored in your example text file:

wget -i example.txt

Wait for the process to finish and you’ll have the installations of three most popular content management systems.

Using Wget Command to Get Files Under Different Names

In this wget example, we will save a file using a different name with the help of -O option:

wget -O wordpress-install.zip https://wordpress.org/latest.zip

In this case, the downloaded resource will be saved as wordpress-install.zip instead of it’s original name.

Looking for a better way to host your website?

Transfer your website to the fastest web hosting platform with 24/7 dedicated support.

Using Wget Command to Save Files in Specified Directory

You can utilize wget to place a file in another directory using -P function:

wget -P documents/archives/ https://wordpress.org/latest.zip

The file you retrieve using this syntax will appear in documents/archives/ folder.

Using Wget Command to Limit Download Speed

With wget, you can also limit the download speed. This is useful when retrieving huge files and will prevent it from using all of your bandwidth. This wget example will set the limit to 500k:

wget --limit-rate=500k https://wordpress.org/latest.zip

Using Wget Command to Set Retry Attempts

Internet connection problems can cause your download to interrupt. To tackle this issue, we can increase the retry attempts using -tries function:

wget -tries=100 https://wordpress.org/latest.zip

Using Wget Command to Download in Background

For extremely large files, you may take advantage of -b function. It will download your content in the background.

wget -b http://example.com/beefy-file.tar.gz

A wget-log will appear in your working directory, which can be used to check your download progress and status. This command will also do the trick:

tail -f wget-log

Using Wget Command to Download via FTP

The command is also usable with FTP. You’ll only need to specify the username and password as in this wget example:

wget --ftp-user=YOUR_USERNAME --ftp-password=YOUR_PASSWORD ftp://example.com/something.tar

Using Wget Command to Continue Interrupted Downloads

Your download can get interrupted if you lose the internet connection or experience a power outage. This is quite a common occurrence when getting huge files. Instead of starting over, it’s possible to continue the download using -c function:

wget -c https://example/very-big-file.zip

If you proceed without the -c function, the new file will have .1 added at the end as it already exists.

Using Wget Command to Retrieve Whole Websites

It is also possible to use wget command to download the content of an entire site. This will let you view it locally without an internet connection. Here is an example:

wget --mirror --convert-links --page-requisites --no-parent -P documents/websites/ https://some-website.com

Lets analyze the ingredients of this wget command:

–mirrorIt makes your download recursive.
–convert-linksAll links will be converted for proper offline usage.
–page-requisitesThe following will include all necessary files such as CSS, JS, and images.
–no-parentIt ensures that directories above the hierarchy are not retrieved.
-P documents/websites/This ensures that all content goes to our specified directory.

Once the process finishes, you’ll be able to open the downloaded website locally and find all the files in documents/websites/ folder.

Using Wget Command to Locate Broken Links

Lets try something more advanced. We can use the wget command to locate all broken URLs that display 404 error on a specific website. Start by executing the following:

wget -o wget-log -r -l 5 --spider http://example.com
-oGathers output into a file for later use.
-lSpecifies the recursion level.
-rMakes the download recursive.
–spiderSets wget to spider mode.

We may now investigate the wget-log file to find the list of broken links. Here’s the command to do it:

grep -B 2 '404' wget-log | grep "http" | cut -d " " -f 4 | sort -u

Using Wget Command to Download Numbered Files

If you have files or images numbered in a certain list, you may easily download all of them with the following syntax:

wget http://example.com/images/{1..50}.jpg

Conclusion

Congratulations! By completing this tutorial you have learned various uses for the wget command. You can now utilize it to get a single or multiple files. In addition, you’ve learned some advanced uses, such as downloading a whole website or locating broken URLs. For more information you may also check the official documentation.

Are there any hidden tips or tricks that you want to share? Feel free to do so in the comments below!

1 Comment

Click here to post a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share this