AnuTechVista: Wget

Notes to self from the excellent source: http://www.thegeekstuff.com/2009/09/the-ultimate-wget-download-guide-with-15-awesome-examples/

wget utility is the best option to download files from internet.
wget can pretty much handle all complex download situations including
large file downloads, recursive downloads, non-interactive downloads, multiple file downloads etc.,

1. Download Single File with wget

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

--2013-11-25 19:22:10-- http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Resolving www.openss7.org (www.openss7.org)... 142.59.210.7
Connecting to www.openss7.org (www.openss7.org)|142.59.210.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3852374 (3.7M) [application/x-bzip2]
Saving to: `strx25-0.9.2.1.tar.bz2'

14% [=============> ] 574,462 60.7K/s eta 47s

While downloading it will show a progress bar with the following information:

%age of download completion (for e.g. 31% as shown below)
Total amount of bytes downloaded so far (for e.g. 1,213,592 bytes as shown below)
Current download speed (for e.g. 68.2K/s as shown below)
Remaining time to download (for e.g. eta 34 seconds as shown below)

2. Download and Store With a Different File name Using wget -O

By default wget will pick the filename from the last word after last forward slash, which may not be appropriate always.

Wrong: Following example will download and store the file with name: download_script.php?src_id=7701

$ wget http://www.vim.org/scripts/download_script.php?src_id=7701

--2013-11-25 19:25:32-- http://www.vim.org/scripts/download_script.php?src_id=7701
Resolving www.vim.org (www.vim.org)... 216.34.181.97
Connecting to www.vim.org (www.vim.org)|216.34.181.97|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50243 (49K) [application/octetstream]
Saving to: `download_script.php?src_id=7701'

100%[===================================================================================================>] 50,243      --.-K/s   in 0.06s

2013-11-25 19:25:32 (836 KB/s) - `download_script.php?src_id=7701' saved [50243/50243]

psqa@psqa-Precision-WorkStation-T3500:~$ ls
about.html       Documents                        examples.desktop Music           Public                    Templates
centos-org.html Downloads                        gettext.html      mygettext.html strx25-0.9.2.1.tar.bz2    Videos
Desktop          download_script.php?src_id=7701 index.html        Pictures        strx25-0.9.2.1.tar.bz2.1

Correct: To correct this issue, we can specify the output file name using the -O option as:

wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

--2013-11-25 19:27:12-- http://www.vim.org/scripts/download_script.php?src_id=7701
Resolving www.vim.org (www.vim.org)... 216.34.181.97
Connecting to www.vim.org (www.vim.org)|216.34.181.97|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50243 (49K) [application/octetstream]
Saving to: `taglist.zip'

100%[===================================================================================================>] 50,243      --.-K/s   in 0.06s

2013-11-25 19:27:12 (837 KB/s) - `taglist.zip' saved [50243/50243]

3. Specify Download Speed / Download Rate Using wget –limit-rate

While executing the wget, by default it will try to occupy full possible bandwidth. This might not be acceptable when you are downloading huge files on production servers. So, to avoid that we can limit the download speed using the –limit-rate as shown below.
In the following example, the download speed is limited to 200k

wget --limit-rate=200k http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

--2013-11-25 19:31:33-- http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Resolving www.openss7.org (www.openss7.org)... 142.59.210.7
Connecting to www.openss7.org (www.openss7.org)|142.59.210.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3852374 (3.7M) [application/x-bzip2]
Saving to: `strx25-0.9.2.1.tar.bz2.2'

100%[===================================================================================================>] 3,852,374 57.8K/s in 45s

2013-11-25 19:32:18 (84.4 KB/s) - `strx25-0.9.2.1.tar.bz2.2' saved [3852374/3852374]

psqa@psqa-Precision-WorkStation-T3500:~$

4. Continue the Incomplete Download Using wget -c

This is very helpful when you have initiated a very big file download which got interrupted in the middle. Instead of starting the whole download again, you can start the download from where it got interrupted using option -c

Note: If a download is stopped in middle, when you restart the download again without the option -c, wget will append .1 to the filename automatically as a file with the previous name already exist. If a file with .1 already exist, it will download the file with .2 at the end.

$ wget -c http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

5. Download in the Background Using wget -b

For a huge download, put the download in background using wget option -b as shown below.

$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

Continuing in background, pid 2883.
Output will be written to `wget-log'.

psqa@psqa-Precision-WorkStation-T3500:~$ tail -f wget-log
3450K .......... .......... .......... .......... .......... 93% 102K 3s
3500K .......... .......... .......... .......... .......... 94% 102K 2s
3550K .......... .......... .......... .......... .......... 95% 98.8K 2s
3600K .......... .......... .......... .......... .......... 97% 102K 1s
3650K .......... .......... .......... .......... .......... 98% 102K 1s
3700K .......... .......... .......... .......... .......... 99% 98.6K 0s
3750K .......... .. 100% 110K=42s

2013-11-25 19:38:45 (89.9 KB/s) - `strx25-0.9.2.1.tar.bz2.4' saved [3852374/3852374]

6. Mask User Agent and Display wget like Browser Using wget –user-agent

Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.

$ wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" www.google.com

--2013-11-25 19:43:29-- http://www.google.com/
Resolving www.google.com (www.google.com)... 74.125.28.105, 74.125.28.99, 74.125.28.147, ...
Connecting to www.google.com (www.google.com)|74.125.28.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.1'

[ <=> ] 56,271 --.-K/s in 0.05s

2013-11-25 19:43:29 (1.03 MB/s) - `index.html.1' saved [56271]

7. Test Download URL Using wget –spider

When you are going to do scheduled download, you should check whether download will happen fine or not at scheduled time. To do so, copy the line exactly from the schedule, and then add –spider option to check.

$ wget --spider www.google.com

Spider mode enabled. Check if remote file exists.
--2013-11-25 19:46:19-- http://www.google.com/
Resolving www.google.com (www.google.com)... 74.125.28.104, 74.125.28.99, 74.125.28.105, ...
Connecting to www.google.com (www.google.com)|74.125.28.104|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

You can use the spider option under following scenarios:

Check before scheduling a download.
Monitoring whether a website is available or not at certain intervals.
Check a list of pages from your bookmark, and find out which pages are still exists.

8. Increase Total Number of Retry Attempts Using wget –tries

If the internet connection has problem, and if the download file is large there is a chance of failures in the download. By default wget retries 20 times to make the download successful.
If needed, you can increase retry attempts using –tries option as shown below.

$ wget --tries=75 DOWNLOAD-URL

9. Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

$ cat > download-file-list.txt
URL1
URL2
URL3
URL4

Next, give the download-file-list.txt as argument to wget using -i option as shown below.

$ wget -i download-file-list.txt

10. Download a Full Website Using wget –mirror

Following is the command line which you want to execute when you want to download a full website and made available for local viewing.

$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

–mirror : turn on options suitable for mirroring.
-p : download all files that are necessary to properly display a given HTML page.
–convert-links : after the download, convert the links in document for local viewing.
-P ./LOCAL-DIR : save all the files and directories to the specified directory.

11. Reject Certain File Types while Downloading Using wget –reject

You have found a website which is useful, but don’t want to download the images you can specify the following.

$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED

12. Log messages to a log file instead of stderr Using wget -o

When you wanted the log to be redirected to a log file instead of the terminal.

$ wget -o download.log DOWNLOAD-URL

13. Quit Downloading When it Exceeds Certain Size Using wget -Q

When you want to stop download when it crosses 5 MB you can use the following wget command line.

$ wget -Q5m -i FILE-WHICH-HAS-URLS

Note: This quota will not get effect when you do a download a single URL. That is irrespective of the quota size everything will get downloaded when you specify a single file. This quota is applicable only for recursive downloads.

14. Download Only Certain File Types Using wget -r -A

You can use this under following situations:

Download all images from a website
Download all videos from a website
Download all PDF files from a website

$ wget -r -A.pdf http://url-to-webpage-with-pdfs/

15. FTP Download With wget

You can use wget to perform FTP download as shown below.
Anonymous FTP download using Wget

$ wget ftp-url

FTP download using wget with username and password authentication.

$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL

AnuTechVista

Monday, November 25, 2013

Wget