cURL : Client for URLs /Client URL Request Library/Curl URL Request Library

cURL is the name of the project. The name is a play on 'Client for URLs', originally with URL spelled in uppercase to make it obvious it deals with URLs. The fact it can also be pronounced 'see URL' also helped, it works as an abbreviation for "Client URL Request Library" or t the recursive version: "Curl URL Request Library". 

curl is a tool to transfer data from or to a server, using one of the supported protocols

 (FTP, FTPS, GOPHER, HTTP, HTTPS, DICT, FILE,  IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP (Real Time Message Protocol), RTSP(Real Time Streaming Protocol), SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.

curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more.

curl is powered by libcurl for all transfer-related features.

Notes to self from an excellent resource: http://www.thegeekstuff.com/2012/04/curl-examples/

1. Download a Single File

The following command will get the content of the URL and display it in the STDOUT (i.e on your terminal).
 

$ curl http://www.centos.org

To store the output in a file, you can redirect it as shown below. This will also display some additional download statistics.

2. Save the cURL Output to a file

We can save the result of the curl command to a file by using -o/-O options.

• -o (lowercase o) the result will be saved in the filename provided in the command line
• -O (uppercase O) the filename in the URL will be taken and it will be used as the filename to store the result

$ curl -o mygettext.html http://www.gnu.org/software/gettext/manual/gettext.html

$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html  

3. Fetch Multiple Files at a time

$ curl -O URL1 -O URL2

$ curl -O http://www.kasani.org/about.html -O www.kasani.org/index.html
 

When we download multiple files from a same sever as shown above, curl will try 

to re-use the connection. 

4. Follow HTTP Location Headers with -L option

By default CURL doesn’t follow the HTTP Location headers

For example, when someone types google.com in the browser from India, it will be automatically redirected to ‘google.co.in’. This is done based on the HTTP Location header as shown below.

We can insists curl to follow the redirection using -L option, as shown below. Now it will download the google.co.in’s html source code.

$ curl -L http://www.google.com 

5. Continue/Resume a Previous Download

Using curl -C option, you can continue a download which was stopped already for some reason. This will be helpful when you download large files, and the download got interrupted.

$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
##############             20.1%

$curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html

$  curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html

I noticed a problem with curl. When a file is already downloaded completely and curl is executed again using -C - then curl says "curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume". I looked up on the web and found that this is cURL Bug

6. Limit the Rate of Data Transfer

You can limit the amount at which the data gets transferred using –limit-rate option. You can specify the maximum transfer rate as argument.

$ curl --limit-rate 1000B -O http://www.gnu.org/software/gettext/manual/gettext.html 

7. Download a file only if it is modified before/after the given time

curl -z -21-Dec-11 http://www.example.com/yy.html 

8. Pass HTTP Authentication in cURL

Sometime, websites will require a username and password to view the content ( can be done with .htaccess file ). With the help of -u option, we can pass those credentials from cURL to the web server as shown below.

$ curl -u username:password URL

 By default curl uses Basic HTTP Authentication. We can specify other 

authentication method using –ntlm | –digest. 

9. Download Files from FTP server

cURL can also be used to download files from FTP servers. If the given FTP path is a directory, by default it will list the files under the specific directory.

$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php

$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/  

10. List/Download using Ranges

cURL supports ranges to be given in the URL. When a range is given, files matching within the range will be downloaded. It will be helpful to download packages from the FTP mirror sites.

$ curl   ftp://ftp.uk.debian.org/debian/pool/main/[a-z]/

The above command will list out all the packages from a-z ranges in the terminal.

11. Upload Files to FTP Server

Curl can also be used to upload files to the FTP server with -T option.
$ curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com

The above command will upload the file named myfile.txt to the FTP 
server. 

You can also upload multiple files at a same time using the 
range operations.

$ curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.testserver.com 

12. More Information using Verbose and Trace Option

You can get to know what is happening using the -v option. -v option enable the verbose mode and it will print the details. This verbose and trace option will come in handy when curl fails due to some reason and we don’t know why.

 curl -v http://google.co.in
* About to connect() to google.co.in port 80 (#0)
*   Trying 74.125.239.120... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: google.co.in
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.co.in/
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 26 Nov 2013 02:43:48 GMT
< Expires: Thu, 26 Dec 2013 02:43:48 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 221
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Alternate-Protocol: 80:quic
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.co.in/">here</A>.
</BODY></HTML>
* Connection #0 to host google.co.in left intact
* Closing connection #0
psqa@psqa-Precision-WorkStation-T3500:~$

13. Get Definition of a Word using DICT Protocol 

Dictionary LookUp via command line

You can use cURL to get the definition for a word with the help of DICT protocol. We need to pass a Dictionary Server URL to it.

 curl dict://dict.org/d:bash
220 pan.alephnull.com dictd 1.12.0/rf on Linux 3.0.0-14-server <auth.mime> <14611418.17764.1385434166@pan.alephnull.com>
250 ok
150 3 definitions retrieved
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, v. t. [imp. & p. p. {Bashed}; p. pr. & vb. n.
   {Bashing}.] [Perh. of imitative origin; or cf. Dan. baske to
   strike, bask a blow, Sw. basa to beat, bas a beating.]
   To strike heavily; to beat; to crush. [Prov. Eng. & Scot.]
   --Hall Caine.
   [1913 Webster]

         Bash her open with a rock.               --Kipling.
   [Webster 1913 Suppl.]
.
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, n.
   1. a forceful blow, especially one that does damage to its
      target.
      [PJC]

   2. a elaborate or lively social gathering or party.
      [PJC]
.
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, v. t. & i. [OE. baschen, baissen. See {Abash}.]
   To abash; to disconcert or be disconcerted or put out of
   countenance. [Obs.]
   [1913 Webster]

         His countenance was bold and bashed not. --Spenser.
   [1913 Webster]
.
250 ok [d/m/c = 3/0/19; 0.000r 0.000u 0.000s]
221 bye [d/m/c = 0/0/0; 0.000r 0.000u 0.000s]

ere are many dictionaries are available. We can list all the dictionaries using

$ curl dict://dict.org/show:db

14. Use Proxy to Download a File

We can specify cURL to use proxy to do the specific operation using -x option. We need to specify the host and port of the proxy.

$ curl -x proxysever.test.com:3128 http://google.co.in

15. Send Mail using SMTP Protocol

cURL can also be used to send mail using the SMTP protocol. You should specify the from-address, to-address, and the mailserver ip-address as shown below.

$ curl --mail-from blah@test.com --mail-rcpt foo@test.com smtp://mailserver.com

Once the above command is entered, it will wait for the user to provide the data to mail. Once you’ve composed your message, type . (period) as the last line, which will send the email immediately.

Subject: Testing
This is a test mail
.