cURL is the name of the project. The name is a play on 'Client for URLs', originally with URL spelled in uppercase to make it obvious it deals with URLs. The fact it can also be pronounced 'see URL' also helped, it works as an abbreviation for "Client URL Request Library" or t the recursive version: "Curl URL Request Library".
curl is a tool to transfer data from or to a server, using one of the supported protocols
(FTP, FTPS, GOPHER, HTTP, HTTPS, DICT, FILE, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP (Real Time Message Protocol), RTSP(Real Time Streaming Protocol), SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.
curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more.
curl is powered by libcurl for all transfer-related features.
1. Download a Single File
The following command will get the content of the URL and display it in the STDOUT (i.e on your terminal).
$ curl http://www.centos.org
To store the output in a file, you can redirect it as shown below. This will also display some additional download statistics.
2. Save the cURL Output to a file
We can save the result of the curl command to a file by using -o/-O options.
- -o (lowercase o) the result will be saved in the filename provided in the command line
- -O (uppercase O) the filename in the URL will be taken and it will be used as the filename to store the result
$ curl -o mygettext.html http://www.gnu.org/software/gettext/manual/gettext.html
$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
3. Fetch Multiple Files at a time
$ curl -O URL1 -O URL2
$ curl -O http://www.kasani.org/about.html -O www.kasani.org/index.html
When we download multiple files from a same sever as shown above, curl will try
to re-use the connection.
4. Follow HTTP Location Headers with -L option
By default CURL doesn’t follow the HTTP Location headers
For example, when someone types google.com in the browser from India, it
will be automatically redirected to ‘google.co.in’. This is done based
on the HTTP Location header as shown below.
We can insists curl to follow the redirection using -L option, as
shown below. Now it will download the google.co.in’s html source code.
$ curl -L http://www.google.com
5. Continue/Resume a Previous Download
Using curl -C option, you can continue a download which was stopped
already for some reason. This will be helpful when you download large
files, and the download got interrupted.
$ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
############## 20.1%
$curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html
$ curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html
I noticed a problem with curl. When a file is already downloaded completely and
curl is executed again using -C - then curl says "curl: (33) HTTP server doesn't
seem to support byte ranges. Cannot resume". I looked up on the web and found that this is cURL Bug
6. Limit the Rate of Data Transfer
You can limit the amount at which the data gets transferred using
–limit-rate option. You can specify the maximum transfer rate as
argument.
$ curl --limit-rate 1000B -O http://www.gnu.org/software/gettext/manual/gettext.html
7. Download a file only if it is modified before/after the given time
curl -z -21-Dec-11 http://www.example.com/yy.html
8. Pass HTTP Authentication in cURL
Sometime, websites will require a username and password to view the
content ( can be done with .htaccess file ). With the help of -u option,
we can pass those credentials from cURL to the web server as shown
below.
$ curl -u username:password URL
By default curl uses Basic HTTP Authentication. We can specify other
authentication method using –ntlm | –digest.
9. Download Files from FTP server
cURL can also be used to download files from FTP servers. If the
given FTP path is a directory, by default it will list the files under
the specific directory.
$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php
$ curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/
10. List/Download using Ranges
cURL supports ranges to be given in the URL. When a range is given,
files matching within the range will be downloaded. It will be helpful
to download packages from the FTP mirror sites.
$ curl ftp://ftp.uk.debian.org/debian/pool/main/[a-z]/
The above command will list out all the packages from a-z ranges in the terminal.
11. Upload Files to FTP Server
Curl can also be used to upload files to the FTP server with -T option.
$ curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com
The above command will upload the file named myfile.txt to the FTP
server.
You can also upload multiple files at a same time using the
range operations.
$ curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.testserver.com
12. More Information using Verbose and Trace Option
You can get to know what is happening using the -v option. -v option enable the verbose mode and it will print the details. This verbose and trace option will come in handy when curl fails due to some reason and we don’t know why.
curl -v http://google.co.in
* About to connect() to google.co.in port 80 (#0)
* Trying 74.125.239.120... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: google.co.in
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.co.in/
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 26 Nov 2013 02:43:48 GMT
< Expires: Thu, 26 Dec 2013 02:43:48 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 221
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Alternate-Protocol: 80:quic
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.co.in/">here</A>.
</BODY></HTML>
* Connection #0 to host google.co.in left intact
* Closing connection #0
psqa@psqa-Precision-WorkStation-T3500:~$
13. Get Definition of a Word using DICT Protocol
Dictionary LookUp via command line
You can use cURL to get the definition for a word with the help of DICT protocol. We need to pass a Dictionary Server URL to it.
curl dict://dict.org/d:bash
220 pan.alephnull.com dictd 1.12.0/rf on Linux 3.0.0-14-server <auth.mime> <14611418.17764.1385434166@pan.alephnull.com>
250 ok
150 3 definitions retrieved
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, v. t. [imp. & p. p. {Bashed}; p. pr. & vb. n.
{Bashing}.] [Perh. of imitative origin; or cf. Dan. baske to
strike, bask a blow, Sw. basa to beat, bas a beating.]
To strike heavily; to beat; to crush. [Prov. Eng. & Scot.]
--Hall Caine.
[1913 Webster]
Bash her open with a rock. --Kipling.
[Webster 1913 Suppl.]
.
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, n.
1. a forceful blow, especially one that does damage to its
target.
[PJC]
2. a elaborate or lively social gathering or party.
[PJC]
.
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, v. t. & i. [OE. baschen, baissen. See {Abash}.]
To abash; to disconcert or be disconcerted or put out of
countenance. [Obs.]
[1913 Webster]
His countenance was bold and bashed not. --Spenser.
[1913 Webster]
.
250 ok [d/m/c = 3/0/19; 0.000r 0.000u 0.000s]
221 bye [d/m/c = 0/0/0; 0.000r 0.000u 0.000s]
ere are many dictionaries are available. We can list all the dictionaries using
$ curl dict://dict.org/show:db
14. Use Proxy to Download a File
We can specify cURL to use proxy to do the specific operation using
-x option. We need to specify the host and port of the proxy.
$ curl -x proxysever.test.com:3128 http://google.co.in
15. Send Mail using SMTP Protocol
cURL can also be used to send mail using the SMTP protocol. You
should specify the from-address, to-address, and the mailserver
ip-address as shown below.
$ curl --mail-from blah@test.com --mail-rcpt foo@test.com smtp://mailserver.com
Once the above command is entered, it will wait for the user to
provide the data to mail. Once you’ve composed your message, type .
(period) as the last line, which will send the email immediately.
Subject: Testing
This is a test mail
.
