Curl is an incredibly useful tool and has all sorts of flags and options available for every situation. I tend to use curl quite a lot for all kinds of stuff, and not just downloading large files. So I thought I would post a few of the most common things that I use the tool for. Note that most of the following URLs don't really exist, they are just for demo purposes. I have also left out the output of these commands as they vary from a few lines to many pages of output.
To run a basic curl request to a website open up a terminal and run the following.
curl http://www.example.com/
This will just split out the content of the page into the command output. You can use quotation marks around the URL, but this generally isn't needed unless you have a character that would break the shell. For example, all URLs the include the & symbols should be enclosed within quotation marks.
curl "http://www.example.com?parameter1=1&meter2=1"
You can run a headers request that will only return the headers from the request using the -I (upper case i) flag.
curl -I http://www.example.com/
To get the headers and the content use the -i (lower case i) flag.
curl -i http://www.example.com/
If you want to curl via basic authentication then you can include the user details with the -u flag.
curl -u username:password http://www.example.com/
To append one or more headers to the request use the -H flag. The following sets the user agent to be 'Mozilla/5.0'.
curl -i -H "User-Agent: Mozilla/5.0" http://www.example.com/
You can supply multiple headers by supplying the -H flag multiple times.
curl -i -H "User-Agent: Mozilla/5.0" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" http://www.example.com/
If you want to start messing about with the HTTP verb you can set this with the -X flag. Every request we have made so far has been a GET request, this can be set manually in the following way.
curl -i -X GET http://www.example.com/
This flag allow you to start testing RESTful services. For example, if you want to send a request to a service to delete an item you would send the request using the DELETE HTTP verb, which would look something like the following.
curl -i -X DELETE "http://www.example.com/service?id=123"
To send a POST request (or a request that contains some data payload) you would use the -X, -H and -d flags. The following example sends a POST request with JSON data, but also tells the web server that it should expect JSON data as the payload.
curl -i -X POST -H "Content-Type: application/json; charset=UTF-8" -d '{"id":"123","name":"bob"}' "http://www.example.com/service?id=123"
With lost of JSON data this can quickly gets messy, but it is possible to put the data in a file and reference it from the curl command. The following example pulls in the POST payload from a file called 'data.json' and sends it with the request.
curl -i -X POST -H "Content-Type: application/json; charset=UTF-8" -d @data.json "http://www.example.com/service?id=123"
This syntax (called @readFile macro) is also a good way of sending files as POST data. This can be done using the -F flag. You should also include the correct header of "Content-Type: multipart/form-data" in this instance.
curl -i -X POST -H "Content-Type: multipart/form-data" -F "[email protected]" http://www.example.com/upload.php
If you want to see just the headers returned by a request you can't use the -I flag in conjunction with the -X flag as they interfere with each other. You can get around this by using the -D, -o and -s flags together. The -D (or --dump-header) flag will dump the headers to a specified location, in this case we are using - which means that we want the headers to be dumped to the command line. The -D flag on it's own will still return the page output so we use the -o flag to dump the output of the page to /dev/null (i.e. just throw it away). Lastly, when using the -o flag to dump the contents of a page to a file curl will output a progress bar, so we include the -s flag to suppress this progress bar.
curl -s -D - -o /dev/null "http://www.example.local/"
You can combine the -s & -D flags together to simplify things slightly.
curl -sD - -o /dev/null https://www.example.com/
Another method of testing headers (and following redirects) is to use a collection of flags together at the same time to spell out -sSLIXGET. The -s flag will make curl silent, the -S flag will show any errors produced (if things fail), the -L flag allows curl to look for and follow redirects. The -I flag tells curl to fetch the header only, but the addition of the -X flag means that we will issue a GET request instead of a HEAD request. I use this quite a lot when testing redirects as it tends to work quite well and is (fairly) easy to remember.
curl -sSLIXGET http://www.hashbangcode.com
If you are using a self signed SSL certificate then curl will reject the request. In order to bypass this you need to supply the --insecure flag.
curl -i --insecure http://www.example.com/
You can also send a request to a server that doesn't have a DNS entry by using the --resolve flag. This is a more recent addition to curl so this flag might not exist on your system. The --resolve flag must contain a string with the following details:
[DNS address]:[port]:[IP address]
So, to redirect a request to example.com to a local IP address you would do the following.
curl -i http://www.example.com/ --resolve 'www.example.com:80:192.168.100.124'
An alternative syntax to using --resolve is to use the raw IP address and send a Host header with the -H flag.
curl -i -H "Host: www.example.local" 192.168.100.124
To capture the output of a curl request you can supply the -o (lower case letter O) flag, supplying the name of the file to create/update.
curl -i http://www.example.com/ -o output.txt
Alternatively, you can use the -O (upper case letter O) flag to use the filename in the URL as the filename of the output file.
curl -i http://www.example.com/robots.txt -O
You can use this flag along with the -u flag to download files from an FTP server.
curl -u username:password -O ftp://www.example.com/file.txt
You will sometimes find that pages have moved, in which case they will issue a 300 header. To get curl to follow these redirections use the -L flag. A good example of this in action is if you try to visit Google. The site will generally redirect you to a local version of the site but it will ensure that the URL has 'www' at the front.
curl -I -L http://google.com/
If you are using curl to download a file and can't guarantee that the file will be downloaded in one go then you can use the -C - flag. This allows curl to continue from where the download left off when it was interrupted.
curl -C - -O "http://www.example.com/bigfile.zip"
When curl is downloading large files you can get curl to produce a progress bar instead of the usual statistics readout with the -# flag.
curl -# -C - -O "http://www.example.com/bigfile.zip"
If you want to see your request in enormous detail you can use the -v flag to get curl to print out everything.
curl -v http://www.example.com/
You can get even more detail than this by using the --trace flag. This flag takes a file as a parameter and drops a trace dump of all incoming and outgoing data into this file. There is a lot of data here but it can help to track down certain issues like encoding errors.
curl --trace trace.txt "http://www.example.com/"
Finally, one useful thing to do can be to issue commands to a PHP application and trigger Xdebug to start the debugger. This can be done by using the -b flag to send a cookie called XDEBUG_SESSION to the server with the value that will trigger your IDE to start the Xdebug process. I use PHPStorm and as such my value is PHPSTORM, but your value might be different here. This method is especially useful when testing POST requests, especially when used over a webservice like SOAP.
curl -i http://www.example.com/ -b XDEBUG_SESSION=PHPSTORM
Is there anything I have missed out here that are useful curl commands? Post your favourite curl snippets below.
Add new comment