Some Useful Curl Snippets

14th July 2015

Curl is an incredibly useful tool and has all sorts of flags and options available for every situation. I tend to use curl quite a lot for all kinds of stuff, and not just downloading large files. So I thought I would post a few of the most common things that I use the tool for. Note that most of the following URLs don't really exist, they are just for demo purposes. I have also left out the output of these commands as they vary from a few lines to many pages of output.

To run a basic curl request to a website open up a terminal and run the following.

curl http://www.example.com/

This will just split out the content of the page into the command output. You can use quotation marks around the URL, but this generally isn't needed unless you have a character that would break the shell. For example, all URLs the include the & symbols should be enclosed within quotation marks.

curl "http://www.example.com?parameter1=1&meter2=1"

You can run a headers request that will only return the headers from the request using the -I (upper case i) flag.

curl -I http://www.example.com/

To get the headers and the content use the -i (lower case i) flag.

curl -i http://www.example.com/

If you want to curl via basic authentication then you can include the user details with the -u flag.

curl -u username:password http://www.example.com/

To append one or more headers to the request use the -H flag. The following sets the user agent to be 'Mozilla/5.0'.

curl -i -H "User-Agent: Mozilla/5.0" http://www.example.com/

You can supply multiple headers by supplying the -H flag multiple times.

curl -i -H "User-Agent: Mozilla/5.0" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" http://www.example.com/

If you want to start messing about with the HTTP verb you can set this with the -X flag. Every request we have made so far has been a GET request, this can be set manually in the following way.

curl -i -X GET http://www.example.com/

This flag allow you to start testing RESTful services. For example, if you want to send a request to a service to delete an item you would send the request using the DELETE HTTP verb, which would look something like the following.

curl -i -X DELETE "http://www.example.com/service?id=123"

To send a POST request (or a request that contains some data payload) you would use the -X, -H and -d flags. The following example sends a POST request with JSON data, but also tells the web server that it should expect JSON data as the payload.

curl -i -X POST -H "Content-Type: application/json; charset=UTF-8" -d '{"id":"123","name":"bob"}' "http://www.example.com/service?id=123"

With lost of JSON data this can quickly gets messy, but it is possible to put the data in a file and reference it from the curl command. The following example pulls in the POST payload from a file called 'data.json' and sends it with the request.

curl -i -X POST -H "Content-Type: application/json; charset=UTF-8" -d @data.json "http://www.example.com/service?id=123"

This syntax (called @readFile macro) is also a good way of sending files as POST data. This can be done using the -F flag. You should also include the correct header of "Content-Type: multipart/form-data" in this instance.

curl -i -X POST -H "Content-Type: multipart/form-data" -F "[email protected]" http://www.example.com/upload.php

If you want to see just the headers returned by a request you can't use the -I flag in conjunction with the -X flag as they interfere with each other. You can get around this by using the -D, -o and -s flags together. The -D (or --dump-header) flag will dump the headers to a specified location, in this case we are using - which means that we want the headers to be dumped to the command line. The -D flag on it's own will still return the page output so we use the -o flag to dump the output of the page to /dev/null (i.e. just throw it away). Lastly, when using the -o flag to dump the contents of a page to a file curl will output a progress bar, so we include the -s flag to suppress this progress bar.

curl -s -D - -o /dev/null "http://www.example.local/"

You can combine the -s & -D flags together to simplify things slightly.

curl -sD - -o /dev/null https://www.example.com/

Another method of testing headers (and following redirects) is to use a collection of flags together at the same time to spell out -sSLIXGET. The -s flag will make curl silent, the -S flag will show any errors produced (if things fail), the -L flag allows curl to look for and follow redirects. The -I flag tells curl to fetch the header only, but the addition of the -X flag means that we will issue a GET request instead of a HEAD request. I use this quite a lot when testing redirects as it tends to work quite well and is (fairly) easy to remember.

curl -sSLIXGET http://www.hashbangcode.com

If you are using a self signed SSL certificate then curl will reject the request. In order to bypass this you need to supply the --insecure flag.

curl -i --insecure http://www.example.com/

You can also send a request to a server that doesn't have a DNS entry by using the --resolve flag. This is a more recent addition to curl so this flag might not exist on your system. The --resolve flag must contain a string with the following details:

[DNS address]:[port]:[IP address]

So, to redirect a request to example.com to a local IP address you would do the following.

curl -i http://www.example.com/ --resolve 'www.example.com:80:192.168.100.124'

An alternative syntax to using --resolve is to use the raw IP address and send a Host header with the -H flag.

curl -i -H "Host: www.example.local" 192.168.100.124

To capture the output of a curl request you can supply the -o (lower case letter O) flag, supplying the name of the file to create/update.

curl -i http://www.example.com/ -o output.txt

Alternatively, you can use the -O (upper case letter O) flag to use the filename in the URL as the filename of the output file.

curl -i http://www.example.com/robots.txt -O

You can use this flag along with the -u flag to download files from an FTP server.

curl -u username:password -O ftp://www.example.com/file.txt

You will sometimes find that pages have moved, in which case they will issue a 300 header. To get curl to follow these redirections use the -L flag. A good example of this in action is if you try to visit Google. The site will generally redirect you to a local version of the site but it will ensure that the URL has 'www' at the front.

curl -I -L http://google.com/

If you are using curl to download a file and can't guarantee that the file will be downloaded in one go then you can use the -C - flag. This allows curl to continue from where the download left off when it was interrupted.

curl -C - -O "http://www.example.com/bigfile.zip"

When curl is downloading large files you can get curl to produce a progress bar instead of the usual statistics readout with the -# flag.

curl -# -C - -O "http://www.example.com/bigfile.zip"

If you want to see your request in enormous detail you can use the -v flag to get curl to print out everything.

curl -v http://www.example.com/

You can get even more detail than this by using the --trace flag. This flag takes a file as a parameter and drops a trace dump of all incoming and outgoing data into this file. There is a lot of data here but it can help to track down certain issues like encoding errors.

curl --trace trace.txt "http://www.example.com/"

Finally, one useful thing to do can be to issue commands to a PHP application and trigger Xdebug to start the debugger. This can be done by using the -b flag to send a cookie called XDEBUG_SESSION to the server with the value that will trigger your IDE to start the Xdebug process. I use PHPStorm and as such my value is PHPSTORM, but your value might be different here. This method is especially useful when testing POST requests, especially when used over a webservice like SOAP.

curl -i http://www.example.com/ -b XDEBUG_SESSION=PHPSTORM

Is there anything I have missed out here that are useful curl commands? Post your favourite curl snippets below.

Add new comment

The content of this field is kept private and will not be shown publicly.