Allowing Cached HTTPS Traffic From Drupal With Varnish And Pound

29th January 2015

Varnish is a web application accelerator that helps to speed up websites. It works by sitting in front of the web server and caching web pages that are served by it.

When a request for a web page is made Varnish passes this request on to the web server, which then responds to the request as it normally would. Varnish then caches the result of this request before sending it to the user. This means that the next time the page is requested the cached version of the page is sent to the user and the web server isn't actually part of the process. Because Varnish doesn't run any website code or access any databases the response is therefore much faster than the original response from the web server. In fact, simply having Varnish in place can provide large speed improvements on sites without much configuration. Varnish is exceptionally good at this job and can speed up sites significantly.

One thing that Varnish doesn't cater for is HTTPS traffic. This is by design and as such if you have a Varnish setup you will need to bypass it entirely if you want to use encrypted HTTPS traffic.

One way around this is to use a reverse proxy system called Pound. Pound sits in front of Varnish on the HTTPS port and decrypts all HTTPS requests before sending them to Varnish. All normal HTTP traffic is ignored by Pound and goes through Varnish and is treated as it normally is.

I'm not going to go through installing and setting up Varnish here as that is a lengthy operation in itself. If you are interested then I have already written a description of the process of installing and configuring Varnish, which also includes a description of configuring Drupal.

To install Pound on a Debian based system just run the following.

sudo apt-get install pound

RedHat based systems are pretty similar, but in this case they use yum.

sudo yum install pound

Out of the box, Pound will not start on system boot, to set this edit the setting in the file /etc/default/pound.

startup=1

Before editing the main Pound configuration we need to generate a certificate file that Pound understands so that it can decrypt our HTTPS transmissions and forward them onto Varnish. If you already have a trusted SSL certificate you may still need to complete this step in order to generate a certificate that Pound is happy with. For testing purposes we are going to use the existing test SSL certificate that comes with Ubuntu to demonstrate how to do this (kept at /etc/ssl/certs/ssl-cert-snakeoil.pem). To generate a certificate file for Pound you need to combine the pem and key files together like this.

  1. mkdir keys
  2. cp /etc/ssl/certs/ssl-cert-snakeoil.pem keys
  3. sudo openssl rsa -in /etc/ssl/private/ssl-cert-snakeoil.key >> keys/ssl-cert-snakeoil.pem

The main Pound configuration file is found at /etc/pound/pound.cfg, which needs to be edited to do two things. As a default Pound will listen to port 80, which is already being listened to by our Varnish server and as such needs to be removed. We then need to add a rule make Pound listen to HTTPS traffic and tell it what to do with the requests.

Your Pound configuration file should look like the following.

  1. User "www-data"
  2. Group "www-data"
  3. RootJail "/var/pound"
  4. LogLevel 1
  5.  
  6. ## check backend every X secs:
  7. Alive 30
  8.  
  9. # poundctl control socket
  10. Control "/var/run/pound/poundctl.socket"
  11.  
  12. ListenHTTPS
  13. Address 10.0.0.1 # put your server's public IP address here
  14. Port 443
  15. Cert "/path/to/ssl-cert-snakeoil.pem"
  16. HeadRemove "X-Forwarded-Proto"
  17. AddHeader "X-Forwarded-Proto: https"
  18. Service
  19. BackEnd
  20. Address 127.0.0.1
  21. Port 80
  22. End
  23. End
  24. End

The above options set some default settings and then allow Pound to listen to incoming HTTPS requests. We tell Pound to decrypt these requests and then forward them to port 80 as unencrypted traffic. The X-Forwarded-Proto header is set to 'https' so that Varnish and the web application know that it should be dealing with HTTPS traffic. The 'Cert' parameter references the pem file that we generated above, but should be changed to your trusted certificate.

Also note that we aren't adding a 'ListenHTTP' directive here as we don't want Pound to listen to HTTP traffic at all.

If you have already setup your web server to listen to HTTPS traffic then you'll need to turn this off before starting Pound. In Apache this is normally kept in the form of a Listen directive. You can either remove this or just change the port by altering the following configuration directive.

  1. <ifmodule mod_ssl.c="">
  2. Listen 44300
  3. </ifmodule>

Finally, we just need to add a vcl_hash callback to our default.vcl file. This will allow us to differentiate between HTTP and HTTPS traffic when trying to cache pages.

  1. sub vcl_hash {
  2. hash_data(req.url);
  3. if (req.http.host) {
  4. hash_data(req.http.host);
  5. } else {
  6. hash_data(server.ip);
  7. }
  8. # Use special internal SSL hash for https content
  9. # X-Forwarded-Proto is set to https by Pound
  10. if (req.http.X-Forwarded-Proto ~ "https") {
  11. hash_data(req.http.X-Forwarded-Proto);
  12. }
  13. return (hash);
  14. }

With all that in place you should be able to restart Apache, Varnish and Pound to get everything talking together.

sudo service apache2 restart && sudo service varnish restart && sudo service pound restart

You can double check what services are listening to what ports by running 'netstat -nlp', which should produce something like the following. This assumes that the backend port that Varnish is sending traffic to is 8080, which is what Apache is responding on.

  1. $ sudo netstat -nlp
  2. Active Internet connections (only servers)
  3. Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
  4. .....
  5. tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 11605/varnishd
  6. tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 11554/apache2
  7. .....
  8. tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 11647/pound
  9. ......

If you are having trouble starting Pound then take a look at the Pound log, which will be kept in /var/log/syslog on Debian systems.

sudo tail /var/log/syslog

Here it will tell you about any port conflicts or incorrect SSL certificate files so you should be able to figure out what is wrong and correct it.

There are one or two security considerations to bear in mind here. If you are unencrypting traffic and then sending this traffic over a network to a different server then there is a small chance of that traffic being intercepted. For this reason it is essential that Pound is only used on a private network or on the same machine as the Varnish instance.

Finally, those of you who are following along at home probably realise the elephant in the room here. Although both HTTP and HTTPS anonymous traffic will be cached, authenticated traffic will not be. So if all of your users login to view your site then using Pound to send traffic through Varnish will not reduce the amount of resources you need. The main problem is that you don't really want to serve cached pages to users who have logged in as there will be a chance that they will see the cache that another user has generated. There are a couple of things you can do to allow caching involving setting custom headers and cookies and even ESI (edge side includes).

As a bonus I thought I would create an Ansible playbook that will install Pound alongside an existing Varnish on Ubuntu setup. The two templates used in this playbook (pound_config.j2 and pound_initscript.j2) can be found in the article above.

  1. ---
  2. - hosts: all
  3. tasks:
  4. - name: install Pound
  5. apt: pkg=pound state=installed
  6. sudo: true
  7.  
  8. - name: setup Pound initscript
  9. template: src=pound_initscript.j2 dest=/etc/default/pound
  10. sudo: true
  11.  
  12. - name: create snakeoil pem certificate directory
  13. file: path=/home/{{ user }}/keys state=directory
  14.  
  15. - name: copy snakeoil certificate pem file
  16. command: cp /etc/ssl/certs/ssl-cert-snakeoil.pem /home/{{ user }}/keys creates=/home/{{ user }}/keys/ssl-cert-snakeoil.pem
  17.  
  18. - name: generate pem file with key file
  19. shell: openssl rsa -in /etc/ssl/private/ssl-cert-snakeoil.key >> ssl-cert-snakeoil.pem chdir=/home/{{ user }}/keys creates=/home/{{ user }}/keys/ssl-cert-snakeoil-pound.pem
  20. sudo: true
  21.  
  22. - name: add Pound config file
  23. template: src=pound_config.j2 dest=/etc/pound/pound.cfg
  24. sudo: true
  25. notify:
  26. - restart pound
  27.  
  28. - name: ensure pound is started
  29. service: name=pound state=started
  30. sudo: true
  31.  
  32. - name: get iptables rules
  33. shell: iptables -L
  34. register: iptablesrules
  35. sudo: true
  36. always_run: true
  37. changed_when: False
  38.  
  39. - name: add apache iptable rule
  40. command: iptables -I INPUT 1 -p tcp --dport 443 -j ACCEPT -m comment --comment "pound https port"
  41. sudo: true
  42. when: iptablesrules.stdout.find("pound") == -1
  43. notify:
  44. - save iptables
  45.  
  46. handlers:
  47. - name: restart pound
  48. service: name=pound state=restarted
  49. sudo: true
  50.  
  51. - name: save iptables
  52. command: iptables-save
  53. sudo: true
  54. notify:
  55. - restart ufw
  56.  
  57. - name: restart ufw
  58. service: name=ufw state=restarted
  59. sudo: true

Comments

Permalink
thx

Anonymous (Tue, 03/03/2015 - 18:25)

Permalink
It helped a lot for my website. you are my hero

Ryan (Sun, 05/31/2015 - 16:32)

Permalink
If you are using Cloudways platform, then you won't need to manually setup varnish cache on your Drupal website. Their platform is already using Varnish in the stack along with Memcached, MySQL, Apache and Nginx. Also, using their platform, you can enable or disable SSL on Drupal website by just clicking a button.

Azaz Qadir (Wed, 07/13/2016 - 13:09)

Add new comment

The content of this field is kept private and will not be shown publicly.