Automating Headless Selenium PHPUnit Tests

I have talked before about running Selenium tests in PHPUnit but I have only recently come to properly automate things. Getting a Selenium server to start and stop in a script is relatively easy and can be done in a simple script. My original script for running a directory of PHPUnit tests was as follows. I will explain more about how this all works later on in this post.

# start selenium server
java -jar selenium-server-standalone-2.24.1.jar -firefoxProfileTemplate SeleniumFirefoxProfile &
# run unit tests
phpunit Tests/
# stop selenium server
wget -O - -q -t 1 http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer

The only problem with this method was that is would plaster my screen with Firefox windows whilst running. This became a big problem when I had over 10 tests (which happened quite quickly) and took over my machine when it needed to run. So I sat down and looked at how to run Selenium tests without showing the Firefox on screen, or what is called 'headless' mode. What I needed to do was use a server called xvfb (X virtual framebuffer) which is an x11 server that performs all graphical operations in memory, rather than send them to a display. x11, more commonly referred to as X, is the graphical user interface engine used by Linux that displays all of the windows and other graphics. By attaching the Selenium server to the xvfb server all of the Firefox windows will be opened up there and not shown to screen. This is also the way in which things should be done on a server that has no graphical output. I'm going to assume that you already have all of the things needed to run Selenium tests in PHPUnit, but to install xvfb on an Ubuntu system use the following.

apt-get install xvfb

The next step is to start the xvfb server. The following command creates a virtual screen with a size of 1024x768 (with a depth of 24) on port 99, saving the frame memory to /tmp and with no access control. The -ac flag is important for headless mode as it means that we don't have to worry about setting up permissions for this virtual screen.

xvfb :99 -screen 0 1024x768x24 -fbdir /tmp -ac

The trouble I had here was that once started it wasn't easy to stop it again without using ps and kill or pressing Ctrl+x. To this end I had a bit of a look around on the internet and put together the following script for starting and stopping a xvfb server. I have seen a few variations of this particular script, but I couldn't find out for sure who did the original.

#!/bin/bash

XVFB=/usr/bin/Xvfb
XVFBARGS=":99 -screen 0 1024x768x24 -fbdir /tmp -ac"
PIDFILE=/tmp/xvfb.pid
case "$1" in
  start)
    echo -n "Starting virtual X frame buffer: xvfb"
    start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS
    echo "."
    ;;
  stop)
    echo -n "Stopping virtual X frame buffer: xvfb"
    start-stop-daemon --stop --quiet --pidfile $PIDFILE
    echo "."
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  *)
        echo "Usage: xvfb {start|stop|restart}"
        exit 1
esac
 
exit 0

I saved this as xvfb_control.sh, made sure it was executable (via the command chmod +x xvfb_control.sh), and ran the xvfb server using the following.

./xvfb_control.sh start

The next thing to do was make sure that the Selenium server attached to this xvfb server when it ran, rather than to the primary display (i.e. my monitor). To do this you just need to export a system property called DISPLAY, which contains the location of the virtual screen.

export DISPLAY=:99

After this we can start the Selenium server in much the same way. This command starts the server in the background and logs any errors to the file /tmp/selenium_server.log. This is a slightly more robust way of doing things as it allows me to see any errors that Selenium has raised.

java -jar selenium-server-standalone-2.24.1.jar -firefoxProfileTemplate SeleniumFirefoxProfile > /tmp/selenium_server.log 2>&1 &

Once done we can run the PHPUnit tests in the normal way, after which we need to stop the xfvb and Selenium servers as they aren't needed any more. To shut down the xvfb server we just run the script from before, but this time issue a stop command.

./xvfb_control.sh stop

Selenium can be stopped simply by visiting a webpage, so we get wget to do that for us silently.

wget -O - -q -t 1 http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer

I collated this into a single script file to that all of the Selenium unit tests in a directory were run all at once.

#!/bin/bash

./xvfb_control.sh start
export DISPLAY=:99
java -jar selenium-server-standalone-2.24.1.jar -firefoxProfileTemplate SeleniumFirefoxProfile > /tmp/selenium_server.log 2>&1 &
echo "waiting for Selenium server to finish loading"
sleep 3
echo "running tests"
phpunit Tests/
echo "shutting down Selenium server"
wget -O - -q -t 1 http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer
./xvfb_control.sh stop

This worked ok, but I found that some of the tests were failing with a weird error. Here is the output from one of the test failures.

1) SeleniumHashbangcodeTest::testMyTestCase

Invalid response while accessing the Selenium Server at 'http://localhost:4444/selenium-server/driver/': Timed out after 30000ms

/Tests/SeleniumHashbangcodeTest.php:14
/Tests/SeleniumHashbangcodeTest.php:14

Caused by
RuntimeException: Invalid response while accessing the Selenium Server at 'http://localhost:4444/selenium-server/driver/': Timed out after 30000ms

/Tests/SeleniumHashbangcodeTest.php:14
/Tests/SeleniumHashbangcodeTest.php:14

I tracked this down to what I think was the Selenium server taking too long to start up Firefox before the PHPUnit test started to run. I noticed that when I started Firefox normally with the SeleniumFirefoxProfile it did a compatibility check on all of the extensions before loading. This was pointless, especially in headless mode as I would never see any updates if they were found, but it seemed to be contributing to PHPUnit timing out before the test was run.

In order to stop Firefox automatically checking for updates to plugins every time it loads you'll need to do the following:

1. Open up FireFox and select the correct profile
2. Go to about:config
3. Right click anywhere and select New > Boolean to create a new value.
4. Enter extensions.checkCompatibility into the window that appears and click ok.
5. Select False from the list of available options and click ok.

You can optionally do the same for the value extensions.checkUpdateSecurity if that is also causing issues. With the extensions.checkCompatibility turned off my PHPUnit Selenium tests then ran without fail.