PHP TestFest UK 2010 And Testing PHP

The UK PHP TestFest this year was held at MadLab in Manchester on the 11th September. I was one of the 12 people who went along to learn about testing the PHP language. So I thought that I would collate some of the things that we went through during the session.

Before creating a test you need to set up your testing environment, you can do this by going to the TestFest site and running through the tutorial on setting up your system for testing PHP. When you have finished setting up your test environment you will have a folder containing three folders, these are php52, php53 and php-trunk, these are the different versions of PHP which you can test, although we will be concentrating on the php53 version in this article the same practices can be used for the other trunks.

The last thing you have to do when setting up the test environment is to set up a global variable called TEST_PHP_EXECUTABLE. This will point to the PHP executable that you want to test and will allow you to run a PHP server and test PHP on the same machine.

export TEST_PHP_EXECUTABLE=php53/sapi/cli/php

You can run PHP tests as you are now, but the best thing to do is to recompile PHP with code coverage, so that everytime we add a test we can see what we are testing and how it is run inside the PHP source. To recompile PHP with code coverage enabled, run the configure command with the enable-gcov flag.

./configure --enable-gcov

Rather than compile and test everything every single time it might be easier to disable everything and enable only the code coverage. This can be done by using the following options.

./configure --disable-all --enable-gcov

Once you have configured your PHP instance you need to make it again. A number of us during the session had trouble getting PHP to compile with the --enable-gcov switch due to the LTP packages missing. If this error occurs we found we could easily fix it by installing the lcov libraries using the command sudo apt-get install lcov. I was using Ubuntu at the time so I was able to find the packages I needed in the Synaptic Packages Manager.

Once you have compiled PHP you can stop. There is no need to run the make install command as we will be working on the PHP executable locally.

Inside each version directory of PHP you will find a file called run-tests.php. This script will allow you to run all of the PHP tests available, You need to run it with the PHP client you just compiled, like this:

sapi/cli/php run-tests.php

This will test the entire code library and produce a code coverage report for the entire codebase. I should warn you that doing this can take quite a while as there are some 7-9,000 plus tests available depending on the version directory you are looking at. When running tests it is best just to run one or a few tests at a time, rather than many hundreds as it will help you make sure that the tests you have written work.

To test just a small subsection of the PHP code you can run the script and pass in a parameter of the directory (or even file) you want to test. For example, to test just tests available in the sql extension you would do this.

sapi/cli/php run-tests.php ext/spl

Ben Longden was kind enough to share a script that uses run-tests.php to test a version directory, but it also then goes on to generate the code coverage report that we wanted to create when we recompiled PHP with gcov enabled.

ROOT='/home/bl/Development/php/php-src/branches/'
BRANCH='PHP_5_2'
BASE_DIR="$ROOT/$BRANCH"
TEST_PHP_EXECUTABLE="$BASE_DIR/sapi/cli/php"
TESTS='ext/spl'

$TEST_PHP_EXECUTABLE $BASE_DIR/run-tests.php $BASE_DIR/$TESTS
lcov -directory $BASE_DIR/$TESTS -c -o tests.info
genhtml tests.info -o coverage
firefox coverage/index.html

You can save this as whatever you want to, but to run this script you'll need something to test first. To do this you need to be able to read a little bit of C code, which is what is run when PHP functions are called. Take a look at the LCOV report on the GCOV site for the DNS functions. You will notice that every line of code is coloured in one of two ways. Blue lines mean that the code has been executed by tests and red lines mean that the line hasn't been executed by tests. Lines that haven't been colour coded will not be run during normal code execution, these are things like comments and statements.

The actual code we run in PHP tests is PHP, so we somehow need to decipher this mass of red and blue lines of code and convert them into PHP functions which we can then test. What you need to look for are PHP function wrappers, these map the C code to the PHP function calls. Take the following example for the PHP function gethostname() from the DNS functions.

Many functions in PHP are changed by giving them parameters, these are parsed by PHP using the C function zend_parse_parameters(). So if you see this function in the source code then you can assume that the function takes at least one parameter. The thrid parameters and everything after this are the important parts of this function in terms of understanding enough to run the PHP code. The third parameter is used by PHP to figure out what sorts of data have been passed to the function. Take the following example in the PHP function dns_check_record().

The third parameter here is "s|s", which means that two strings can be passed to the function. Every time a string is passed to this function two parameters are passed by reference. The first is the actual string itself, and the second is the length of that string. So as the function call above has two strings there are four variables being created.

This isn't the extent of the code you'll have to read, sometimes you will only be able to run a particular section of code after some careful reading of what PHP function is called and under what circumstances. There are, however, plenty of red lines to get started with.

We are now ready to test some PHP code, for this we need to create phpt files and run them using run-tests.php. These files are self contained unit test files, each of which should be used to test a single operation. The file is split up into sections, which are created by adding different headings. The best way to explain the structure of a phpt file is to show an example one. The following file is used to test that the dns_get_record() function returns an array.

--TEST--
dns_get_record() function - basic type return test
--CREDITS--
Philip Norton
#!code www.hashbangcode.com
--FILE--
<?php
  var_dump(is_array(dns_get_record("www.example.com")));
?>
--EXPECT--
bool(true)

For consistency you should create your test files with the following filename template. The type of test would be things like basic, variant, bug or error and will be used to let another tester know what sort of thing the test will consist of. For example, basic tests will only test the function with default parameters, optional parameters for the function would be included in variant tests.

<function_name>_<type>_<number>.phpt

The above example might be saved as dns_get_record_basic_001.phpt as we are not passing any optional parameters to the function call. I won't go into to much detail about PHP test files here, but if you are interested then have a look at the PHPT - Test File Layout page on the PHP QA site.

To run a test you can either use the standard run-tests.php like this:

:$ /php53/sapi/cli/php /php53/run-tests.php /php53/svn/testfest2010/

Or, you can use the script I printed above that will run the test file, generate a code coverage report and open it up in Firefox.

One thing to note is what happens when a test fails. If this happens then the testing script will create a bunch of files. Here is a list of the files generated and what each one has. All of these files will have the same name as the original test file, just with different extension.

• diff - This is a difference file that will show the difference between the expected and actual result of the code run.
• exp - This is the contents of the expected section in your phpt file.
• log - This file contains a log of the code run and why it failed.
• out - This is the actual output of the code run.
• php - This file contains the actual PHP code run during the test, essentially the file section of your test file.

When you run the test again these files will be deleted so there is no need to remove them manually.

On the day we managed to write about 30-40 tests between us before we ran out of time and headed to the nearest pub. Overall it was a great day out and an excellent way to get to grips with the world of PHP testing, with experienced people present. Even though I attended last years PHP TestFest I hadn't actually written any tests, but I think I might actually continue to write tests this time.

On a final note, I would also like to take this opportunity to thank iBuildings (again) for organising the event and even supplying free pizza for lunch.