C++ Test Framework

I had a couple of questions about my test framework, so I thought I'd come clean and show just how quickly you can get off the ground with a couple of macros in C++

With all of the code in Pioneer, my goal has been to do the minimum required to get the job done and my test framework is no exception. I rolled my own test framework as an exercise in understanding what a test framework does, is and should do. Needlessly reinventing the wheel is core to the development philosophy of pioneer and a great way to learn about systems.

I decided that I want to be able to tag tests at the end of the source file for the class they test, and that the syntax should be simple. I also wanted them to be easy to compile out so I knew macros were likely to be involved.

/////////////////////////////////
#include <unittest.h>
START_TEST( ExampleTest )
... test code ...
END_TEST


My minimum test for a class is a smoke test that instantiates an object and confirms that its non null. This is a sanity test for the constructor and checks for crashes, assertions, exceptions, errors etc.  I prefer to only use tested dependencies - test friendlys - and any other test on the object would require an instantiated object

/////////////////////////////////
#include <unittest.h>
START_TEST( World_SmokeTest )

{
WorldPtr world( new World() );
CheckNonNULL( world.get() );
}
CheckTrue( true );

END_TEST

This smoke test makes two checks, the first is that a world is instantiated without error and the second is that the object can be destroyed without error. I've used a scoped pointer, so this test just needs to report two successful checks and I'm happy. The default behaviour for my test log is to report the number of passed tests, the number of successful checks and a verbose list of all of the failures.

Within each test I can call on any of these Checks, which have so far been enough.
CheckNonNULL( a ) 
CheckNULL( a )
CheckEqual( a, b )
CheckNotEqual( a, b )
CheckTrue( a )
CheckFalse( a )

The START_TEST macro declares a class with the name of the test. It also registers it with a static instance of the test suite.  The test suite is the only global static object I use, and is compiled out in production code. Its the worst way I could configure the test runner except all of the other options.

With my START_TEST macro, the test case is registered and becomes part of the global test suite, which makes adding new tests super easy. There is no excuse for not adding a test after writing (most) public methods and its easy to write the test before the method too.

However, a few improvements could still be made. As the START_TEST macro declares a class with the test name, all test names have to be globally unique. This hasn't been a problem because I prefix the test name with the name of the class being tested but its a weakness in the system nonetheless.

I'd like to be able to declare tests per class. For example the test declaration might be:

START_TEST( World, SmokeTest )
... test code ...
END_TEST

So now I could report the classname of the system under test in the test report along with the test name, and potentially use it for test-coverage metrics as I could count the number of tested classes, tests per class and potentially add instrumentation for untested classes so I had a good idea of coverage.
I have an idea that a test-energy graph could be overlaid on a class graph so I could visualise test coverage as a heat map.

Secondly I might want a test suite to contain subsets of classes. Input/GUI tests, gameplay tests, network code tests, etc... This could be grouped with an ADD_TEST_TO_SUITE macro or with an extra parameter in START_TEST for the suite name.  I don't have a reason to run a subset of tests though - I really like having them all run every time and they are so fast that there is no cost to doing this. As soon as I fall into the practice of running a subset I might suffer from slow-to-execute test, or from breaking a test that I'm not running.
I've not got a good reason to add test suites, yet, but I've got a niggling feeling its a good idea.