C++ testing with t-unit

First of all: t-unit is released as open source under the MIT license, and the repository is available at: https://github.com/Lavesson/t-unit. There's also a quick start guide and a wiki available - For some background info, keep on reading.

The need for something simple

So, I wanted a simple unit test environment to test the logic in the 3D engine that I'm writing. What I really wanted was a unit test environment AND some way to output the results in an easy way.

I looked at a couple of different approaches, and most of the frameworks seems to be either, unportable, unmaintained, a hassle to build or simply just a piece of a much larger framework that you have no intention of using.

So my thoughts went to TAP (Test Anything Protocol). If I could just output a series of ok/not ok to a console, that would suffice for now. I hacked together a couple of header files which I could include in my project to make simple assertions and print ok/not ok with some debug info to standard output. I wrote this first implementation in a couple of days. It wasn't perfect, but it worked pretty well. I open-sourced the header files and called them tap-unit.

When I hit a large number of tests (or at least 100+), I realized that I wanted to condense the information in some way. I toyed with the idea of having an HTML report generator, as well as compress the test results into *just* showing the failing tests from a console. At first, I experimented with taking the TAP stream and piping it to a parser which produced either HTML or some nicely formatted output to the console. At this point I basically had my own assertion classes and macros for expanding test cases and fixtures, so TAP wasn't really central anymore, and constructing TAP just to parse it again just added another layer of indirection (without really gaining anything from it).

I rewrote the basic output generator and simply created instances of test cases as I ran them and just wrote a bunch of different report generators, such as TAP, HTML and console. This was much more modular. I changed the name from tap-unit to t-unit.

Introducing t-unit

First of all, t-unit is entirely header-based. There's no implementation files and nothing to build in advance (expect if you want to build the sample apps). You simply include t-unit.h and start writing tests. I wanted to use t-unit on some other projects as well, some of them on Linux, which means that I had to get it to compile on both MSVC and GCC. 

The tests are written and compiled as an executable file which is run manually. Tests are contained in fixtures the same way as many xUnit frameworks do (I happen to like this model of organizing tests). A typical test fixture with setup/teardown might look like:

FIXTURE(account_tests)
{
    account* sut;

    SETUP()
    {
        // Using a raw pointer. We need to clean this up after every test.
        sut = new enterprise_account(100);
    }

    TEST(starts_with_initial_sum)
    {
        assert.are_equal(100, sut->balance(), "expected correct balance at startup");
    }

    TEST(can_withdraw)
    {
        sut->withdraw(25);
        assert.are_equal(75, sut->balance(), "expected correct balance after withdrawing");
    }

    TEARDOWN()
    {
        // delete the account created in setup
        delete sut;
    }
}

Of course, the TEARDOWN macro could be avoided here by using a unique_ptr instead. I'm not going to get into the details here, but instead urge you to read the quick start guide or take a look at the examples in the wiki on Github.

Report generators

In the main method of the executable containing the file, you write some simple boilerplate code to wire everything up. For instance, to create a console report, you would write:

#include <t-unit.h>
#include <console-report.h>

int main()
{
    test_runner::run_all_tests();
    console_report cr;
    test_runner::create_report(cr);
    return 0;
}

You build and execute your project and should (hopefully) get a console report. A simple report example is shown below:

There's more report generators available (TAP and HTML). Again, for more information, consult the wiki. Most notably the generating reports section.

It would be nice to see other people use this, but even if I'm the only one doing so, I will still have gained something - A simple header-based testing utility which I can include in any project and just rebuild it as I write tests (or updates t-unit).

Oh: Bug reports and suggestions on improvements are, of course, welcome :)

C++ ownership semantics

I always get a distinct feeling that smart pointers in C++ are seen as a magical way to handle memory automatically. A lot of discussions often end up in why you should pick smart pointers over raw pointers. Even though I think that it's more of a question in WHICH cases you should do this, I also think it's a secondary point to be made. As I interpret it, the primary point to be made is rather the semantics bound to the smart pointers, and the notion of ownership associated with pointers in general. Choosing the correct type of pointer is simply an effect of this. I'll try to describe what I mean in this post.

When I started programming C++ (around 1998-1999 I think) one of the most annoying things were (and in a sense, still is) memory management. It's also one of the hardest things to get right, even today.

At the time, I usually tried to stick to some hard rules, the most important one being: don't allocate things on the heap if you don't have too. Basically, if the following suits your purpose:

// Stack-allocated auto-variable. No need for manual cleanup
MyType t;

... Then don't do this:

// Heap-allocation - Needs to be deleted at some point
MyType* t = new MyType();

However, sometimes (certainly when working with polymorphic types) we actually need to use pointers and initialize them using new. At that point, it becomes a question of whose responsibility it is to delete the actual pointer when we're done. Because yes - someone must have that responsibility if you want to avoid memory leaks.

The stack is awesome in the sense that stack-allocated variables are easily cleaned up at the end of their defining scope. The usual way to utilize this is through smart pointers. The general idea around smart pointers is easy: Encapsulate a raw pointer inside a stack-allocated type and delete the pointer in the destructor.

There's a bunch of smart pointers out there, both in the Boost C++ library and in the standard library. I'm not going to rant about auto_ptr from C++ 03 here - there's plenty of articles describing the faulty behaviour of auto_ptr out there. It's broken, and we all know it. I'm actually not going to rant at all, but rather point out (no pun intended) some ideas around the semantics of  these pointers. The reason is that I've (more than once) come across the misconception that raw pointers should never ever be used under any circumstances at all. Ever. You should only use smart pointers. I get what people are usually saying when they make that claim, but I think that a standpoint such as that one obscures the idea around ownership. I'm somewhat guilty of giving a half-assed view of this myself in a previous blogpost, where I wrote something along the lines of "Bottom line: avoid raw pointers, use smart pointers". It's not entirely wrong, but it's a lot deeper than that.

Determining ownership

It's not that raw pointers are bad, per se. It's just that they make it hard for the programmer to actually tell who the owner is. To illustrate this point, consider the class interfaces below. Since I'm in the midst of writing a 3D-renderer right now, I'll use some rendereresque class examples:

The Device-class presumably creates a VertexBuffer and returns a pointer to the instance through CreateVertexBuffer. This can then be passed to another class (such as the Mesh-class through the constructor). So, we might use these classes in something resembling the following code snippet:


The problem with raw pointers becomes painfully apparent after writing a bunch of code in this style for a while. For instance, who OWNS the VertexBuffer-pointer? Who is responsible for actually cleaning up after it? Is the lifetime of the buffer-variable bound to the lifetime of the device-variable? In other words, will it be removed in the destructor of the Device class? Or is it up to you, the consumer of this instance to actually delete the pointer? At least let's hope that it's not the Mesh-destructor that removes it. That would be REALLY bad since we could actually have passed the same pointer to several Mesh-instances, leaving every other Mesh with a dangling pointer inside.

If you were to assume that the Device class handles this internally and you never delete the VertexBuffer-pointer, as shown below then one of two things will happen

The two things that can happen are:

1. You were correct in your assumption and the program works as expected.
2. Your guess was wrong - The pointer was never deleted, and now you have a memory leak.

If, instead, you were to assume that the Device class does NOT handle this internally and were to delete it yourself just to be sure as shown below, one of two things could still happen:

The two things in this case are:

1. You were correct in your assumption and the program works as expected.
2. Your guess was wrong - The device class WILL try and delete the pointer. But since you've already done that, it'll try to delete a dangling pointer, and your program would most likely crash.

This is mostly to demonstrate the problem with a non-obvious ownership in C++. And don't get me wrong. This isn't always apparent - Managing memory correctly is HARD. If you just link against the header files and don't have access to the implementation (which you shouldn't need in this case anyway) you have no way of knowing the semantics of any of these classes in terms of how they handle memory.

This is exactly why the semantics of ownership are important - If we can establish ownership, then we at least have a lot of help when managing memory in C++.

Smart pointers

Smart pointers (as described before) help us a lot. There are different semantics tied to each one, determining how/if they are copyable or movable. Since C++ does stack unwinding when an exception is thrown (basically, calls the destructor on stack-allocated objects), it also helps us with cleanup when something goes wrong, just to mention a few perks.

The ones I'm going to focus on are the two widely used unique_ptr and shared_ptr. I tend to use unique_ptr in a lot more places than I do with shared_ptr - I'll try to explain why in a while. A short description of these pointers is in place:

1. unique_ptr encapsulates a raw pointer and can't be copied. Trying to assign one unique_ptr to another will give a compile error. It basically replaces the deprecated auto_ptr, but which much clearer semantics. A pointer of this type CAN be moved though, meaning that it gives up ownership of its inner pointer to another unique_ptr instance. This has to be done explicitly by the developer by using std::move
2. As opposed to unique_ptr, shared_ptr can be copied. It involves somewhat more overhead since it relies on reference counting. Everytime a shared_ptr runs out of scope, the reference counter goes down. Or, in the case of 0 references, the pointer is deleted.

I'm not going to describe the functionality in much more detail than that right now. Other articles and blog posts do that. I'm going to focus on how they help with memory management.

If I know for a fact how each of these pointers actually work, then rewriting the Device class from before to the following gives me a lot more information to work with:

I know, without a doubt, that this class gives me a unique_ptr for a VertexBuffer. I know that I'm now the sole owner of this pointer. It's unique, so there can't be a copy of it inside the Device class. Granted, the Device implementation could store the raw pointer as a private member and delete that one. But that's just an extremely sinister thing to do when returning a unique_ptr, so I'll ignore that.

Basically, the Device type is now telling us that WE are the sole owners of the buffer. It's in our control and it's up to us what will happen from this point forward. Now, it's easy to think that this unique_ptr is only useful in one scope only, since it is apparently unique and can't be copied. But I have to pass a VertexBuffer* to the Mesh implementation, and obviously I can't change the signature to take a unique_ptr and pass the unique_ptr I obtained from Device on - That would be a copy operation.

And no - I'm not going to tell you to use a shared_ptr instead, even though it would work. Since we are then doing reference counting instead, we actually can't claim to have "sole ownership" anymore. A shared_ptr more or less means shared ownership. So here is the thing: I don't really think that leaving the Mesh interface as-is is a bad thing in this case, simply because it is not the owner! Let's look at a snippet of code again, assuming that we're now returning a unique_ptr from Device. I'll also use a unique_ptr to create my Device and Mesh (I'll explain my reasons for this later on)

And yes - I am using raw pointers here. Using std::unique_ptr::get() allows you to get the underlying pointer. And the thing is, I don't really think this is a bad thing to do, granted that the class that you're passing the pointer to is soundly written. I know that when I pass a pointer to a constructor in this way, then it's more of a DI-pattern. It wouldn't make sense for the Mesh-class to delete this resource. Also, notice that the parameter is a VertexBuffer* const. The "const" is important, since it stops that class from modifying the pointer itself (it can still mutate the object being pointed at, but not "re-point" at another object).

Finally - There's still the option to use the shared_ptr and its reference counter. It also has a corresponding weak_ptr that lets you model a temporary ownership (and to break circular references of shared_ptr). As I said before, I usually prefer the unique_ptr, and the reason is simply because it's much clearer in its ownership semantics. Using a shared_ptr by default implies a shared ownership of a pointer, which muddles the boundaries of ownership. When talking to people about shared pointers, I sometimes get the distinct feeling that some programmer's uses them to mimic the way they would write code in a garbage collected language, such as C# or Java. However, just a ref-counter does not a GC make, and we shouldn't suddenly pretend that we can use a shared_ptr and do away with memory issues.

Don't get me wrong - There are times when the shared_ptr is just the thing to use. I just think those times are far apart.

Anyway - This is basically my interpretation of the whole RAII-idiom and the usage of smart pointers. I think smart pointers are a great help in memory management, but not ONLY for the simple reason that they "get cleaned up automatically". I think it all comes down to semantics and understanding how your code is partitioned, and not necessarily on "smart pointers vs. raw pointers", which a lot of debates (but not all of them) seem to focus on.

My take on this might of course be flawed, and I'm always up for discussion. So please, do comment ;)