Assertion Injection

Summary and Introduction

Assertion Injection (AI) is a design pattern for automated tests. It can enable precise, pinpoint reporting of test failure causes, and makes the use of more complex and fragile mocks unnecessary in some cases. Generally, AI will not make new tests possible. Its value is that it makes certain powerful tests much easier and simpler to express in code.

Assertion injection works in languages featuring functions as first-class objects and support for closures. It will also require an xUnit-like testing library, or something sufficiently similar.

This article's code examples use Python 3 and its built-in "unittest" library. The concepts can more or less translate to other high-level languages.

Basic Use

Consider this class:

Listing 1

from urllib.request import urlopen class RemoteResource: 'Represents some sort of remotely-fetched resource' server = 'example.com' def __init__(self, rel_url): 'rel_url is the HTTP URL of the resource, relative to the server' url = 'http://{}/{}'.format(self.server, rel_url) self.resource = urlopen(url)

At object creation, the constructor triggers an HTTP GET to a URL. The exact URL is calculated, based on the server attribute, and the rel_url argument.

Now, think how this would be tested. You might want to verify the correct URL value is indeed being constructed, and that it works right even if a subclass changes the value of server.

On top of that, you might prefer to test this without taking the trouble of setting up a test web server that responds to each test URL. (And forever maintaining it!)

There are of course ways to deal with all this. One is to structure the class differently, so that each part can be tested in isolation. Here is one way:

Listing 2

from urllib.request import urlopen class RemoteResource2: 'Represents some sort of remotely-fetched resource' server = 'example.com' def __init__(self, rel_url): 'rel_url is the HTTP URL of the resource, relative to the server' url = self.mk_url(rel_url) self.resource = self.load_resource(url) def mk_url(self, rel_url): return 'http://{}/{}'.format(self.server, rel_url) def load_resource(self, url): return urlopen(url)

Then, in a test case, you could instantiate a subclass of RemoteResource2 that overrides load_resource with a no-op, and test mk_url directly. This kind of thing is the bread and butter of unit testing in languages like Java and C#, and certainly does the job fine.

While this works, it's not ideal. There are now two new methods that exist soley to make a very simple and basic test possible. The added complexity – both in extra lines of code, and added development time – is arguably not worth it.

Assertion injection helps tremendously in this kind of situation. A code change for dependency injection is still required. But it is simpler and much less invasive than what we did in Listing 2. In fact, the change is only one line: the constructor's argument signature, which now takes an optional argument specifying the URL opening function.

Listing 3

from urllib.request import urlopen class RemoteResource: 'Represents some sort of remotely-fetched resource' server = 'example.com' def __init__(self, rel_url, urlopen=urlopen): '''rel_url is the HTTP URL of the resource, relative to the server urlopen is the function to open the URL''' url = 'http://{}/{}'.format(self.server, rel_url) self.resource = urlopen(url)

A test case can now be simply written as so:

Listing 4

import unittest from rr import RemoteResource class TestUrl(unittest.TestCase): def test_build_url(self): def myurlopen(url): self.assertEqual('http://example.com/testpath.json', url) rres = RemoteResource('testpath.json', myurlopen)

What happens if the test fails? Suppose we neglect to put in the slash between domain and relative URL. Perhaps with a line like this in the constructor:

Listing 5

# format string should be 'http://{}/{}' url = 'http://{}{}'.format(self.server, rel_url)

Then the test will fail with the following stack trace:

Listing 6

amax@crispy:~/wdir/writing/articles/dai/code python3.1 test_rr.py F ====================================================================== FAIL: test_build_url (__main__.TestUrl) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_rr.py", line 7, in test_build_url rres = RemoteResource('testpath.json', myurlopen) File "/home/amax/wdir/writing/articles/dai/code/rr.py", line 8, in __init__ self.resource = urlopen(url) File "test_rr.py", line 6, in myurlopen self.assertEqual('http://example.com/testpath.json', url) AssertionError: 'http://example.com/testpath.json' != 'http://example.comtestpath.json' ---------------------------------------------------------------------- Ran 1 test in 0.001s FAILED (failures=1)

What does this immediately tell us?

The assertion on line 6 of test_rr.py is what failed
The computation that triggered the failed assertion is on line 8 of rr.py
The exact failure condition - in this case, showing the expected URL string, contrasted to the actual value created

This is an outstanding set of information! It is likely to help us quickly and immediately understand exactly what went wrong, and how to fix it.

How It Works

In its simplest form, assertion injection relies on two properties of the language: support for closures, and functions as first-class objects (that can be dynamically created and passed as arguments, just like any data object).

A closure is an encapsulation of an execution context - the identifier table, including local object references, that defines all identifier names at a particular time and place in the program's execution. This allows code in other contexts to reference the local scope in a controlled way.

This is directly exploited by the myurlopen function above: its execution context includes the TestCase instance, whose reference is available to the injected function. As a result, the assertion can hook directly into the test running and reporting framework.

The AI Recipe

Here's a summary of how to use this pattern:

Structure the SUT (system under test - the code being tested, basically) to allow the dependency injection. It needs to enable the test, obviously, but otherwise can be as minimally invasive as possible.
In the test case, dynamically create a function (or class, or object instance) implementing the appropriate interface, that can be injected in the SUT. In the function, embed test case assertions using the standard test library framework.
Inject, and exercise the SUT to trigger the assertions.

Mocks

I'm not aware of any test situation that AI could handle but a good mock object library could not. The question is, then, why bother? Why not just use the mocking library?

There are certainly times when mocks are greatly useful. I'm grateful to the authors of mock-object libraries. If you are one, thank you for your work - the world's software is significantly better because of your efforts.

This is not diminished by the fact that using test mocks, and other forms of fake objects, carries undesireable baggage.

Typically, a mock library is an external library, and so introduces an external dependency on running tests. This creates another hurdle for people who would use or modify your code. This isn't helpful when you want people to contribute to your open source project, but also matters in a more proprietary context.
The mock library introduces an external dependency that can change on you, and which you don't necessarily have much control over.
Mocking libraries tend to work by using the more "magic" features of the language. Thus, they can conflict with advanced or subtle language features used by your own code. I've had this problem in at least two languages; the mocking libraries got confused by the the code they were testing.
I have often found that setup of the mocks must be tied to sensitive details of of the SUT's implementation, and thus can easily become fragile. (AI suffers this as well, though to a lessor degree, I believe.)
The interface of the mock object library is another thing you have to learn, and another thing you have to remember. These libraries are generally not hard to use, but they aren't trivially and completely intuitive either. Most of us have plenty of APIs to memorize already.

Given all this, I will use mocks when called for. And when there is a way to create a quality test that does not use mocks, I'll choose that option. Assertion injection sometimes provides that choice.