Zen's Blog

In a previous post (not true, not written yet), I outlined some of the properties that an automated test must ascribe to in order for it to be considered “well structured”. In this post I’ll dig into the concept of independent tests a bit further.

I have found that even though most developers by now know that tests should be written so that they are independent of each other, I’m not sure if they have realized the full implication of that statement. Let me explain what I mean.

For an automated test to be considered independent, it should not only code-wise not rely on state created in previous test cases, hence allowing the test code to be moved or refactored without affecting other tests; it should also be practically possible to execute any test case in any order –without rewriting the tests.

The implication of this is that all test suite creation must be completely separate to how we construct and write our tests. It is not okay to be forced to execute all test cases in a file. It is not okay to be forced to include all files in a directory in a given test suite.

The consequence of this is that the test runner must be able to take a suite-description that allows for execution of individual test cases. Even if the test case is stored in a file with a bunch of other test cases, and the file is stored in a directory with a bunch of other test files.

Further, the development environment itself must be flexible enough for easy temporary modification, such as a command line argument.

$> make run-tests #runs all test cases
$> make run-tests TEST=”monkey_can_peel_banana, fish_cannot_peel_banana” #runs specific test cases

Why is this important?

I, as a user of a test framework, would go bananas if I was forced to run all test cases every time. Yes, we strive for quick running tests, but quality cannot be achieved with unit tests alone. We also need higher level tests such as integration tests, component tests, system tests, feature acceptance tests, stress-tests…and they can take time.

Secondly, if we cannot execute test cases in different contexts, we are bound to copy paste the tests or not run them at all. For example, test case TC-1045 may be part of both customer A’s acceptance test suite, as well as customer B’s. We don’t want to be forced to copy the test code just because the framework doesn’t allow for the same test case be run in two different suites.

This has implications on how we structure our tests. Test fixtures become more important, since the execution context may have wider variations. It also puts greater strain on the test framework itself. The proper interpretation of setup/teardown, before/after suite and other such event-hooks becomes very important.

In summary, independent does not only mean code independent, but also practical test execution independence.