I've been asked to give an impromptu talk about unit-testing to another team. I looked over our test codebase for inspiration as to what points I should cover...
Distinguish between integration testing and unit testing. It
is key that units can be isolated. Interfaces are the tool to
achieve this. Good unit testing can reduce the need for
integration testing.
Unit testing: automated vs. manual, every build vs.
infrequently (e.g. load-testing, performance-testing,
torture-tests)
Use logging to manually review complex processes (e.g.
pooling, threading, resource acquisition and release,
synchronization)
Even in mature systems, write unit tests for small pieces as
you go in to touch them. Don't be discouraged just because you
can't get full coverage.
Instrument user interfaces to record and playback test cases.
It can then be much simpler to write test cases.
Create simple (unpublished) interactive interfaces to complex
components so that tests can be made up and run on the spot. Such
a facility often suggests new automated test cases. And there is
nothing like human eyes to spot errors that even carefully design
test suites fail to anticipate.
Try to keep tests self-contained. Test database components
against local databases (e.g. Access files, In-memory databases,
etc). It is frustrating when the nightly build fails because a
unit test fails when some externally managed server goes down --
or the data changes.
Use non-deterministic tests to test performance requirements,
or to detect memory leaks, dangling pointers, etc. In the latter
case, running the test suite itself a few hundred times (minus
the leak test, of course) provides an excellent test
environment.
If you must use external databases, try to generate the test
data with each test run.
Use all of your normal programming ingenuity to write
comprehensive test cases. Create supporting test frameworks. Use
O-O techniques to capture commonality between test cases. Use
data structures to, for example, compare two randomly sorted
result sets for equality. Assemble a library of useful test
support routines, within projects and across projects. Use good
algorithms to generate credible test data and to verify the
results. Dispose of temporary resources, especially persistent
ones like temporary files or database tables.
Document test suites so that they read like detailed
specifications.
Use the same test cases against all implementers of an
interface. This may necessitate mandating that implementers must
have associated test code that will produce and consume standard
data sets. The interface test code should first verify that the
implementing component meets this criterion before running the
rest of the tests.
Leave hooks in the real code that test code can exploit.
Strive to keep the hooks private to the component, especially in
components that have security roles.
Use a development environment that allows test cases to be run
piecemeal, preferably avoiding a full compile-link-load-run
cycle, and avoids running the whole test suite (or system) just
to run one test. A tight cycle reduces the temptation to skip
running (or writing) tests.
If your environment does not have a testing framework, write
one. They are not too hard, and you can grow it
incrementally.
Stay focussed in your testing. Just because a component is
usually invoked within a certain environment, does not mean that
a test has to take place there. All you have to do is simulate
the environment (but make sure your simulation is accurate!).
Test cases grow exponentially with component complexity, so keep
components small. Design for isolated testability. Good unit
tests for the containing environment will test the
integration.
Try to write good black box tests, that have complete
coverage, but do not neglect white box tests. Some possible test
scenarios are not identifiable from outside a component, such as
strategy changes.
Make tests repeatable. If generating random data, make sure
that the seed is either hard-coded or at least reported so that a
failure can be reproduced as necessary. If creating simulated
load or concurrency conditions, use positive measures to ensure
repeatability (such as explicitly synchronizing the events in two
threads instead of leaving their coincidence to chance, or
relying on certain performance characteristics [e.g. clock speed]
of the test machine).
Avoid enshrining accidental features of an interface in the
test code. For example, if an interface returns a list of items
with no ordering guarantee, don't rely on the fact that the
current implementation happens to return them in a certain
order.
Use UI robot technologies to test UIs. Don't be afraid to
write code that injects simulated events to your UIs -- in many
frameworks this is not terribly hard to do.
Verify that your tests fail when erroneous behaviour occurs.
If you write your tests before you write the code that implements
the tested functionality, you get this check for free -- until
the implementation occurs. After that, this situation becomes
more difficult to verify. There exist utilities that provide
automation for this task, 'breaking' unit tests by changing
subtle features in the tests like expected literal strings and so
on, but these automated methods only scratch the surface. This is
an outstanding unsolved problem. Occasionally break a test on
purpose, just to make sure things are acting as expected.
Use instrumented implementers of interfaces with 'dispose()'
and 'close()' methods to verify that infrastructures that use
them do, in fact, dispose of the objects exactly once.
Test source code is *real* source code. Keep it in the project
tree. Keep it in the version control system.
Parameterize variable and sensitive test configuration items,
such as database server names, user ids, passwords, etc, and make
it possible to configure these items externally when the test is
run. (e.g. command line arguments, environment variables, INI
files, etc).
There is no need to exhaustively test every capability of a
third-party library -- unless you are publishing all of that
capability as part of your system. More likely, your system only
uses a fraction of the third-party library's functionality. Test
that, either directly, or indirectly by means of the test cases
that verify the dependent component's behaviour. Better still,
wrap the third-party library in a new facade component and test
that. It provides a more controllable dependency, and may even
permit replacement of the third-party library by one from a
different vendor should conditions demand it. The same reasoning
holds for other third-party items such as SQL statements,
database drivers, and so on.
Don't worry about testing functionality in any given order,
e.g. from least complex to most complex, or in reverse dependency
order -- just test!
