Virtual Surreality

Most of my ThoughtWorks colleagues love tests. We love writing tests and watching them fail, and we love writing the code that then makes them pass. We love tests so much that most of the time taken in building the software is in the running of the tests to make sure we haven’t done something dotty to break the software.

We hate when the time taken to run all those tests takes more than a few minutes, and this causes a lot of discussion internally as well as on our blogs. My strategic preference is to make the application smaller and thus the builds quicker, by building the software as discrete components (built and deployed independently) that are composed into an application. But before that, there’s a long list of things to make the builds and tests quicker. The list is never long enough, though.

We regularly discuss/concoct innovations in testing at the Sydney TW office. A few weeks ago Stacy Curl started a discussion on an idea he’s had for a while around optimising acceptance tests. During this discussion, we talked about various techniques, including profiling and coverage, which then prompted thoughts around knowing which code each test covered.

This then blossomed into a requirement which was to instrument the code that is covered by each test such that when that code is changed, the tests that cover it are known. This then lets us run only the tests that “need” to be run when we change some code. I say “need” because there are always things below or outside the code that may alter behaviour, so a full test suite still has to be run at some point, just not in a way that delays feedback on the most likely problems.

In essence, we wanted a tool that defines the smoke tests (pre-commit build, initial CruiseControl pipeline build) dynamically. The “changed” code could be based on file system modified times, Subversion status, CruiseControl modificationset, etc.

Unfortunately, the context provided by tools like Clover, Emma and Cobertura isn’t sufficient to indicate which test is running (no stack is available in the metrics). The upshot was I went off and looked at using AspectJ aspects to trace the code during a test run. Gianny Damour helped with early optimisations of my advices and gave me the introduction to “cflow” which allowed me to remove my own stack-peeking logic. I was going to call it TestThis! (as in this.method()) but because of the annoyance in lowercase called it TestMe instead (no throwback to VB intended), with the almost obligatory “J” prefix for Java. Josh Price rightly pointed out that the resultant name “JTestMe” is lame – another reason to merge with ProTest (see below).

Only the source file name is tracked. While we could instrument to the method level, we felt that this was too granular and that any change to the source file should trigger all tests that touch any of that class’ logic. It also works for superclasses, inner classes and called classes – that is, the tests may not call the code directly, but because the code is within their control flow, they’re all tracked.

public aspect TestCoveringAClassObserver {
    pointcut allTestMethods(TestCase test):
        target(test) &&                                                   // capture the TestCase
        execution(public void TestCase+.test*())                          // and advise execution of any JUnit test methods
    ;

    before(TestCase test):
        cflow(allTestMethods(test)) &&                                    // within the control flow of the allTestMethods advice
        (execution(!private * *(*)) || execution(!private * *())) &&      // when executing any non-private method
        (execution(* !TestCase+.*()) || execution(* !TestCase+.*(*))) &&  // that is not in the test case class
        within (!com.thoughtworks.testme..*)                              // and not within any JTestMe class
        {
            recordLink(thisJoinPointStaticPart.getSourceLocation(), test);
        }
    ;
}

Even with the extra guards to avoid recursive advice explosions and to ignore instrumenting the test code itself, you can see that the core piece of code is a ridiculously few lines of AspectJ. The “database” updated by the recordLink() method is a fat ugly (but very fast) object de/serializer that I haven’t bothered cleaning up much.

To allow you to easily run your code with or without instrumentation (you may be nervous that AspectJ has subtly altered behaviour), I’ve used the AspectJ Load Time Weaving (from the AspectWorkz merge). It also means you can continue to use your own custom JUnit test runners. Instrumented – use aj5’s java --javaagent). Normal – use java.

At CITCONF in Sydney a few weeks back, where I gave this a quick demo, the crowd were very encouraging and the Atlassian Clover (nee-Cenqua) guys, who demonstrated a tops new version, were academically interested in the neat and tidy use of aspects (they can’t use them for a reason I can’t remember?? Will find out next meetup).

There’s a long to-do list, the first of which are make it more “one-click” to use and explore a merge into ProTest (a test prioritising tool – and I figure the tests that JTestMe identifies are just another way to prioritise tests). Others are SVN integration and Ant task, and a filter for the excellent Continuous Testing tool (which also allows for prioritising tests).

It’s just been accepted into the Codehaus so have a peek and let me know. The usage is clunky (change the JTESTME_HOME values and the aj5 java agent path and test cases in the Windows cmd files) but it works for HelloWorld demonstration purposes (no automatic file modification detection yet).