Virtual Surreality

Should it be assertEquals(expected, actual) or assertEquals(actual, expected)?

This discussion came about because at the Chicago Code Camp, Jim Suchy from 8th light was demonstrating TDD in JavaScript by building a simple unit testing library. One of the methods he created was the typical assertEquals. In the code on github he had a signature of assertEquals(expected, actual) while during the demo he swapped the order, and I pointed out that the original was what I would have anticipated. He commented that there was an interesting discussion about it at his office a few days before. I’m interested to hear more about it, and I tweeted that I was inclined to get a debate going about it just to see what the different rationales were. This post is a collation of thoughts and opinions to date.

For those of us using xUnit unit testing frameworks (e.g. JUnit, NUnit, HUnit), we’re used to seeing expected first. Those from the newer TestNG school have actual first.

If you’re a Scala developer you may be used to seeing actual should equal (expected) or expect (expected) { actual }, or the myriad of varied approaches supported by wrapping other testing frameworks.

You might receive a test error message like “Expected expected but got actual” or “actual should be expected” depending on the framework.

Sometimes the issue is dodged altogether. Scala developers also know assert(left === right) so one receives the useful but less informative “left does not equal right” test error.

The awesome Hamcrest (an anagram of “matchers”) matching library (now partially included in JUnit) takes a more fluent approach with assertThat(left, equalTo(right)) which has the same slight lack of information. Looking at Joe Walnes‘ docs I feel he favours expected as the first parameter. However, as Pat Kua and Nat Pryce point out, most matchers are awkward if the actual value isn’t first assertThat(size, greaterThan(9)) is easier than assertThat(10, lessThanOrEqualTo(size)) however I personally would put assertThat(size, atLeast(10)) in such a case (more on why below).

Furthermore, Hamcrest has powerful matchers that allow you to express things like assertPresenceOf(exactly(1), link().with(urlParameter("name", containingString("Joe")))) in which the expected values come later in the expression as it follows (English) natural language idioms.

The “given/when/then” style of Behaviour-Driven Development also puts the expected results at the end. This is due not only to the lexical structure but also because it “feels right” to declare pre-conditions before the events and post-conditions after the events, and the wording from specifications tend to end with the expected values.

    Given "my savings account balance is", 100       #pre-condition
    Given "my cash account balance is", 10           #pre-condition
    When "I transfer", 20                            #event
    Then "my savings account balance should be", 80  #post-condition
    Then "my cash account balance should be", 30     #post-condition

Why the difference? Which is “better”? Should we care?

We should care for a couple of reasons:

1. Consistency is important. At least within the one codebase only one convention should be used.
2. The order of things tends to subtly enforce a mental model about what’s important.
3. Software development is hard enough without developers (new or experienced) being tripped up by such things.
4. For those of use that work in lots of different codebases, various testing frameworks, and even various languages, having to recall the correct order is an annoyance we can do without.

During a mailing discussion with many of the people mentioned in this post, Nat Pryce gave the following example to show how certain APIs, predicates and their order in the expression can prove to be unwieldy or otherwise, comparing assertThat(actualValue, greaterThan(10)) with assertThat(10, lessThanOrEqualTo(actualValue)).

This example actually helped prove my second point about the mental model and the physiological psychology involved. Apart from the fact that the two statements aren’t equivalent after all (as they deal with different equivalence partitions), it also shows that because the constant being used is the boundary value it needs to be used explicitly and in a way that is obvious. My response to Nat was that what should have been written is:

assertThat(actual, atLeast(10));   // or notLessThan(10) or greaterThanOrEqualTo(10)

because 10 is the boundary value, not 9 or 11. Even though in this particular case integer arithmetic makes it irrelevant, our brains and those of the developers reading it late don’t make it irrelevant. Expressing it in other ways leaves us open to subtle errors either in writing or comprehending the code.

Nat Pryce and Jonathan Wolter point out that languages with named arguments or those with loose typing and anonymous classes allow the developer to choose the order. For example, in JavaScript:

var x = someCodeBeingTested();
assertEqual({actual: x, expected: 20});
assertEqual({expected:20, actual: x});

Darren Hobbs made a suggestion for statically-typed languages using the type system to help make the order explicit (or varied). Wrap the values being evaluated in an Expected or Actual instance. With some tweaks using static factory methods from me and Ian Robinson we arrive at something like this in Java:

assertEquals(Expected.expected(10), Actual.actual(testSubject.methodUnderTest())

Where the factory methods have signatures like:

public static Expected expected(Object o);

public static Actual actual(Object o);

and using different signatures to allow the reverse order so that the assertEquals/assertThat methods look something like:

public static boolean assertEquals(Expected expectedValue, Actual actualValue);
public static boolean assertEquals(Actual actualValue, Expected expectedValue);

So with static imports for brevity the resulting assertion might look like this:

assertEquals(expected(10), actual(testSubject.methodUnderTest());

If only now we could remove the parentheses, commas and dots! Some unambiguous alias methods might also help with readability.

As to which is “better”, I think the consensus from those I’ve asked so far seems to be expected first when no other impetus comes from the API or fluent interface. Mind you, I haven’t asked the TestNG team members yet and when I get to them it might also help answer why the difference from their point of view too.

Some summary opinions I’ve collected so far …

In an otherwise unstructured API where parameter order is the only differentiator, “expected” is the known, and usually shorter, value and thus should be expressed first. It emphasizes test-first practices because the developer will tend to write down what is expected first, then the expression that returns the actual value.

Fluent interfaces and DSLs allow this to be easily changed around because they “read” a certain way that is familiar to the human developer. What’s important is that what is written is intended by the writer and obvious to the reader.
— JoshG (that’s me)

I will say this: I’ve been typing them in the wrong order since I started using JUnit way back when… I’ve well-conditioned people who see me screw up the order, they *know* I’m short-bus-special to start with.
— Michael D. Hill

I try to have the shorter arg first: assertEquals(5, expressionToTest()). assertThat gives you lots of options.
— Kent Beck

What about assertThat(actual, is(expected))? I think #hamcrest is nice syntactic sugar.
— Neal Ford

I prefer the more English-like “[actual] should be [expected]“.
— Carlos Villela

If you’re writing a new framework, I would focus more on providing meaningful errors. I like the better descriptions of hamcrest over junit defaults, and particularly mockito’s exceptions that provide tips and suggestions about what you do.

If you’re using the framework, I personally prefer the “expected” just because that’s often easier though I’ve noticed most people new to any testing framework naturally put “actual” first.

I think it would be interesting to do a usability study into this. Just get some people who’ve never done automated testing (there’s a lot of them) to try to write some. Watch if they put actual or expected first.
— Patrick Kua

For an existing framework whichever produces the most meaningful and
readable failure message which is going to depend on the framework.
I’d just hate to run into code that swapped actual and expect in the
*error message* because some dev decided to do it back to front in the
code.

For a brand new framework I’d favor expected first actual second
because personally I think that is closer to how we’d say it out loud
or write it down.
— Ian Cartwright

Expected, Actual feels more natural for me, probably because I’m used to xUnit frameworks. I do my best to expect literals, so like you, I think it makes sense to put the shorter and more obvious value first.

Does it help with readability? It depends on what’s being tested. (foo, 123.45) seems as readable as (123.45, foo), but I’d probably favor (expected, actual) when actual is some kind of built value
- e.g. assertEquals(123, aNew.trade.withCurrentBid(122).withCurrentAsk(124).withLastTrade(123).withLean(1).bid)
— Jay Fields

I prefer the syntax from hamcrest/junit4 here assertThat(expected, eq(actual))
I find this to be more explicit and wouldn’t think to reverse the parameters.
— Nate Austin