Sunday, 9 January 2011

BDD: Concrete Examples Aren't Enough

The Given/When/Then style of Behaviour Driven Development (BDD), favoured by Cucumber and JBehave, puts a lot of context in the examples that I claimed was unnecessary clutter, but I had quite a few comments that puzzled me until it dawned on me there's another difference in my approach compared with the Given/When/Then style.

I always state the required behaviour in a sentence or two before giving the examples. Each behaviour is described by a specification like this:

Maybe it looks like a lot of work to write, but it isn't really. The structure is standard and each part is just a sentence or two.

Whereas in the Given/When/Then approach the business rule describing the behaviour generally isn't made explicit. The reader is expected to guess the rule from the examples, so naturally they have to have more context.

Occasionally it's difficult for the domain expert to immediately state the abstract rule and easier to start with concrete examples, but once the expert has explained a few examples then we can usually begin to take an educated guess at the rules behind them—"Ah OK, so the rule is X?"—and then have a useful discussion.

But if we don't make the rule explicit we only have the concrete examples, so they have to be made much more verbose—and potentially implementation-specific—so that readers can correctly interpret them.

It is true that Given/When/Then style examples are much better at avoiding implementation lock-in than test scripting ("click this, click that..."), but I would argue that context-free examples with explicitly stated rules are even better, both in terms of avoiding lock-in and in terms of readability (how long it takes the reader to understand the behaviour expected).

8 comments:

Matt Wynne said...

Quite right David. The Cucumber community has been learning that a 'declarative' style of Given / When / Then scenario is much easier to understand, and more maintainable.

See http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html

Matt Wynne said...

Before I started systematically using examples to communicate business rules with stakeholders, we ended up guessing what the scenarios were when writing tests or code, and missed them or got some of them wrong altogether. Equally as you say, when the examples are the only thing you have, the bigger picture gets missed. I've seen this happen where the code that was written from the examples was far more complex than it needed to be, because the examples were slavishly implemented without looking for a pattern.

I like your technique for using plain english to describe a rule, and since I implemented markdown descriptions for Relish[1], I've been using them more and more in my Cucumber tests. I think most people don't realise they can put a free-text description with each Scenario in Cucumber.

I prefer having the rules specified as executable tests though. I think what's important is to keep the test scenarios free of what Dale Emery calls incidental details [2] and make them as declarative [3] as possible.

I think you can get this wrong or right whatever test tool you use. The important thing is to try to think at the highest level of abstraction you can, whilst still communicating enough with everyone.

[1] http://relishapp.com/relish/relish/dir/projects/render-description-as-markdown
[2] http://dhemery.com/pdf/writing_maintainable_automated_acceptance_tests.pdf
[3] http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html

David Peterson said...

Thanks for the comment Matt.

I agree, if there is only one possible case, the rule can directly function as a test. But most of the time there is an underlying rule and multiple examples of applying that rule. In this case, it's hard to write the rule as a single executable test. You have to write multiple tests - the examples. If you don't explain the rule it's left to the reader to figure out.

All the "action" so-to-speak is in the examples, so writing a free-text comment explaining the rule doesn't really seem important.

What I've found is that it really helps to make things clearer not only for the reader, but also in your own mind, so that when you choose examples you can choose them well and keep them simple.

Anonymous said...

I find that whenever a scenario becomes too 'wordy'/repetitive, a regular test case format (input -> expected output) can make them easier to read. With so much emphasis on scenario testing in ATDD I think we sometimes forget other approaches to specify behaviour.

With tools like cucumber you can achieve this with tables and scenario outlines.

https://github.com/aslakhellesoy/cucumber/wiki/scenario-outlines

Do you use this sort of format yourself? Wouldn't you get similar benefits from this approach?

David Peterson said...

I usually use a tabular layout to show several examples, but I prefer Concordion to Cucumber. I like having control over the look and feel - it's HTML, so I can link specifications together, use stylesheets, emphasis etc. and make them work as documentation as well as tests.

See: http://www.concordion.org/Tutorial.html#executeTable

David Peterson said...

@Anonymous
All the scenario thing does is remove some duplication from the examples. Don't get me wrong, that is a good thing. But what it doesn't do is explain the underlying rule.

I've found that describing the rule is super-useful. Try it and you'll see what I mean. You can do it in Cucumber. You just have to write a comment above the examples.

Liz Keogh said...

Hi David, I agree. I normally suggest that people describe the behaviour in general terms (acceptance criteria), then provide some examples around that behaviour to help ensure a common understanding (or more usually, to make a lack of common understanding more transparent and uncover scenarios we forget about).

Cucumber and JBehave allow either approach, as the scenario title itself is part of the description. That's usually got the summary of behaviour you're looking for (and there's a free text field if that isn't enough).

Liz Keogh said...

Hi David, I agree. I normally suggest that people describe the behaviour in general terms (acceptance criteria), then provide some examples around that behaviour to help ensure a common understanding (or more usually, to make a lack of common understanding more transparent and uncover scenarios we forget about).

Cucumber and JBehave allow either approach, as the scenario title itself is part of the description. That's usually got the summary of behaviour you're looking for (and there's a free text field if that isn't enough).