Introduction to Approval Testing

This first module goes through the main characteristics of Approval testing:

Test cases check actual program output against a previously approved value, and any difference will fail the test.
Normally, a human inspects and approves some actual program output when creating a test case.
Raw program output may be processed into a more convenient format before being used for approval and comparison.
Design a Printer to display complex objects, instead of many assertions.
If actual program output is not yet available, the approved value may be a manual sketch of the expected output.
Approved values are stored separately from the sourcecode for the test case, although in the same VCS repository.
When a test case fails, you can use a tool to inspect the differences and easily update the approved value.

We work through some exercises and gain experience with the ‘Approvals’ tool. We will work in small groups and each group can agree which programming language to use. The exercises are generally available in Java, C#, Python and C++.

Session Outline

10 min connect: Approval Testing warm up questions
10 min connect: Short introductions and form groups
10 min concept: Demo
30 min concrete: Do some approval testing
5 min conclusions: How about?

short break

5 min connect: Read through the How about? questions and add more you think of
5 min connect: Does fixing the tests take longer than fixing the code?
10 min concept: Characteristics of Approval testing
10 min concrete: Update some existing tests
5 min concept: Approval testing detects both bugs and features
20 min concrete: Update tests making one change at a time
10 min conclusions: How can you reduce test maintenance?

short break

2 min connect: is testability an architectural concern?
10 min concrete: analyse the problem
10 min concrete: Demo pre-comparison processing
20 min concrete: fix the problem your own way
10 min concept: Pre-comparison processing
5 min conclusions: Strategies for testability

short break

15 min connect: introduce Approvals Puzzles
10 min concept: Text-Based testing
15 min concrete: Demo Approvals Puzzles
5 min conclusions: Strategies for varying output

short break

5 min concept: Remaining characteristic - sketching
15 min concrete: demo of sketching a new test case
10 min conclusions: walkabout posters & code review
15 min retrospective: gather observations

Part 1

Use this learning hour “Approval Testing plain text strings with Verify”. As well as answering warm up questions, introduce yourselves and form groups of 2-4 where you agree on a programming language. Mix up people from different companies.

Conclusions - How about?

On a sticky note, write a question you want answered about Approval Testing
Stick it on the “How about?” board
Take a break

Part 2

Work in small steps. First go through ‘how about’ questions. Link to them when explaining approval testing characteristics. Follow this learning hour: “Test Maintenance”

Part 3

Handling tricky output. Follow this learning hour: “Design for Approval Testing”

Part 4

More tricky output - filtering and sorting. Follow this learning hour: “Approvals Puzzles”

Part 5

Sketching.

Concept - Sketching

If actual program output is not yet available, the approved value may be a manual sketch of the expected output.

Concrete: demo of sketching a new test case

Add Tax calculations to the receipt. Sketch it first.

Conclusions: walkabout posters & code review

Get people to look through everything we’ve done today and discuss with someone what they’ve learnt. Write answers on the mind-map as sub-nodes.

Nodes on the mindmap:

When should an Approval Test fail?
How do you create a new Approval Test?
Is an Approval Test a kind of unit test?
Where should you store Approved output?
Can I combine approvals and assertions in the same test case?
Can you use Approval testing with all kinds of program output?
Can you write Approval tests for a program that doesn’t exist yet?
How should you update a program that has Approval tests?
How should you update Approved output?

Retrospective: gather observations

Ask people to note down what has happened today in these categories:

Liked
Learned
Lacked
Longed for