Simplify Test Cases
Writing and maintaining test cases per current industry standard is detrimental to the engineering effort of testing. A major component of this lack of realism stems from the belief that test cases must be written in such a way that anyone can execute them.
The dream makes sense: in a crunch it would be great if the entire company could pitch in and execute tests.
The beauty of this idea is skin deep at best. First, when was the last time that anyone in your organization expressed interest in helping test?
It’s not their job.
Should development be running behind, I never volunteer to help write code to get it done in time. That’s not my job, and I am not a professional software engineer. I would have too many questions about the code base and not have enough skill to write code that meets the standards of the company.
Similarly, test engineering is a profession and a discipline. The idea that “anyone can hop in and test” borders on being offensive. Test engineers are specialized professionals that exercise code in very specific ways in order to find incongruities. Incongruities within the code, incongruities with the requirements, or incongruities with the gut-feel of what an end user will expect or want.
Simply reading a test case and running through the steps like a robot is insufficient to exercise the code in a way that a test engineer would. In fact, if a test is written in such a way that anyone can execute it, I would argue that test should be automated.
We are not gaining anything by writing and maintaining a collection of tests that anyone can execute.
So what can we do?
I believe that written manual tests should be disposable. Automate everything else, and fill in the gaps with exploratory testing.
If a test is going to be executed repeatedly over the course of multiple releases, ie regression, automate it. A majority of web applications are modernized to the point that they have API’s and standard UI element identifiers.
What if a web application that I test is not modern enough to automate? That application is not going to survive in its current state for much longer. The software industry is rapidly changing and aggressive. That application that you’re working on has the attention of enterprising individuals who will make a better one. When they do, its testing will be automated. Be wary, and consider your future options.
An automated regression test bed ensures that the mindless, repeated work is left to the robots. New development deserves new tests, and there will be nooks and crannies which need to be tested during a single release cycle but not repeatedly. These are our disposable tests.
What information needs to be in a disposable test? It depends on who the audience for the tests will be. Usually that audience consists of three parties:
- Other test engineers
- Software engineers
- Product management
Other testers need to know the acceptance criteria. Each action that could fail and the non-failing result of that action.
Software engineers need to know that their code is being tested in such a way that it catches potential mistakes within that code. Realistically, unit testing and integration testing should be handled by the software engineer. Most developers I have worked with are only concerned about a discovered but and want to see proof that it is actually a bug.
This leaves functional testing of the developed software, and explaining a bug. So we don’t even need any information in a test case for the software engineer, we just need a well-written bug.
Finally, the product managers, or whomever on the project represents the business. They want to ensure that the software does what the business wants. This should tie back through the acceptance criteria. Those criteria are the engineers’ definition of what the product needs to do.
There needs to be a step between the definition of the acceptance criteria and the development of a piece of software. The business should not see the acceptance criteria for the first time when the test engineers meet to get sign-off that the tests cover all portions of the project’s expectations.
So, the business must sign off on the acceptance criteria just as the engineers must sign off on the acceptance criteria. Those criteria are the single common source of truth for what will be built.
Test cases need to cover the acceptance criteria. In order to validate that an acceptance criteria is met, tests should be able to be defined purely by their points of potential failure.
If we think of this in logical terms, we should be able to say if I do this, then this happens. That’s a test case.
All manual written test cases should be a single if/then statement.
Scenarios that cannot be written as a single if/then statement are too complex and can be broken down. Situations where many and’s are needed should lead the author to consider that too much detail is going into the test case.
Verbosely stating “And I sign in with username
I should be able to say:
If I log in and add an item to my cart
Then that item is added to the bottom of the list of things in my cart
admin/password1234
Full test case. Full stop. Plus that one can even be automated, so bonus.
Do we need to explain in multiple lines how to add an item to the cart? If so, then we need to re-evaluate the user experience of this application. Users do not read the manual, and we need to think about how we are writing our software in addition to how we are
We should stop copying and pasting lines and lines of words for manual test cases. It is a waste of time and it is impossible to maintain. Those tests often become a blob of ignored tests in the future anyway.
This application is going to get destroyed by users doing stuff that we do not expect, and we will catch those issues in exploratory testing. Many many test cases are written to validate basic usability, but usability should not require test cases.
Let’s redefine how we write our test cases and stop wasting time. A test case can be one or two lines, those lines can be tied back to AC and shared with the business. The simplicity will save time in creation, execution, and in maintenance of test suites because we can just archive or throw them away.