Introduction
Yesterday in Part 1, we looked at what equivalence partitioning is and how we use it in Jira QA. In Part 2, we will cover how equivalence partitioning might explain some odd behaviour you’ve noticed from your friendly neighbourhood QA/test engineers.
Explaining QA behaviour
Here are some symptoms of equivalence partitioning that you may have noticed:
We want to know how things work
When I first transferred onto the OnDemand project, it initially involved taking our existing Jira Studio product and making it run on a new platform. I asked for information about the new platform and how it differed from the old. Initially I was told, “You don’t need to know all this. Can’t you just… you know… test Jira Studio on the new platform and see if it works?”. But it doesn’t work that way – there are tens of thousands of potential tests we could have executed; how could we have determined which were useful and which weren’t? Would we have to test every single checkin twice – once on each platform? What were the equivalence classes?
By learning more about the infrastructure and how it differed, we could focus on the cases that were in different equivalence classes on the two platforms. Everything that was the same on the two platforms fell into the same, existing, well-tested bucket and didn’t need to be duplicated. So the areas that needed to be explicitly tested on the new platform changed from “all of Jira Studio, which is every feature in of all of our products” to a handful of specific areas, for example:
- Where it touches the file system.
- Where the server sends external requests.
- Where it deals with requests/responses that were likely to be mangled by sitting behind a proxy.
Based on that, we could focus on specific scenarios and pinpoint the bugs without wasting a lot of time retesting safe areas. And it worked – we found many issues specific to Jira Studio running on the new platform and fixed them before release. No platform-related issues were found after release.
We overreact to some kinds of minor bugs
Sometimes bugs come up that indicate our equivalence classes are incorrect. For example, we might be operating on the understanding that a certain field behaves the same way on the Create Issue screen as it does on the Edit Issue screen. If a bug comes up that affects only one screen and not the other – even if it’s a really minor bug – we’ll be jumping up and down wanting to know the root cause. It’s not that we really care about the bug itself; it’s that we care about the breakdown of the equivalence.
If it turns out that the field’s behaviour on the two screens cannot be considered equivalent, then that suddenly doubles all testing required for changes to that field. On the other hand, if it turns out there’s a really specific reason why they’re different for that case and not otherwise, the equivalence can remain solid with an exception or two.
We get fussy about test environments
Since we base equivalence classes partly on past bug history, we have some strongly-held beliefs about valid test environments.
For example, we often bundle plugins with Jira. I used to consider “testing a plugin uploaded via UPM, then later checking that the same version number is bundled” to be equivalent to “bundling a plugin, starting up that build, then testing it as a bundled plugin”. This was until we had a few issues that broke down the equivalence – one where a plugin jar was mislabelled in the maven repo, one where classes in the plugin were being instantiated incorrectly, and a few cases where a third-party plugin developer re-released a new version of the plugin with the same version number. Now we require that all bundled plugins have a final sanity check after bundling to ensure they work as expected.
We are confident about when to stop testing
The beauty of equivalence classing is that we have a better idea of when to stop – that is, when all classes have been covered. Without it, it’s easy to drag testing on for weeks; executing more test cases at random, never feeling quite confident about the software quality, and pretending that more time testing means higher-quality software. But with equivalence classing, we have the knowledge and the confidence to say, “We have covered all equivalence classes for this feature, and throwing more random sets of data at it is just wasting time and gives us no further coverage. Let’s move on.”
Being confident about most features means that when a change really is risky and we haven’t yet got complete coverage; and we say that we are not confident and it shouldn’t ship yet, we are listened to.
Conclusion
Equivalence partitioning is critical to efficiently testing a large, complex product like Jira. It helps us reduce testing time while increasing the number of bugs caught and squashed, helping us ship higher-quality software sooner.