The 10 Laws of Testing

Words we as developers should live by: “Quality comes not from inspection, but from the improvement of the production process.” — W. Edwards Deming

Axiom: “Tests are code.”

Too many organizations treat anything that isn’t coded as disposable. It is obvious that testing is indispensable, yet time and time again we find teams who treat test automation and related material as second-class citizens. The tests are documents of user behaviour, inextricably linked to the requirements produced by the Product organization and connected at the virtual hip to the code used to create the feature.

If it provides value, it should be versioned, maintained, cared for, and respected as though it were a core feature of the product itself. This should include test case specifications, design and technical documentation, and bug reports.

Also: “Time kills confidence.”

You would think the more time you spend on a feature, the more time you have to refine, hone, test, and poke around at it. Counterintuitively, this is suitable for the “old world” style of development, with a test environment, a staging environment, and many grand hypotheses around how users will interact with it.

These laws attempt to pull the SDLC into the cloud-enabled world: tiny, incremental changes, rolled out to limited audiences before being put out into the world at large. “Keyboard-to-production in 10 minutes”-- This leads to faster feedback, fewer escapes, and higher confidence. The following are 10 laws for the new world of coding. 

The 10 Laws

1. Everyone Tests — Everyone

Every member of the team, no matter what process is used, no matter what product is being produced, no matter what industry—everyone—is responsible for the quality of the product. Product, engineering, testing, and even the surrounding functions: customer support, sales, marketing, business development, early access beta customers, the C-suite — everyone tests.

2. Measure Risk, Not Coverage

Assuming the team can even agree on a working definition of “perfection”, the mere pursuit of it causes attention to be diverted from the most important thing: the risks to the business of a critical defect escaping to production.

Before you start to worry about comprehensive coverage of all features, become obsessed with the half-dozen user flows that are the most critical to your business.

3. Test What the “Money” Wants

Every business, every division, and every team deploys a set of core features that impact revenue more than others. Alternatively, every team must maintain a set of features that are less impactful, but still necessary. Focus test efforts on the revenue-impacting parts before other considerations. For eCommerce, prioritize checkout flow over user profiles. For finance, prioritize security & money-handling workflows over informational pages.

4. Breadth Is More Important Than Depth

It is more important to test all areas of the product in a shallow way than it is to test some of the product in a deep way. Deep, multivariate combinations of business logic are intended to find the most obscure edge cases: this risks missing more obvious flaws in other high-priority areas.

Once breadth is achieved, how deep do you go on a particular feature? See Law 2.

5. The Only Perfect Signal Is the User's Signal

Until your users interact with your software, everything you do is theoretical.

Tests are models. They are approximations of user behaviours, based on information gained from past user behaviours. The signal we get from a test might be flawed by the environment, by logic flaws in the tests themselves, by unconscious bias, and even by incorrect data from previous models!

The only way to know what happens when a user uses the software is to observe what happens when users use the software. User journeys from production analytics should be correlated with test coverage to evaluate the effectiveness of a test strategy. 

Also, consider that the user experience includes elements that wouldn’t even be regarded as bugs, and might not be reflected in the analytics. Your job isn’t necessarily done when the build turns green.

6. Code Is Not Complete Until It Is Testable (And Tested)

Testability is the act of instrumenting various parts of the code for inspection. It is difficult to judge correct behaviour without allowing these signals to be polled and interpreted. This leads to a disproportionate amount of extra work, which adds time to the release cycle and draws focus away from the customer experience. Time kills confidence.

7. Every Test Should Lead to a Clear Action

If you don’t understand what to do when a test fails (either from a test point of view or a product point of view), the test is not providing value. 

This usually comes from tests having too many steps, or from the product not providing enough information on failure (which includes not being adequately testable. See Law 2).

8. Always Test Where the Signal Is Highest

Software testing has “layers” (from high to low): Production, UAT, Functional, Integration, and Unit. 

Higher-layer tests are critical to force interaction between disparate components developed by different teams, but the minutiae of edge cases can likely be moved down to lower layers. These lower-layer tests have fewer dependencies, avoid costly pipeline configuration/orchestration, and run faster than higher-layer tests.

As an example, a UI test should only be used to determine whether the User Interface is capable of rendering the outputs of an API. If you repeatedly test business logic through the same UI, you should move most of these tests “down” to the API layer.

9. Never Link Tests

All tests should execute without regard to the state of any other test. Test data should be managed such that each test lives in its own segregated scenario, and cannot be altered by another test.

Tests should be atomic and autonomous.

10. Prefer the Tightest Feedback Loop

All tests are feedback loops. They run through the product from a particular perspective and provide feedback to a particular person or team. The tightest feedback loop is the one that cuts out as much as possible to test the particular operation in question. Testing a wider loop than necessary introduces variables that can muddle the signals you get from the test.

Conclusion

People love to argue. You don’t have to look far to find it. These ten laws aren’t complete, and of course, there are many caveats and qualifiers for which neither readability nor word count will allow. I'd love to hear feedback from folks sitting in the chair; what are your 10 Laws of Testing? What did I miss?

These items are framed as laws, but I recognize that context matters and that you can’t always be perfect. Sometimes these laws are aspirational rather than practical. Sometimes you’ll find yourself breaking one or two (or ten) at a time—the important thing is to seek continuous improvement, not perfection.