Non-Functional Testing

Functional testing is fairly straight-forward and easy-to-understand: does your software behave as designed, or, perhaps, as your users expect?  Examples of functional testing could be:

• Are the items selected correctly added to the user's shopping cart? 
• Does a patient's dashboard show dangerous readings in red?
• Is the email address validated when the user updates her profile? 
• Does a RESTful API call return the correct HTTP status code upon success or error?

Applications are only applications due to their raison d'être, e.g., their functional requirements; otherwise why bother.

Opposite functional testing is non-functional requirements testing, which is more difficult to define, execute, and interpret.  Without properly testing your non-functional requirements, based on whatever your organization deems important, your software may be unusable to your customers regardless of the functionality provided!

Understanding Non-Functional Testing

A former manager stated in a team meeting (paraphrased):

I want to start chaos testing our microservices to better understand how the loss of resources impacts performance and at what point our services tip over as user load increases.

Huh?  I initially believed the manager had lumped performance, scale, reliability, and robustness testing into one thing; further discussions added efficiency, capacity, and elasticity testing to the mix.  Most revealing, most discouraging, was the overall cluelessness of our supposed technical leader.

Clearly understanding the goals of your non-functional requirements is required if you are to correctly define and execute the supporting tests: what to test, how to test, and how to interpret. Unfortunately, no universal, ubiquitous template for non-functional testing exists; therefore, you'll need to determine what leadership expects as you (ideally) explore what it means to test the non-functional requirements.

Common Non-Functional Testing

Despite the extensive list of non-functional requirements your organization may deem important, certain types of non-functional testing are more prevalent than others.  In my opinion, performance, scale, and limits testing are the most common, the easiest to understand, execute, and explain.

Performance Testing

Performance testing is most easily explained and most obvious to techies and non-techies alike: how long does Activity X take to complete?  Directly observable, performance tests declare what is tested and what is expected elapsed time, e.g. for each unique execution or the average of all executions, thereby easily confirmable.  That said, what is a performance test varies widely:

• Adding a single item to a user's shopping;
• Inviting n people to your party, where n=1, 5, 20, 50, 100;
• Bulk uploaded medical readings (blood pressure, blood sugar, cholesterol, etc.) available on patients' online charts;
• Identifying, uploading, and installing new apps on your mobile device.

The results of performance testing are affected by other factors, i.e., the number of rows of documents persisted in the database, the number of real/emulated users during the test, the resources involved in testing the specific activity, and the hardware or cloud resource required.  You must understand or define these dependencies to ensure your performance tests are usable. repeatable and comparable:  a representative runtime environment is required, as testing against an empty database or with insufficient resources is unlikely to provide realistic results.

Scale Testing

Scale testing, which may also be called load testing, shows how the application or system responds as the workload increases, e.g., what happens when concurrent users, transactions, API calls, persistent storage increases.

What is tested and measured varies widely, depending on form, function, and complexity of the application/system under test.  When designed and executed correctly, scale testing shows the load which results in dramatic changes in the measured non-functional requirements, such as:

• Response Time: After small, geometric increases as the load grows, a dramatic spike - often exponential - impacting the user experience;
• Error Rate: The overall error rate stays stable until a certain load causes more error than acceptable;
• Throughput: For background tasks, the work queue cannot grow faster than the work being completed;
• Failures: Is there a load that results in an overall, systemic failure of the application.

Aside from the well-architected, well-designed, and well-implemented solutions, scale testing often shows the load, which results in dramatic (e.g., exponential) degradation in whatever is being measured, possibly making the application unusable or the system unable to complete its work.  The results are used to determine 1) if you have a problem that needs to be addressed and 2) where the implementation problem may lie.

Benchmark testing refers to formalized scale testing that allows comparison between different implementations (and often vendors), such as industry benchmarks for networking, storage, and processors.  For example, the storage benchmarks define the transaction mix, i.e., 50% read/50% write, 30% read/70% write, for different concurrent clients, queue depths, disk types, networking speed, etc.

Limits Testing

When evaluating technology solutions, customers should be reading specifications and asking questions to ensure the proposed solution meets their technical requirements.

[Yes, should, though buyers' remorse shows that, too often, purchase decisions are made without technical oversight.]

Software and software-as-a-service vendors publish their product's technical limits based either on deployment infrastructure, e.g., physical hardware or target cloud environment, or software architecture and implementation: documents stored, concurrent connections, log file ingress, message process, active LUNs connected to your storage area network.

Limits testing is intended to prove that the product can meet or exceed the published limits without causing errors or otherwise causing customer consternation: be assured that someone is currently testing those limits and will announce their results, especially when the limits are not achievable.

Wrapping Up

First and foremost, this is not intended to be an all-inclusive, deep-dive into application testing, functional or non-function: the topic requires more than a single blog post to adequately explain it.  There's a reason testing has become a career that is so much more important than it was fifteen or twenty years ago.

What I hope you walk away with is enough understanding and enough questions to help your organization better test the non-functional aspects of whatever application or system you're responsible for.  I hear software engineers and leaders preaching the importance of performance and scale testing without defining the problem to solve or declaring a hypothesis to prove or disprove.  You can't adequately test an application if you can't decide what you're testing it for.  It's extremely frustrating.

With that in mind, hopefully you'll ask the questions that clarifies what the objective is the next time you're asked to load test your application!