How to Run User Tests Successfully: The ‘Why’ Question

User tests: In a world, where data-driven decision making is often prevalent, some people feel uncomfortable with agile methodologies, as those provide only a few useful metrics. One of those few, however, is the cycle time from idea to shipping a valuable product increment to your customers.

If you want to optimize this metric for your organization, speeding up your product discovery process is essential. And this requires two things: a) rapid prototyping and b) people to test your prototypes with. That’s the main reason why running user tests continuously is so important. 

Learn how to best organize and run user tests in this series of six blog posts. Today, we start with answering the “why” question and what huge benefits user tests will provide to your product discovery and delivery process.

The Purpose of User Tests

When testing software – for websites, mobile apps or desktop software – these tests are mainly divided into two types of tests: quantitative and qualitative tests.

Quantitative User Tests

Quantitative user tests are being used to determine the effectiveness of certain actions in a realistic environment with literally scientific methods. One example would be an A/B test to optimize a conversion rate with two different versions of a call-to-action element. (For example a button.) Google is known for excessively examining pretty much everything in the context of quantitative testing. It seems that from time to time the results are a little over the top, though, or as Douglas Bowman puts it: “Goodbye, Google”. 

This quantitative method will optimize certain processes very well, such as the registration of new users or the sales funnel from the landing page of a marketing campaign all the way through to the confirmation page of an online sale. The method is based on the evaluation of visitor data and requires a certain amount of participants before the results can be considered statistically relevant.

Conclusion: Quantitative tests provide a picture of what users do, but not why they do it.

Qualitative User Tests

Qualitative user tests, however, are being used to fully understand the usability of an application. Unlike quantitative tests, this kind of test examines the question, why the users react in the way they do. This information cannot be determined in quantitative tests.

Qualitative user tests work very well in practice, as the most urgent problems of a website or application are usually easy to find. If for example four out of five testers ignore or use a feature differently than what it was originally intended for, then this indicates that there is cause for a possible change of this feature. Further testing with a larger quantity of participants is no longer required, as the result is clear – therefore, the marginal utility of each additional test will decrease.

Conclusion: The utilization of do-it-yourself user tests is a valuable and cost-effective means to identify heading in the wrong direction and/or redundant ideas early in the process. User tests enable developers to optimize functions and processes from the start and with that prevent irrelevant applications alltogether. Nothing is more expensive in product development than a “we know what we need to build” attitude.

The benefits described make user testing an important tool of all agile methods of product development. During the Lean Startup method, user tests represent a part of the product discovery process and they support the product owner of the Scrum process during the development of the product roadmap – regardless of your company specific dialect in those agile methods.

Conclusion: Do-it-yourself user tests are an elementary part of the product discovery process because they allow a quick and cost effective testing of hypotheses. Therefore, they are an important asset for any decision regarding any investments towards a valuable, usable and technically functional feature. To be able to push any product decision as far back as economically feasible, running user tests is non-negotiable. And pushing back those decisions is the essence of agile software development.

Even though the B2C e-commerce environment may be easier to handle than the B2B environment, the in the following chapters described procedures can also be applied in an adapted form. 

When Should You Start User Tests?

From my experience the answer is very simple: Very early in the life cycle of any idea. For a successful product management, an early validation of hypotheses is the most important step towards product discovery. 

Conclusion: A good product organization excels at bringing down the cycle-time of testing hypotheses.

It is a misconception to think that user test only make sense, if during the interview the participants are able to work with a fully functional beta version of the new product. In fact the opposite is true – even napkins with drawings on them, prototypes made out of paper and click-dummies are rather grateful test objects and are generally well perceived by all user testing participants. This often has something to do with the initial surprise of the more curious interviewees, as they feel that the user testing seems more sophisticated and that they are taken more seriously.

For the first feedback regarding an idea, you don’t even have to organize a user test. All you have to do is approach the person you would like to interview and ask then for 5 minutes of their time. This can happen in the Mecca of Berlin’s startup scene – the ‘St. Oberholz’ on the ‘Rosenthaler Platz’ – or in larger companies you ask for the first feedback of your colleagues. Highly recommended is here, for example, the canteen hacking: During the lunch hour, you join a table and ask the opinion of those sitting there. In my experience, most people love to help and they appreciate it, that they are seriously being asked for their opinion on an existing or new product.

Conclusion: The sooner the user testing begins, the better the results – even if this means, that an idea will be abandoned due to lack of interest. (You just saved money, brain, and time.)

Download the Ultimate Checklist for User Tests

Never again forget anything necessary to run a professional user test: just download this proven checklist PDF for free, that was fine-tuned over 2 years in the trenches of continuous user testing.

If you do so, I would appreciate your feedback on how this worked out for you. (Please note that downloading the PDF will also subscribe you to our awesome weekly, completely hand-curated newsletter, if you haven’t yet signed up already. You can unsubscribe at any time.)

Download the checklist here…

What Should Be Tested in User Tests?

The extent of user tests can cover a rather wide spectrum: From just one single feature up to complete applications. Therefore the question as to what should be tested can only be narrowed down. However, there are some general rules that apply to the test range:

1. The more complex an application is, the less likely the application as a whole can be tested. Example: eBay – here, singled out processes or features can be tested – but certainly not „eBay“ as a whole.
2. On the other hand a mobile App can often be tested as a complete package, because of its mostly limited functional range. 
3. The more precise the test object is, the shorter will be the time for the interview. As an example: When an idea for a new function (feature) is supposed to be tested, this can be achieved with simple click-dummies within a time frame of around 30 minutes. Here the general perception of the user is sufficient whether the feature is useful or not. Only when the feedback is positive and actually leads to a prototype, the user tests in the second testing round should be more detailed in their approach towards improving the usability of the feature in question.
4. Interviews over 60 minutes long should be avoided. Many participants have a concentration span of around 30 – 45 minutes. If an interview lasts longer, it’s not only featured by a diminishing return. Also the probability of possible “false positives” increases – where interviewees will say, what they think the interviewer would like to hear to finish the interview quicker.
5. The smaller the scale of a test object, the more it makes sense to replace and/or support the testing of individual interviews with crowd testing. For this, the test object will have to be a little more advanced or at a later stage of the product, than it would be required for classic user testing. In a face-to-face user test, the interviewer may interfere during the test, where he/she may adjust the interview questions according to the feedback given by the interviewee. This does not work with crowd testing.

Recommendation: Before you test your own application, you should first test one or two of the main competitors. Why? Somebody has already built a fully functional “prototype”, which addresses a similar problem as your own application. It usually is a great learning experience to receive direct feedback on how your product could be improved with regard to the competition or what the interviewees like about your competitor.

How Often Should You Have User Tests?

User tests should be carried out on a regular basis, which means at least once a month – and that regardless of concrete tasks. It is important that user tests become a critical element of the product development process within your organization. 

It is not important under what flag inside the company this will be established. For example, if the label ‘product discovery’ as a collaborative process helps within the Lean Startup methodology to keep the budget and attention of the stakeholders – then so be it. The main thing is that you can run user tests regularly, also referred to as “continuous user testing”.

With Whom Should User Tests Be Organized?

No user tests without users – and the choice of users will be determined by the application that is to be tested. If for example a general usability test is scheduled, pretty much all users – even strangers – are suitable interviewees. These kinds of tests do not require a certain target group.

However, if the test should be carried out for users, who only use their tablets and smartphones because a frontend of a web application shall become responsive – then obviously only these users need to be addressed, to keep the loss of divergence to a minimum.

Should you be looking for interview partners outside of your own user base? Absolutely. Especially as during the early stages of new products this is the only option – as you don’t have any users yet.

Conclusion: No interviewee is capable to identifying all of the usability problems in one user test. On average, a success quote of 50% is considered a good result. You will always need multiple attempts with different interviewees to collect a useful list of results. And there is nothing wrong with trying to identify these at first amongst your own users.

The Costs of User Tests

The initial costs for regular user tests are rather moderate and – based on my calculation – amount to around 10% of the costs of a usability agency in a similar set-up.

If the team is supposed to receive their own equipment exclusively for user tests, then the initial costs for the one-room method, which will be described later, including laptop, beamer, and screen recording software will amount to approximately $1,500. This is particularly useful, as the software set-up can be maintained easier, for example if a VPN is needed to access staging servers.

The variable costs of the user tests described here are typically around $150 to $200 for room rent, plus $50 for beverages. Where it applicable there may be costs for advertising to acquire interviewees, if you want to use people outside of your own user circles. 

The largest expense is usually the compensation of the interviewees for their time.

No external agency for the casting of interviewees was used. (All participants had answered to our own marketing campaigns.) If you cannot cast the interviewees yourself, hiring a casting service will set you back around $150 to $250 per interviewee. (Berlin rates.)

The overall effort for the organization of these user tests will usually take one whole man day to prepare invitations, create the itinerary, test arrangements as well as the evaluation and write the protocols. 

On top of that, you need to add the staff costs for the participation of your colleagues–the observers–. (These will also apply, though, if you are using an external agency to run user tests.)

Next Post: Interviewees

See you in part 2 of this six parts series, when we will talk about interviewees: where to find them and how to get them interested in your user tests.