This post is not intended to be an all-encompassing guide to when you should do a lab test, but merely a reflection on how my thoughts on lab testing have changed during the course of my career…
I started out my career in UX doing mainly lab tests, or tests in very controlled environments (think large conference rooms at market research firms). For those of us coming from strict Human Factors backgrounds, testing in a controlled environment makes sense because it allows you to weed out the impact of much of ‘the unexpected’ on the outcomes of the study. And for the most part I have no problem with that because it is good practice to maintain strict control over as many variables as possible in order to produce the clearest statements possible about outcomes. But critics of UX testing are right: Testing in a highly-controlled environment does not account for issues arising from real world use. Now, I’m not about to advocate abandoning controlled studies. I am going to suggest, though, that there is a time and place for studies utilizing a high degree of control.
For products that introduce a new interaction paradigm or a new flow to an old and overlearned process, test in a highly-controlled environment.
Whenever something new is being tested you want to understand what issues arise during use under ideal circumstances. This is because if someone cannot use a product as intended under ideal circumstances, then they won’t be able to use it when they’re agitated or sitting in a poorly lit environment surrounded by ambient noise.
If you have no idea how consumers use your products, perform an initial test in a controlled environment.
Many products still go to market without first being vetted by users. Eventually, someone realizes that they have no idea if they’re actually helping users accomplish what they need to, so they decide it’s time to do a user test. This is a tricky scenario for the researcher and the client. The client may have no idea how users actually perform tasks (for example, whether they have to use external aids to assist in task performance), so they may not communicate clearly their expectations for the test, or they may inaccurately describe the core tasks of the product to the researcher. Either scenario could result in the client being shell-shocked and/or angry at the results and at the researcher, and the researcher might get a bad rap if the results don’t meet the client’s expectations. However, in the case of a first time ever user test, it is best to propose a controlled study of an exploratory nature that will give a baseline of task performance and an understanding of users’ needs in an ideal environment.
If you need to run a competitive study, do so using strict controls over variables .
This is an extension of the previous guideline since there is at least one product whose user scenarios are not known or well understood. Essentially, you want to gauge how well users can perform the same tasks on different devices of the same type, but you want to do this under ideal conditions so you can claim it as some sort of a baseline for observed behaviors.
After reading the above explanations, I hope it is clear that I’m advocating for testing with a high-degree of control when you need to know the basics of the user experience. Once you know the basics it becomes easier to know where you stand as a researcher and as a product developer. You’ll know if you have enough information to formulate hypotheses for future testing–you’ll even have an idea if you need to do more controlled testing. Having the basics will also allow researchers to give warnings based on what they know about human behavior–just be sure to clarify your basis for the warnings, otherwise someone working with your client might attempt to discredit your results.