racolour (206)552-0606

Moderated vs. Unmoderated Usability Testing: A Dating Analogy

Have you heard people taking sides about what is better – moderated or unmoderated usability testing? It is time for us to add our two cents to this debate. In our opinion, both methods have equal value – the question is which one do you do first, and why?

Let’s play a little game. Pretend a friend of yours is having trouble getting past a first date. You want to help. How would you help your friend? with some moderated observation or some unmoderated observation? Let’s play out both scenarios:

Moderated Approach: Your friends date has allowed you to tag along. Even better, you get to ask them questions about the interaction during the date (can you imagine?!?). If you friend does a poor job on the “kiss” at the end of the date, you can see their reaction and ask them what happened. If they get up and leave in the middle of dinner because the date is too difficult to stand any longer, you get to ask why. You have the option of digging deeper at any time! An upclose and personal approach to collecting data. Jackpot!

Unmoderated Approach: Now pretend your friends date is not so into the third-wheel scenario. Or perhaps you want to stay out of the spotlight to keep from interfering with the results. You devise a system where you get your friend to carry a little device to record checkpoints and how long they took. The date shows up at the restaurant? Check. They make it to dessert? Check. They leave together? Check. You even get your friend to take a little camcorder with them so you can get more detail. 100 dates, same results – everything is fine until the kiss. Now you know when, but why?

What happened? Well… when you ask 3 dates in a row in the moderated study, they confess that your friend has bad breath. Not terrible, but a bit of a deterrent. On top of that, each person noticed that your friend had a bit of dandruff going on, and talked about their dog a little too much. None of these things alone would be enough to say “no” to a next date… but combined? Wait a minute! You weren’t even making a point to test for dandruff or bad breath, and you definitely weren’t keeping track of how many times you friend mentioned their dog because you didn’t realize that was even happening!

…get it? Unmoderated testing is really good at capturing quantitative data on preplanned events… as long as you know what you need to watch for. It is extremely efficient in that way. The difference is that with moderated testing, you catch lots of things that you weren’t even looking for, and also get more thorough qualitative information on the things you are intentionally observing. You wouldn’t need to go on 100 dates… I’m sure you would catch notice the important details in about… 3 dates. You could tweak the system at that point if the reason was obvious and resume with a modified scenario (breath mints and special shampoo).

Is moderated testing always better? Well, we wouldn’t go that far.  Moderated testing is generally good to start with so that you can catch things that may be issues before you even realize you need to look for them. It also helps you dig deeper into “why” something is happening. Moderated testing is also more expensive, so if you are on a limited budget in fairly well-known territory, it is a good compromise to get some quantitative data that you can work with further.

Usability testing (interactions with “things”) is very similar to relationships (interactions with “people”). We can collect a lot of data to serve us, but a half-decent researcher will always be ready for what they weren’t looking for – immersing themselves in the process rather than simply waiting for automated data to be handed to them.


Leave a reply