Let’s just get it out there right now: All qualified participants are NOT created equally (in usability testing I mean). Well maybe they were created equally, but lengthy experience with a product has a way of changing that. The more experience a group of users has with a product, the more experience a researcher is going to need to run the project.
Two spaces we tend to play in are consumer website usability and enterprise software user experience. The B2C website industry tends to focus mostly on newer or casual users, while enterprise software has far more experienced users that interact with the product frequently. New users need clear updates confirming that they are in the right place and doing the right thing (example: buying a gift for someone online). Experienced users already know that stuff because they’ve used the product repeatedly. They are more interested in efficiency and flexibility (example: running a financial report showing asset depreciation over 5 years).
Now which one of these is an easier experiment to run? Watching a newer user navigate a website that you quickly become more familiar with they they are after the third session, or watching a seasoned user run a complex software program that you may not even understand on a theoretical level? In my opinion this is what separates an “ok” researcher from a research “wizard”. The wizards can analyze abstract patterns that a group of seasoned participants will reveal. Not only that, but they can leverage the experience level of the participants to deepen the study.
A good moderator knows how to script an experiment to get unbiased results with minimal observer effect. A great moderator knows how to spot patterns with experienced participants that such a group may not even be able to articulate. And a “research wizard”? They have the delicate skills to handle both new and experienced users in their own unique way to reach the same ends – to enhance your user experience.