Screening Out Liars From Your Usability Study

A new article on 90 Percent of Everything discusses a few ways to screen out potential “fake users” who lie about their qualifications to participate in your study:

In fact, a lot of liars can be screened out by writing a really good screener questionnaire. For example, here’s a decoy question that the Mozilla metrics team used in their recent Test Pilot survey.

This is really handy to know when live recruiting users from your website–the risk for fakers is even higher when the recruiting pool consists of anyone who comes to your website. Chapter 3 of our Remote Research book also touches on this subject, specifically as it relates to live recruiting. Here are two more pointers:

–Occasionally when people catch wind of a paid survey offer, they like to post it on “bargain hunting sites” like FatWallet. If you get a sudden surge of recruits, that may be the reason; check the referrer data in your traffic log or analytics to confirm where users are coming from.

–Use open-ended questions to test people’s motives for coming to the site. If someone responds to the question “Why did you come to the site today?” with a vague answer like “To check the offerings” or “Just looking around”, consider that a yellow flag, and follow up with more specific interview questions.