Remote Research

No longer must you only test your interfaces with people who come to your office or some expensive lab. So stop doing boring research! Stop having a boring life! Read this book!”

Mark Trammell, User Researcher, Twitter

Nate and Tony have done it! They’ve described beautifully one of the user experience profession’s best-kept secrets—conducting remote usability tests. This book is a must-have for anyone thinking about remote testing, since it tackles everything you need from soup to nuts. I can’t wait to get a copy for my own bookshelf. Oh, it also will melt your face remotely.

Jared Spool, CEO and Founding Principal, User Interface Engineering

• Chapter 1: Why Remote Research?
• Chapter 2: Moderated Research: Setup
• Chapter 3: Recruiting for Remote Studies
• Chapter 4: Privacy and Consent
• Chapter 5: Moderating
• Chapter 6: Automated Research
• Chapter 7: Analysis and Reporting
• Chapter 8: Remote Research Tools
• Chapter 9: New Approaches to User Research
• Chapter 10: The Challenges of Remote Testing
• Conclusion: Don’t Waste Your Life Doing Pointless Research

These common questions about designing for kids and their short answers are taken from Nate Bolt and Tony Tulathimutte’s book Remote Research: Real Users, Real Time, Real Research. You can find longer answers to each in your copy of the book, either printed or digital version.

1. What is remote user research, anyway? Is it anything like focus groups or surveys?
   Remote user research is simply a type of user experience (UX) research that’s conducted over the phone and Internet, instead of in person. In general, UX research seeks to understand how people interact with technology. Unlike focus groups and surveys, market research techniques that are used to learn people’s opinions and preferences, UX research focuses on studying people’s behavior. In that sense, remote user research isn’t really like market research; however, both remote user research and market research can be applied toward improving the design of existing technologies and inspiring new ones.
   See Chapter 1, page 3. 
2. What kinds of remote research are there?
   That’s a huge question, and we spend a good chunk of this book introducing and describing the many varieties and specialties of remote research out there. In general, there are two branches of remote research: moderated and automated. In a moderated study, a researcher talks directly to the participants as they use the interface in question, and it’s good for obtaining rich, qualitative feedback. In an automated study, you use online tools and services to gather behavioral or written feedback and information automatically, without the researcher’s direct involvement.
   For more about moderated testing, see Chapter 2, page 32, and all of Chapter 5. For more about automated testing, see Chapter 6. 
3. I’m skeptical about remote research. If it’s so great, why haven’t I heard of it? 
   A lot of the misgivings that people have about remote research come from its novelty. The method is still cutting edge, and the technique requires a certain degree of know-how. Until now, there hasn’t been a book you could learn the method from—which is, of course, the reason we wrote it.Still, lots of companies have done it, with great success. We’ve done remote studies with Sony, Autodesk, Greenpeace, AAA, HP, Genentech, Wikipedia, UCSF Medical Center, the Washington Post, Esurance, Princess Cruises, Hallmark, Oracle, and Blue Shield of California, among many others.
4. I’m still skeptical. Can you really get valid behavioral feedback without seeing your participants in person?
   Since remote research is conducted over the phone and Internet, many people worry about missing “rich details” like facial expressions and body language. First, we believe that for most user research studies, the way that users interact with the interface and their think-aloud comments are the only really necessary things to focus on. And on top of that, much of the tone does come through the user’s voice and language. We weigh the pros and cons of in-person research and remote research in Chapter 1, pages 5-15, and discuss moderating over the phone in Chapter 5, page 110.
   Still skeptical? Then you should check out our exhaustively documented study for Wikipedia, complete with full-session videos and highlight clips at http://usability.wikimedia.org/wiki/Usability_and_Experience_Study. It includes both lab and remote sessions with identical goals, so it’s a good comparative case study.
   The best way to see if a remote study is for you, however, is by getting your feet wet with a quick, painless pilot study, which we’ll walk you through in Chapter 2, page 30. 
5. I want to cut costs for my user research study. Is using remote methods a good way to do that?
   Not really. You might save on costs related to travel expenses, renting a lab facility, or hiring a recruiting agency, but then there are the expenses of the specialized remote research tools and services you’ll need, and where the researcher’s time, participant incentives, and project timeline are concerned, nothing is much different.
   We cover the equipment requirements of a basic moderated study in Chapter 2, page 28, and the costs of many remote tools and services in Chapter 8. 
6. Where can I get people to participate in my remote study?
   You can technically use any methods to recruit for a remote study that you’d use for an in-person study: email contacts, recruiting agencies, and even craigslist ads (blegh!) are still an option. However, in this book, we introduce a method called live recruiting, with which you intercept visitors to your Web site by using a pop-up form to get them to participate in your study right away. We strongly believe this approach is ideal for remote research because it allows you to do what we call “Time-Aware Research,” which we introduce in Chapter 1, page 10 and discuss in depth in Chapter 3.
7. How on earth can you call this a book about research without way more academic references and doctoral degrees?
   This book is aimed at people who want to hear practical information about how remote research is done in the real world.We don’t claim absolute peer-reviewed scientific rigor, but we are sharing what’s given us the best results after almost a decade of remote research experience: 2,615 moderated users, 2,676 automated users, 234 projects, 19,120 project hours, and 89 clients to date. Simply put, this is just what works for us, and we think it will work for you, too. (And for what it’s worth, Nate and Tony both have cognitive sciences/human-computer interaction degrees: yes, we roll that deep.)

• Chapter 1 (PDF)

You are holding in your hand a portal into the future. The approaches discussed in this book are the first steps toward a user research methodology suited to our technological context. In comparison, lab research will be increasingly seen as an archaic approach to understanding people, akin to phrenology or trepanning.

You see, lab usability engineering was born of a simpler time. Files were stored on floppy disks, and the computer wasn’t connected to a network. People used computers for Calculation (basic math, spreadsheets, etc.), Creation (word processors, graphics programs), and Capture (data entry). Computers would have only one “program” running at a time. And someone could be expected to focus on a single task at hand for many minutes, if not hours on end.

These days, our customers’ technological world is much more complex. The bulk of their time online is spent engaged in Consumption (browsing the Web, listening to music) and Communication (email, instant messaging, Twitter), though they still Calculate, Create, and Capture. They have multiple applications open and multiple windows within those applications. Thousands of files and emails fill their hard drives, and they’re managing multiple devices, including computers, mobile phones, iPods, and digital cameras. Technologies have driven users to a point of extreme distraction—recent research has shown that workers are interrupted an average of every 11 minutes.

Although the world has changed, the methods of standard usability practice are essentially the same as were practiced in the early 1990s. We recruit participants who satisfy demographic requirements, invite them into a fluorescent-lit lab with an imposing mirror along one wall, ask them to use a computer they’ve never seen before, have them engage in a set of scripted tasks, and “think aloud” while doing so. These participants won’t have an IM window pop open unannounced, nor be able to click on the bookmark bar to see their Facebook page, nor be able to pull up the notes they saved in a draft in their email. When everything about the observation environment is so unnatural, how can we expect our findings to provide legitimate insights?

This is where the genius of Nate Bolt and his crew come in. Since I’ve known them, they’ve experimented with technologies in an effort to capture the most authentic user research data. Whether it was the pioneering screen sharing and recording tool Ethnio (which fundamentally changed how I considered usability), or real-time uploading of video captured in a drive-along study for Volkswagen, their mission is to deliver the unvarnished truth.

Now, it’s easy to get caught up in the tools. While remote research is enabled by nifty technologies, which should be of secondary interest at best, I know that for Nate and Tony, this research approach is about mindset. When you simply desire to understand people in the most direct ways, you come up with clever means of doing so. And sometimes, you can’t go “remote.” When I first met Nate, he crowed about how they kitted out a usability lab with hidden microphones and cameras so that the participant would be as comfortable as possible. Or when researching the use of multiplayer video games, he recreated a cozy living room set up in his office, with snacks and lounge chairs, to make gamers feel at home.

And that’s how this book is a time machine. Only the Flying Spaghetti Monster knows what the future will bring. The only thing that’s certain is that it’s going to keep changing. To prepare for that future, you need the mindset represented in this book, in order to figure out which approaches will provide the best user research data given your circumstances. Like Bolt | Peters, don’t settle for standard practices; instead, play with new ideas that will deliver fresh insights.

—Peter Merholz

Louis Rosenfeld of Rosenfeld Media went to bat for our book before it was even a book, and surely allowed this project to become what it is: the greatest work of literature in any language.

Marta Justak, our wonderful main editor, initiated us into the terrifying and desolate world of technical book writing, and also forgave our San Francisco liberal attitudes vis-à-vis deadlines and template conformity. Our technical editors, Chauncey Wilson (Chapters 1-5, 7-9) and Donna Tedesco (Chapters 6-8), helped us dig up references, challenged sloppy thinking, put our thoughts in train, and kept it very real.

Many thanks to all our contributors: Peter Merholz, Andy Budd, Brian Beaver, Julia Houck-Whitaker, Dana Chisnell, Emilie Gould, and Carol Farnsworth. They wrote the words we were simply too afraid to write. Thanks also to Elizabeth Bacon from Devise for providing great supplementary info for Chapter 8 and to Danny Hope for providing photos of the Brighton University Usability Lab.

The early feedback of our preview editors—Lori Baker, Kyle Soucy, and Caleb Brown–helped us to look smarter than we are. Also thanks to our amazing Bolt | Peters team of remote pioneers—Cyd Harrell, Frances James, Kate Nartker, Julia Houck-Whitaker, Mike Towber, Julian Wixson, Brian Enright, and Alana Pechon—for battling out remote testing over the years and for their practical UX research insights, extracted by the authors with many a harassing early-AM phone call.

Now the advisors: many thanks to Donna Spencer and her book, Card Sorting, for the professional advice on (what else but) card sorting. Matt Thomson of Kronenberger | Burgoyne advised us on matters of Internet privacy, recording, and consent law. Susan Kornfield also provided much helpful guidance.

Part of Chapter 5 was adapted from an article Tony Tulathimutte and Cyd Harrell wrote for UPA Magazine (Vol. 7.3). Parts of Chapter 9 were adapted from our earlier writing, which appeared on the Web site Boxes and Arrows (boxesandarrows.com) and in a paper for IA Summit 2008 called “Portable Research.”

Remote Research Blog View all Blog posts

• Remote Research is now on sale!,
  on February 11, 2010
• User Research Friday 2010,
  on February 8, 2010
• Time-Aware Research,
  on December 10, 2009
• Screening Out Liars From Your Usability Study,
  on November 17, 2009
• Remote Testing Tools Round-up by Liz Bacon,
  on October 28, 2009