Surveys That Work

Caroline has created a must-read ‘one-stop source’ for those looking to conduct a survey. Her book guides the reader through all the necessary stages for creating a robust survey, from beginning to end. The book can be used by experienced survey methodologists looking to improve the surveys they run and by those who are new to the survey world. Refreshingly, its clear and simple language makes survey methodology accessible to the masses, and its practical approach supports this. Importantly, it also helps the reader work out whether a survey is indeed the right choice—a step which is all too often overlooked. Helpfully, the book provides suggestions for additional reading for those who want to explore particular aspects of survey design further. Read this book—and learn from one of the best.

—Laura Wilson, Data Quality Hub Lead, UK Office for National Statistics

This book is a sharpening toolkit for taking haphazard surveys and making them useful. As research gets faster and faster, this book will help you keep pace with changing landscapes, especially in digital industries.

—Akil Benjamin, Strategy Director, COMUZI 

If surveys are one of the research techniques you use (or might use), you need this book! In an easy-to-read, conversational style, Caroline takes you through the entire process with wonderful examples, helpful stories, and tons of good advice. And where else would you have an adorable ‘Survey Octopus’ to remind you of all the elements you need for a successful survey and of how to minimize the various problems that can impact the value of your results?

—Janice (Ginny) Redish, author of Letting Go of the Words—Writing Web Content that Works

Those of us in the user experience design practice have needed this book for a long time. If you have avoided using surveys as a tool for understanding the needs of the people who use your products, shy away no more. Caroline’s mastery of method and her encouraging voice come through in every sentence. This book is a delight to read and use.

—Dana Chisnell, Fellow, Belfer Center for Science and International Affairs and Policy Designer, U.S. Digital Service 

All praise for Surveys That Work and the Survey Octopus! This is crucial reading for all those who want to deepen their user-centered practice with quantitative research. A thorough exploration of all dimensions of the thinking process behind good surveys, this book will make you a better questioner!

—Misaki Hata, Service designer, NHS Digital

This book is delightful; I’m excited to share it with you. Caroline will help you know when to do a survey, and how to do it well. When we do good research, we can make good decisions!

—Kathryn Summers, Professor, University of Baltimore

Chapter 0: Definitions Chapter: What Is a Survey? And the Survey Octopus
Spotlight A: Four Different Types of Surveys
Chapter 1: Goals: Establish Your Goals for the Survey
Spotlight B: The Net Promoter Score and Correlation
Spotlight C: Satisfaction
Chapter 2: Sample: Find People Who Will Answer
Spotlight D: Statistical Significance
Chapter 3: Questions: Write and Test the Questions
Spotlight E: Privacy
Spotlight F: Questions to Ask When You Choose a Survey Tool
Spotlight G: Choose Your Mode—Web, Paper, or Something Else?
Chapter 4: Questionnaire: Build and Test the Questionnaire
Spotlight H: On a Scale from 1 to 5: (Likert and Rating Scales)
Chapter 5: Fieldwork: Get People to Respond
Chapter 6: Responses: Turn Data into Answers
Spotlight I: A Good Chart Is Easy to Read and Honest
Chapter 7: Reports: Show the Results to Decision-Makers
Chapter 8: The Least You Can Do

I can remember when—ten years ago!—Caroline first told me that she was going to write this book.

I immediately thought it was a great idea. At the time, I’d never done any surveys myself since I was such a huge fan of qualitative UX research methods (especially usability tests), as opposed to quantita- tive methods like surveys.

But even though I’m biased (after all, I did write two books praising usability testing), I like to think that I’m not a jerk about it. I know there’s value in quantitative methods, and I always thought it would be great to be able to do a quick survey and get useful answers to specific questions, like “How many people do x?” and “How many like x better than y?” So I was delighted to think that Caroline was going to make it easy for me by writing the book I wanted to have:

But as she started describing it, I was surprised to learn that the book she intended to write (or at least the book she wanted to write) was:

I was sure she had to be kidding, and I hastened to point out that the “talking people out of doing a survey” book probably had far less sales potential than the one I was hoping for. But I soon learned she was only half kidding, because when people asked for her help doing a survey, she often ended up feeling that “Don’t do a survey” was the best advice she could give them, for two reasons:

1. She’d seen far too many surveys done wrong, and more importantly
2. She’d seen far too many that shouldn’t have been done at all.

Fortunately for us, she ended up writing the book that I wanted. But it took her more than ten years of very hard work. Here’s what she did:

• Read a staggering number of survey books and research papers (which we don’t have time to do).
• Absorbed and understood it all (which we probably couldn’t do even if we had the time).
• Attended lots of survey conferences and workshops, and talked to experts and non-experts to learn about their best (and worst) survey practices.
• Combined it all with her own practical experience doing surveys and helping others do them.
• Boiled it down into understandable advice (which she happens to be really good at).
• Made it into a book (which, having written books myself, I strongly advise against, since it’s a ridiculous amount of work, especially if you do it well).

Like me, I’m sure you’ve been on the receiving end of plenty of bad surveys. Personally, the thing that bothers me most is when I want to answer “Other” or “N/A,” but they don’t let me. For example, how do I answer this question about the place where I’ve gotten my morning coffee every day for years?

If I answer 1, it makes my friendly baristas look bad, and if I answer 5, it’s a lie. The people who did the survey are forcing me to give them bad data, which means they’re going to get inaccurate results, so why should I spend my time helping them? I always fill it out anyway, and answer 5 because my baristas are very nice people, and I don’t want them to lose any points. But I’m never happy doing it.

In the pages ahead, you’ll find that Caroline talks about how to deal with many, many specific issues, like having an “Other” option. But even more valuable is the clarity she brings to big-picture issues, like what kinds of questions to ask (and how to ask them), what kinds of people to ask (and how many of them), and what to do with the results you get.

Thanks to what I learned from reading several drafts of this book over the years, I’m very comfortable doing what she calls a Light Touch Survey: highly focused, just a few questions, where the results you get will help you make an important decision. In fact, I just did my latest one a month ago to answer a few questions about how people use my website, before finally redesigning it after 20 years.

So if you’ve ever wanted to do a survey—or you want to do better surveys—you’re in the right place.

Just be glad that Caroline did so much of the heavy lifting for us.

—Steve Krug author of Don’t Make Me Think

The Rosenfeld Review

Chapter 1 — Goals: Establish Your Goals for the Survey

These common questions and their short answers are taken from Caroline Jarrett’s book Surveys That Work: A Practical Guide for Designing Better Surveys. You can find longer answers to each in your copy of the book, either printed or digital version.

1. I see so many bad surveys—isn’t the best survey the one that’s not done at all?
   Unfortunately, we are all bombarded with bad surveys. For example, someone in an organization decides that constantly blasting out questionnaires to every customer is a great way to get feedback. Their response rate is terrible, but they don’t consider that this poor response will simply create lots of errors—and annoyed customers. And since these bad questionnaires go to everyone, you’ve got a very good chance of seeing too many questionnaires—and many of them will be rotten ones. A bad survey gets you bad data. A bad application of any method gets you bad data.
2. What’s the best survey tool?
   Survey tools change constantly, so I’ve learned not to make any specific recommendations. But I have written Spotlight F, “Questions to Ask When You Choose a Survey Tool,” that has the crucial questions to ask when you’re picking one.
3. When I’m using a Likert scale, how many response points are best?
   If you want a quick answer for the number of response points, 5 is good. If you want a more complex answer, skip to Figure H.9, which is a flowchart to help you decide on the number of response points. And if you want my reasons for those answers, there’s Spotlight H, “’On a Scale from 1 to 5’ (Likert and Rating Scales).”
4. You’ve included a Survey Octopus with tentacles and a smile—don’t you know that’s all wrong for octopuses?
   You’ll meet the Survey Octopus in the Introduction—it’s a cartoonish representation of Total Survey Error. It’s not a real octopus: they have mouths between their arms and no tentacles. My favorite feature is their blue blood.
5. I got sent this terrible survey—please can I send it to you?
   Of course! I’m always glad to add more examples to my stash. If sharing the pain with me will help, feel welcome—but I won’t be able to do anything about it. You’ll find my contact details on my website: Effortmark.co.uk.
6. Is there any chance that I can persuade someone who sends me a bad survey to do something else?
   Yes, contact the person or organization who sent it to you and ask them to buy this book.