Design Beyond Devices

The future is multimodal because we are multimodal. Here is an essential guide to unleashing the potential within that symmetry.

—Bill Buxton, Partner Researcher at Microsoft Research and author of Sketching User Experiences

This book harnesses our very selfish desire to make cool things into the ability to benefit clients, stakeholders, customers, and society itself. Chock-full of insightful conceptual models and practical applications, designers will be both inspired and prepared.

—Sam Ladner, PhD, author of Practical Ethnography: A Guide to Doing Ethnography in the Private Sector and Mixed Methods: A Short Guide to Applied Mixed Methods

A must-read for anyone wanting to understand the richness and complexity of how users move fluidly between different forms of input, and how to ethically elevate that experience. The attention Platz has given to a manageable structure for ethical reasoning around the problems this tech should be solving is outstanding.

—Per Axbom, co-host of UX Podcast and author of Digital Compassion

Using simple frameworks and questions, Platz creates a practical playbook that challenges teams to understand the problem, reflect on the design constraints, focus on people, and avoid the traps of exclusion in a multimodal world.

—Jose Coronado, Vice President of Global Design Operations, JPMorgan 

Interface design may seem like a solved problem. However, fifteen minutes with this book will show you how many exciting design problems you still have left to solve. Your inseparable guide to multimodal design.

 

—Andy Budd, founder of ClearLeft and organizer of UX London

Chapter 1: Creating the World We Want to Live In
Chapter 2: Capturing Customer Context
Chapter 3: Understanding Busy Humans
Chapter 4: Activity, Interrupted
Chapter 5: The Language of Devices
Chapter 6: Expressing Intent
Chapter 7: The Spectrum of Multimodality
Chapter 8: It’s a (Multimodal) Trap!
Chapter 9: Lost in Transition
Chapter 10: Let’s Get Proactive
Chapter 11: Breathe Life into the Unknown
Chapter 12: From Envisioning to Execution
Chapter 13: Beyond Devices: Human + AI Collaboration
Chapter 14: Beyond Reality: XR, VR, MR, AR
Chapter 15: Should You Build It?

Humans are—as far as we know—the only creatures on Earth capable of communicating the same message in multiple modalities, switching at will depending on context and abilities. Other animals sing, snarl, and stridulate (that thing crickets do), but each species has its set of modes, and each of those is constrained to a range of meanings. A blue-footed booby can’t hum its way into the heart of a mate unable to see the kinetic output of its distinctive dance. Vervet monkeys emit a different warning call for each of their four major predators, but lack the same options for a silent signal when danger approaches. 

We, however, hit the multimodal jackpot. To convey the deeply human message of your love for someone, you may choose to speak aloud, mouth the words, sign the phrase with a single hand, send a GIF, or nudge an arm, based on what’s most comfortable for you and intelligible to your intended audience. In a slightly different situation, you might choose a different way. 

But wow, this gets complicated once we bring digital systems into the mix. No wonder we’ve tried to put off dealing with the full implications, hoping partial solutions like “mobile first” and “conversational UI” could stand in for fully human-centered design. The truth is that focusing on a single device or mode at a time actually leaves a lot of humans behind. 

Fortunately, Cheryl is here, not only as an expert and compassionate guide through this new territory, but also to hold us to task. She reminds us that in order to include everyone, we need to question everything. She shares her enthusiasm for technology grounded in the recognition that we all remain people with bodies—a fact that can appear either awkward or inconvenient to designers and technologists. 

This book represents the essential next evolution of design for digital systems, integrating everything that has come before and making the principles and process clear, regardless of the reader’s previous area of practice. Even if you aren’t going fully device-independent and multimodal in your work immediately, you will come away better equipped to accommodate the incessant interruptions and device switching that are just a part of daily life with technology now. Your work will be more gracious and humane.

As Cheryl explains in one of several case studies, simply offering a single additional input modality can offer an individual agency over their environment that they lacked before. We toss the word “empowerment” around a lot, but this is it.

Humanity is shared, but each of us is unique in our current context and our capabilities. Read this book. Rise to the challenge issued within. And you can help ensure that everyone can fully participate in the promise of our future. If only we could also wag our tails. 

—Erika Hall, Director of Strategy, Mule Design Studio

These common questions and their short answers are taken from Cheryl Platz’s book Design Beyond Devices: Creating Multimodal, Cross-Device Experiences. You can find longer answers to each in your copy of the book, either printed or digital version.

1. What exactly is multimodal design, anyway?
   We define it in Chapter 1, “Creating the World We Want to Live In,” but think of it this way: multimodal experiences are experiences that can engage multiple human senses. Devices that let you speak or touch to make a selection. Twenty years ago, your PC (generally) only engaged you visually, and you only engaged your PC with touch. You have many more options today (see Chapter 7, “The Spectrum of Multimodality,” to fully understand all those options), but those options bring a lot more complexity to the design party [see Chapter 8, “It’s a (Multimodal) Trap!”].
2. What’s so hard about multimodal design?
   One of the biggest challenges is the introduction of invisible inputs like voice and gesture, which complicate both the delivery of designs and the interactions themselves. Many of the existing deliverable standards don’t scale to systems with multiple inputs and outputs, but in Chapter 12, “From Envisioning to Execution,” you’ll see concrete examples of how to transform your complex ideas into tangible designs. As these devices become ubiquitous, interruptions are also a growing challenge. Chapter 4, “Activity, Interrupted,” and Chapter 10, “Let’s Get Proactive,” will help you design predictable, responsible proactive systems.
3. Is this book about artificial intelligence?
   This book is not solely about artificial intelligence (AI). But modes of interaction like speech and gesture are inherently driven by artificial intelligence, so the topic is unavoidable, even for designers. You’ll learn about specific types of artificial intelligence driven input and output in Chapter 5, “The Language of Devices,” and Chapter 6, “Expressing Intent.” Chapter 13, “Beyond Devices: Human + AI Collaboration,” is entirely devoted to artificial intelligence as a concept—how it works, the potential biases, and the ways in which it can most effectively be deployed in your experiences.
4. Does this book cover accessibility and inclusive design?
   Rather than lock accessibility into its own chapter, inclusion has been woven into the fabric of this book. In Chapter 1, you’ll learn more about some of the concerns that have been raised in recent years about the design community’s relationship with the disability community and new ways of thinking to address those concerns. You’ll find content about the potential risks of exclusion—as well as opportunities for inclusion—all throughout Chapter 5 and Chapter 6. And where appropriate, additional content is included throughout the rest of the book. Finally, Chapter 15, “Should You Build It,” introduces a new lightweight framework for querying the potential impact of your work, good or bad.

The Informed Life

Presentable

The Rosenfeld Review

UXPodcast

Chapter 7: The Spectrum of Multimodality