Activity and Process

Early this June we conducted an expert interview with T. Scott Saponas at Microsoft Research. Scott’s expertise in Human Computer Interaction, specifically with experimental input and output technologies and always-available, context-aware computing, seemed like a useful resource for our project as we explore potential ways of interaction with media. The interview was a good way to quickly gain insight into the field of input and gesture technology and supplement what we found through secondary and some primary research that we previously did. We had several questions for him:

• What types of interactions are harder to do with 3D gestures and why?
• How do 3D gestures make certain interactions easier and why?
• What do you see as the future of 3D gesture interfaces?
• How might spatial sensing change our daily interactions with personal devices? Specifically, how might this change how we consume content on these devices?
• How do you think media content could be made context-aware?
• What media types would actually feel compelling to interact with or consume using new input techniques?
• What could 3D interactions with media give us that current media does not?
• What content could be created with context-aware input and interfaces?

Key Findings and Takeaway

Outlined below are some of the key insights we got from the interview.

Challenges of 3D Gestures

We got valuable insights into challenges and considerations in designing for systems that use 3D manipulation and gestures from this conversation. Interactions with 3D gestures are hard because we are used to the idea of direct manipulation with a mouse or touch screen. On the system’s end, it is hard to interpret intentions and separate gestures from other movements. Furthermore, for gesture input and 3D manipulation, the gulf of evaluation is harder to bridge than the gulf of execution. Immediate response and feedback on the input is very important. It becomes really hard when the feedback loop is broken or is very slow. A major advantage of 3D interaction is unanchored interaction. The fact that whatever you are using to manipulate is not physically anchored to the device opens up a lot of possibilities for both design and potential problems simultaneously.

Designing for 3D Gestures

The key is to design the system around an assumption that tasks will afford experimentation and that errors are inherent. The system needs to be designed for better error correction such that the possible errors are not that critical. The penalty for wrong/inaccurate input should be low. Responsiveness is another important thing to keep in mind.

Spatial Sensing and Future

The most powerful and transformative thing about spatial sensing is that it enables the ability to import personal space into a computer simulation system; capturing the real world and transferring it to others. Scott believes people will likely pay a lot for that and this will likely be a big change in sensor use on phones in the future. There are possible applications of such technology in retail, consulting businesses, and service design.

New Media Forms Enabled by Input Technology

Scott mentioned 3D manipulation and augmented reality are two compelling areas to explore. With new forms of interaction like depth cameras, precise control is going to be hard, but he believes these can be very engaging if you incorporate tactile interaction and enable physical manipulation.

How Content and Media Needs to Evolve

In Scott’s opinion, even though technology has changed a lot, the media we consume hasn’t evolved much in the last few years. There should be ways to interact with what we are consuming. Everything is currently linear. Future of media should be interactive and two - directional, not linear. A good, simple example is the ability to change the camera angle on a movie to discover more about the space and immerse yourself in it.

Implications for Design and Moving Forward

We got a great deal of insight out of our conversation with Scott. Among other things, Scott confirmed our hunch that new forms of interactive media could be very critical to affecting real change in the publishing space. Also, Scott pushed the idea that gesture interfaces can trade precision for exploration, which is a powerful restraint that can help us avoid gestural interface designs that require a level of accuracy that simply isn’t attainable.