A recent spate of game-changing applications for mobile devices have been emerging over the past few weeks and months. Their defining characteristic? Real-time global positioning providing helpful data overlays, or so-called Augmented Reality displays.
It's great to see excitement building for the Apple iPhone (launch expected late June '07). Korea's LG Prada, an iPhone competitor product, launched a few months ago in the UK.
Just this week a new touch screen device- Taiwan's HTC Touch was announced for release immediately in the UK and later this year in the US.
The hot attraction of course with the iPhone, the Prada and the Touch is that they offer touch and gesture-only user interaction. Clarification: Only the Apple iPhone offers gesture-like pinching interaction (eg. open-close; zoom in-out).
What's great about the excitement around these new phones is that they set a new precedent in communication device design for 2008-2010. They are, if you will, a generation up from the slide cover keyboard phones. These devices are hip, keyboard-free, MP3 and mobile Web-friendly.
Watch these demos if you are not familiar with how these new mobile devices work:
Bruce "Tog" Tognazzini offered an early usability review of iPhone features, worth a read, but as one of our consultants pointed out- "how can you do a usability review of a device only using a demo from Apple's website? Is that a real usability evaluation?".
What is Multi-modal interaction?
Multi-modal interaction is an area of Human Computer Interaction (HCI) that has a long history of usability research and empirical study. The virtual reality and game design research community have been studying this commercially since the early 1980's with pioneering researchers such as Brenda Laurel at Atari.
Today researchers like Jeff Han, see images left, are meshing with industrial efforts to bring multi-modal
interaction to life. Last week, Microsoft announcedSurface- an interactive coffee-table surface that responds to touch and gesture. Users can explore information linked to objects placed on the desktop (TMobile will use it in stores to encourage handset purchase decision support).
Also, if you haven't seen Jeff Han's TED conference presentation, it's an amazing must see
demo of his Surface-type work-space.
Why multi-modal interaction design?
Multi-modal design brings the spirit of HCI to life by harnessing the rich sensory input afforded by the human body-mind (touch, gesture, sight, sound, voice...smell, well not yet, but it's in the works).
As interface designers, we have really had to compromise with the flat and lifeless limitations of desktop PC's (windows, icons, menus) compared to the original vision of how humans should use computers advocated in the late 1960's:
Computer graphics and interface pioneer Ivan Sutherland told us our computers should not be mere 2D screens that provide information, but instead they should be 'windows upon which we look into a virtual world...where we can see, hear and feel' multi-sensory information.
I began studying multi-modal interaction ten years ago as part of my early virtual reality research. It's an area of interface design that is truly fascinating for it's potential. It's also an area of interface design that is challenging due to the context in which the user is interacting. As a designer the questions become:
"Which sensory pathway does the user have available to complete this goal in that context?"
"Which sensory system is the lead, which is the secondary?"
"How much sensory over-lap is available, tangible, appropriate?"
"How do users back out or recover from a screen event- in a dynamically changing physical environment?"
Does gesture and touch interaction work in other contexts...like cars?
For the past five years I have closely followed the emerging 'Internet in your car' trends in the automotive industry (aka telematics usability). A practical example I can share, which I've written about and spoken about at telematics automotive conferences, is the case of GM's OnStar. Several years ago, GM provided me with a fully loaded Cadillac CTS for a week that I used to evaluate the OnStar, a speech system for help, navigation, communication and additional information.
OnStar weighted it's user interface toward "voice" or speech interaction over a multi-modal interface. The result was a clunky system with a history of poor user adoption and satisfaction. In 2001, 60% of Onstar systems were switched off in the owner's vehicles. BMW on the other hand, weighted it's iDrive telematics solution to a knob-like control (tactile interaction) with 700 features in menus at the turn of a dial.
The result: eroded brand loyalty, confused and frustrated customers (including usability guru and BMW customer, Jakob Nielsen). Jakob Nielsen's wife said at the time she would never buy another BMW again...
The pattern in the design flaws for telematics human factors engineers?
Don't put all your "eggs in one basket" with regard to one modality.
It appears that neither GM nor BMW provided adequate multi-modal support, opting for a "lead" sensory system (speech, touch) over a mixed system.
I believe multi-modal is generally always better than singular modality (as an interaction design technique). But you must be careful if you are designing for multi-modal interfaces, as Oregon Graduate Institute Professor Sharon Oviatt reminds us in her Ten Myths of Multi-modal Interaction (PDF).
Consumers can feel poor usability with great intensity on a cell phone or with a new data enabled device-- unlike a website, software application or operating system that involves more relaxed cognitive states (sitting down and learning the system). When you need to make a call, send a text message or web browse on a handset it becomes obvious what ease of use is and isn't. A large part of the design problem in mobile user experience is user environment (mental, physical and social). Think about the difference between sitting down with a cup of coffee watching windows pop up, pages move around and your ability to close windows, back up with the browser etc. Now think of holding a communication device and interacting with 1.5 inch screens while standing up, walking around or in the social presence of others.
A recent survey by MIT found that adults in the US rated cellphones the top of the list in technologies they hated but couldn't live without:
"An annual Massachusetts Institute of Technology survey, known as the Lemelson-MIT Invention Index, found that among adults asked what invention they hate most but can't live without, 30 percent said the cell phone.
Alarm clocks were a close second, with 25 percent, followed by the television with 23 percent and razors with 14 percent. Microwave ovens, computers and answering machines also earned spots as detested technology".
John Quain at e-week believes the popularity problems wireless technologies have been having in the US in the past few years (compared to Europe and Japan for instance) have to do with desirability.
"Part of the reason again is the added expense in a competitive market, but the main reason is simply that consumers don't think they need it".
The need to target actual user behavior in design requirements. Field testing is essential for devices that are deeply embedded in the users physical and social context. New devices require vigorous and frequent usability tests and fluid user interface specifications that can morph to changes based on new user needs that emerge during your usability test(s). This is especially true for localization of services when introducing new products into new markets.
Mobile devices also need to support "user-compatible" configuration.Mitch Kapor (founder of Lotus Notes) explained his frustration with hitting a dead-end when trying to configure the email settings on his new Nokia 3650 cell phone. His story serves as an important reminder of the need to make settings configuration "user- compatible". More on Configuration Usability later...