What's next in mobile user experience? Augmented Reality

A recent spate of game-changing applications for mobile devices have been emerging over the past few weeks and months. Their defining characteristic? Real-time global positioning providing helpful data overlays, or so-called Augmented Reality displays.

What is Augmented Reality?

Think of Augmented Reality as half virtual and half real. It takes existing physical spaces and supplements or "augments" them with information, objects, actions or interactions. One of the most extensive uses of Augmented or "Artificial Reality" involved interactive art installations pioneered by Myron Krueger, author of the ground breaking books Artificial Reality I and Artificial Reality II.

Put the capabilities of mobile phone GPS and real-time points of interest on a live camera view with data overlay and you have the next big thing (potentially helpful to users) in mobile user experience.

Examples of Augmented Reality:

1. Location based Restaurant finder with reviews

Yelp was one of the first well known sites to go live with Augmented Reality.

2. Location based Travel points of interest

Wikitude AR Travel Guide offers travel based historical context and place information.

3. Virtual pet for the iPhone

ARf is an Augmented Reality Virtual Pet on the iPhone.

4. Public transportation schedule and station finder

The London Underground Tube Map finder provides station way-finding and travel information.

 5. Virtual objects you can interact with in the real world

Gizmodo- an AR toolkit for creating virtual objects. Gizmodo has not released yet due to low frames per second processing (10 fps vs 20-30 fps)- this will change once Apple release a video API for the iPhone. This will allow 3D games and fast Augmented Reality applications to run without delay.

Drawbacks of Augmented Reality

• Current performance levels (speed) on today's 2009 iPhone or similar touch devices like the Google G1 will take a few generations to make Augmented Reality feasible as a general interface technique accessible to the general public.
• Content may obscure or narrow a users interests or tastes. For example, knowing where McDonald's or Starbucks is in Paris or Rome might not interest users as much as "off the beaten track information" that you might seek out in travel experiences such as those our client RentVillas.com offers. 
• Privacy control will become a bigger issue than with today's information saturation levels. (See Privacy- your biggest user experience challenge). Walking up to a stranger or a group of people might reveal status, thoughts (Tweets), or other information that usually comes with an introduction, might cause unwarranted breaches of privacy.

Benefits of Augmented Reality

• Augmented Reality is set to revolutionize the mobile user experience as did gesture and touch (multi-modal interaction) in mobile phones. See Multi-modal design: Gesture, Touch and Mobile devices... This will redefine the mobile user experience for the next generation making mobile search invisible and reduce search effort for users.
• Augmented Reality, like multi-modal interaction (gestural interfaces) has a long history of usability research, analysis and experimentation (particularly in Automotive Telematics research) and therefore has a solid history as an interface technique.
• Augmented Reality improves mobile usability by acting as the interface itself, requiring little interaction (this Interaction Design technique is known as Direct Manipulation). Imagine turning on your phone or pressing a button where the space, people, objects around you are "sensed" by your mobile device- giving you location based or context sensitive information on the fly. See my musing on where this might go, written in 2004: " Would you like a Pop-Up with that Traffic Jam?"

Augmented Reality for mobile phones is an exciting development and has the power to take the mobile user experience to the next level, offering users the value proposition they have long been waiting for on mobile phones and devices: helpful, simple, convenient just-in-time information and services. 

Best Wishes,
Frank Spillers, MS

Comments

[Frank Spillers]

Very interesting Manoj- thanks for sharing. Challenges outlined in your deck are helpful. I didn't see your case studies. I would also like to know more about the localization factor you see for India. Is it at the infrastructure level or are we talking content and context as well?

Frank

[ManojKandasamy]

Hi Frank,

Your article is quite interesting. And thank you for providing excellent reference. I'm working for a mobile solutions company based out of Bangalore, India. Though AR has tremendous potential in engagement, will it scale up since its limited to few hi-end mobile phones? We believe if we innovate AR and make it available to all users, it will provide much more value for both brands and end users.
We have created a customized AR app called intARact for Indian market. Please find few details about our product in http://telibrahmaindia.blogspot.com/2009/12/intarct-ideal-solution-to-make.html

You can also go through our AR campaigns for NIKE India and HSBC India in our blog. Let us know what you think about it..

Good day,
Manoj Kandasamy.

[joelamantia]

Under the "Benefits of AR" section, you mention that it requires little interaction. I'm thinking you mean little interaction with the device that's mediating your AR experience?

Even with the limited set of interaction patterns driving AR experiences for the moment, there is a great deal of physical and social interaction that happens in relation to / with the real world around you while you're having a small set of interactions (say, not much control usage, in conventional usability / hci schemes) with your AR enabled device.

This shift in the context of the interactions presents tremendous challenges for all parties (designers, users, developers, etc.), but it's also absolutely fabulous, b/c it's indisputable evidence that the experience is no longer confined to the interface. Mobile AR has completely shattered the context barrier.

Or, at least, that's how I'm seeing it so far...

[twitter.com/floozyspeak]

Nice summary on AR today. As a long time follower of the AR trend I'm a bit baffled as of late in how much sheer OMFG is happening around AR. All of this really due to folks coding up interesting hacks and ideas on Android and the iPhone.

I see more cons on the horizon then pros sadly. I think AR will be mass gimmick ware for a good 1-2 years, amassing 50+ apps that all can do something supposedly truly amazing with an AR stint to them, but like most innovation on the web, true stickyness to applicable need/want/desire is short lived. This list will get paired down to 5 or less and those will battle it out fiercely for consumer attention and real behavior shift/adoption.

In some ways AR's experience is going to parallel LBS (location based services) experience on the app scene. Dozens if not hundreds of apps can tell you where the nearest starbucks is in your hood- but odds are, you probably sorta knew already. Its not to say LBS isn't useful, it is, just when ya need it. But I also fear at times, that these new technology trends make us dumber in that we don't give our own sense of direction and general awareness powers credit.

The other big con on for AR is the layer scene. There will be dozens of competing layers of AR related data. Many will compete to get us all on the same platform of data. Clearly I see players like Google having the most to win, and they are basically watching the scene whip out a few thousand ideas around AR and looking to see what will stick, and then they just have follow along and turn it on. Having Google Maps and all their massive info all knowing structure is huge, yet you don't see them festering about the AR scene. They too are waiting to see what a few thousand people make, see what real users use and adopt, and then, they'll get in on it.

So for now, we'll have alot of AR gimmick soft, cool ideas, hacks and wonder stuff, thrown together to make us dream of the flying car of the future and more. Hopefully some startups will be mindful of Google's all observing eye and focus on real applicable consumer need and experience. The biggest gain you could have is building something that really sticks, showing everyone, and then getting acquired.