Augmented Reality Goes Mobile

The technology for meshing digital data with actual images, or augmented reality, makes a jump to smartphones, and the market is taking off

It was the shake heard 'round the world. On Aug. 27, 2009, überblogger Robert Scoble uncovered a hidden software feature buried within an iPhone application that provides access to reviews.

The secret, it turns out, was that users needed to shake the phone to activate the capability, known as Monocle. In the wake of Scoble's discovery, shared publicly via FriendFeed, iPhone users far and wide could be seen shaking their iPhones to get access to the new feature. Yet the frenzy was about more than the novelty of how to open it, or even the trove of reviews.

The bigger prize was in how that information shows up on the phone. Once Monocle is activated, users looking through the iPhone camera can see reviews and other information about restaurants, stores, and other businesses in the direction the camera is pointing. Monocle was one of the first smartphone applications in the U.S. to use a technology known as augmented reality, which meshes digital information with actual images of the subject of that data. For many, augmented reality evokes images of what the Terminator sees as he homes in on a potential target, or the real time data seen by Luke Skywalker as he scans the barren Tatooine desertscape through a pair of field goggles.

A High-End Start

Augmented reality, also known by the acronym AR, has been around for at least two decades, but it has been largely relegated to applications in key areas such as training and inspection in automotive and aerospace manufacturing. In 1992, Tom Caudell coined the term augmented reality when he was working at Boeing (BA) on a project to make it easier to assemble large bundles of electric wire for aircraft on the factory floor. But recently, the technology has started to jump from high-end expensive equipment to Web cams and mobile devices.

The market for AR applications on smartphones is so new that it has gone from virtually no users in 2008 to an expected 600,000 by the end of 2009, says Christine Perey, a principal at Perey Research & Consulting, who advises companies on mobile augmented reality. Mobile AR is especially popular in places such as Europe, Japan, and Korea. Much of the activity in the U.S. has happened since August, following the June release of the iPhone 3GS, which includes technology that helps determine the user's location—a key ingredient in AR apps. In the future, some augmented reality applications will come pre-loaded on smartphones, further boosting the number of people who have access to it, Perey says.

By 2012 there will be 150 million to 200 million users, she estimates. That would make up only about 3% of the world's mobile-user base but still a high percentage of smartphone users. "In the next couple years, we see augmented reality crossing over from being a rather niche technology for military, medical, and heavy-industry-type applications to being a much more mainstream set of applications in the consumer market," says Jackie Fenn, vice-president at Gartner (IT). In fact, Gartner has said that augmented reality will be one of the top 10 disruptive technologies from 2008 through 2012.

AR on the Yelp app traces its origin to a self-fulfilling prophecy. During the summer, Scoble remarked online that he'd heard a rumor Yelp was working on augmented reality. At the time it wasn't true, but after the rumor took fire, a Yelp intern lobbied to get the project off the ground, says Eric Singley, product manager for Yelp iPhone. "So we gave him license to work on it," Singley says. It took Ben Newhouse, a 21-year-old Stanford student, about a month to develop Monocle for the Yelp iPhone app.

Smarter Smartphones

The pace of mobile augmented reality development has quickened as new smartphones such as the iPhone 3GS and devices sporting the Google-backed (GOOG) Android operating system now come equipped with the essential components for mobile augmented reality: a global positioning system, a compass, and a camera. Another necessary ingredient is content that has been tagged with geographic information such as latitude and longitude, so that others can find it when they are in the same location. Wikipedia, Flickr (YHOO), and Brightkite are among companies that provide so-called geotagging.

Augmented reality is difficult to do well, and Singley and Newhouse spent a lot of time polishing Monocle. "There's a lot of advanced mathematics involved in getting augmented reality right," Singley says. Early iterations showed information that was too far from the user, for instance. But feedback to the version released to the public in late August was so positive that Yelp decided to turn the feature on for everyone on Sept. 11. Shaking the iPhone is no longer required. Now users can access Monocle by pushing a button in the top corner of the Yelp app.

AR apps for the iPhone still leave something to be desired. During recent testing, the Yelp augmented reality feature showed an establishment that was far from the user while missing one that was closer—even though both are reviewed on Urbanspoon, another iPhone app with restaurant reviews that recently added augmented reality, in some cases is more precise.

Developers are working hard to improve what's known as registration, where the virtual data is more tightly associated with the objects consumers see through their smartphone cameras. Right now applications such as Yelp and Urbanspoon estimate where users are located based on the phone's compass and GPS information and will serve up restaurant reviews based on that data, as a kind of overlay to the image consumers see in the camera.

Not Enough Access

To make the apps work more accurately, developers need better access to the mobile hardware, says Tish Shute, an independent consultant who blogs about augmented and virtual reality. "Apple is not yet giving developers access to the iPhone's video application programming interface," she says.

Augmented reality features have application far beyond the consumer wireless sphere. Soldiers in Iraq use see-through displays from Microvision (MVIS) that look like clear glasses and can keep them connected to real-time digital battlefield information, such as enemy troop movements.

That same technology may find practical uses in a corporate setting. CEOs, for instance, may someday wear specially equipped glasses that function much like a teleprompter. Such eyewear, connected to Facebook or LinkedIn, could also help the wearer associate a name and other information with a person's face.

Businesses are shifting strategies to better accommodate expected demand for consumer-friendly apps. Metaio, a German AR company founded in Munich in 2003, began with highly specialized industrial applications such as service and maintenance training. CEO Thomas Alt co-founded Metaio after he had worked for several years researching augmented reality at BMW (DB:BMW).

More Mobile Interest

In the past year, Metaio has worked with LEGO Group to install kiosks in several retail outlets that can help kids see what the LEGOs will look like once they're assembled. The company is also increasingly interested in developing mobile applications. On Nov. 2, Metaio released an iPhone app called Junaio that lets consumers see location-based content through the phone's camera and lets them easily add 3D images to photos. "From a development point of view, the strategy internally is now to develop for mobile platforms first," Alt says.

Total Immersion has done 3D simulation for companies such as Boeing, BMW, Peugot, and Renault. In September, Total Immersion signed an exclusive distribution partnership with int13, a French software firm that specializes in next-generation smartphone applications. The aim is to create apps that integrate real-time interactive 3D graphics into a live video stream. The first application will be developed for Symbian and Microsoft (MSFT) Windows Mobile devices, with iPhone and Android support following.

Developers are also working on another kind of mobile augmented reality application: AR browsers, such as SPRXmobile's Layar and Mobilizy's Wikitude, which essentially pick up geo-tagged content such as information from Wikipedia or, say, real estate listings. The risk with browsers is that it's easy to be inundated with tagged information, and it might be difficult to wade through the information. "The advantage of the Layar browser is that it lets you do filtering by picking which layer of reality you want to see," says software developer William Hurley, who goes by the name Whurley. The Layar browser, which was first released for Android devices in June and the iPhone in October, has 176 so-called layers in categories ranging from real estate, health care, and transportation to entertainment, tourism, and social networks. Yet Whurley says there will be a general need for content-management tools to simplify the AR experience.

Singley at Yelp says the company has more improvements in the works. It is planning another release of its application for the iPhone by the end of the year. "As iPhone technology gets better, augmented reality will get better too," he says.