What Is Augmented Reality?

Augmented reality is a view of the real world interlaced with computer-generated elements. These elements serve to “augment” the viewing experience via the virtual addition and/or removal of 3D objects, images, sound, and overlays. The term “Augmented Reality” was coined by a Boeing researcher named Tom Caudell in 1990 (Berryman).

Content is delivered by a wearable or mobile device that processes real-world input and renders the augmented view in real-time. The information provided by the device is often very context-sensitive and interactive, combining input from a variety of sensors (Barfield). The level of scene interactivity can be immensely increased with the addition of computer vision and object recognition libraries.

While most AR experiences bring virtual information to the user’s immediate surroundings, this is not a requirement. Furht writes that AR can be applied to “any indirect view of the real-world environment, such as a live-video stream.” Furthermore, AR is not necessarily restricted to the sense of sight. While visual AR is by far the most popular, any type of sensory input is eligible for augmentation.

The State of the Industry  (late 2016)

AR is still very much in its infancy. While some enterprise and military applications have been in use since the early 2000s, commercial AR is only now coming to fruition. The first case of a large-scale commercial AR headset was the release of Google Glass and the Glass Explorer Program in 2013.

Four years later, Google and Apple are just now releasing AR SDKs to developers. Until two days ago at the time of writing, Google did not have an AR SDK available to the public, and Apple’s ARKit was released a mere six months ago. Many more small-scale ventures are being developed as well, but the state of the industry as a whole is comparable to the mobile phone market in 2006 (read: pre-iPhone).

Merel writes that the number one challenge in achieving a viable AR market is developing a “hero” device capable of processing and rendering AR while maintaining all-day battery life. Simply put, consumers are open to AR, but would prefer an AR experience that did not require looking through one’s smartphone camera.

While computer engineers are working hard on creating such a hero device, the majority of AR development will target mobile phones. The recent release of the aforementioned SDKs has opened many doors for software developers and consumers alike.

Applications

The use cases of augmented reality are as unlimited as the amount of 3D space around us. Many different applications are under development, including cooking tutorials, games, location-based apps, and virtual menus.

The most popular augmented reality app on the market is Niantic’s Pokémon Go. While Pokémon Go’s AR component is extremely basic, it is a landmark for the AR industry. Even before the release of ARKit, Pokémon Go’s AR was widely accepted by the public, and the game still has a user base of 65 million monthly active users today.

View a list of sample ARKit applications here.

ARKit

According the Apple Developer documentation, ARKit is “a new framework that allows you to easily create unparalleled augmented reality experiences for iPhone and iPad.” Because of the level of performance required for real-time AR, ARKit is restricted to devices equipped with an A9 processor or above. By combining data from the CoreMotion sensors (accelerometer, gyroscope, pedometer, altimeter, and magnetometer) and the camera, ARKit can track the device’s position and accurately scan the camera scene in front of the device. This method of position tracking is called Visual Inertial Odometry (VIO).

VIO allows the device to detect horizontal planes, place and anchor virtual objects in the scene, and estimate the scene’s lighting and apply a similar amount of lighting to virtual objects. Unfortunately, light detection will not automatically apply shadows to virtual objects at the time of writing.

On iPhone X and above, ARKit is able to detect the “position, topology, and expression of the user’s face… in real time” using the TrueDepth camera. By creating a 3D map of the user’s face, ARKit can apply live “selfie” effects or read certain facial expressions. Facial expressions can be used as controls, mimicked by a 3D character, or utilized in other creative ways.

ARKit works with additional Apple and third-party technologies. Metal, SceneKit, Unity, and Unreal Engine already provide optimizations for ARKit. For example, SceneKit scenes are mapped to the real-world scene by ARKit, so placing an object into a view’s SceneKit scene, the object appears to be anchored to real-world coordinates.

Works Cited

ARKit - Apple Developer. (2017). Developer.apple.com. Retrieved 12 September 2017, from https://developer.apple.com/arkit/

Borko Furht (31 August 2011). Handbook of Augmented Reality. Springer Science & Business Media. pp. 3–. ISBN 978-1-4614-0064-6.

Berryman, D. R.. (2012). Augmented reality: a review. Medical Reference Services Quarterly, 31(2)

Merel, T. (2017). The reality of VR/AR growth. TechCrunch. Retrieved 1 September 2017, from TechCrunch

Woodrow Barfield (29 July 2015). Fundamentals of Wearable Computers and Augmented Reality, Second Edition. CRC Press. pp. 1–. ISBN 978-1-4822-4351-2.