Apple's got big plans for augmented reality, but the company's iPhone X reveal was a missed opportunity.
Apple was supposed to blow us all away with its vision for the future of augmented reality. And what did we get? Animated poop emoji.
Did Apple drop the ball?
Three months ago, Apple revealed ARKit — a toolkit for building augmented reality applications, to overlay digital content into the world around us. It's a tremendous piece of technology and since its release, we’ve seen hundreds of developers experiment with ARKit.
For the first time, AR applications became both easy to create and a pleasure to use. Why then, was this technological and strategic advantage so massively undersold?
In the tenth anniversary year of the iPhone, in the first presentation hosted in the Steve Jobs Theatre, I was expecting Tim Cook to show what Pokémon Go could look like if it were combined with Apple’s augmented reality technology. However, the only demo we saw was a tabletop game called The Machines which, whilst beautiful, didn’t deliver the fanfare many of us were looking forward to.
We saw nothing of the incredible work indie developers and established companies alike have been building with ARKit. We’ve already seen a demonstration of Pokémon Go using ARKit during their WWDC developer conference, so why hold back? Is it to keep ARKit in the developer community before enough applications have been created? Perhaps Apple is deliberately underselling AR in an effort to manage expectations? Or maybe the app creators decided not to reveal their new developments for other strategic reasons, like waiting out for Google’s new Pixel phone?
Either way, it seems like a loss of momentum from what originally looked so promising. Whilst Apple has big ambitions for AR, it’s clear from their last event that it will be a long road to get there. There are still a number of fundamental technological problems that need to be solved before this vision of the future can become a reality.
However, whilst a front-facing TrueDepth camera will undoubtedly improve the robustness of facial tracking today, it is unlikely to be necessary for AR's future. This is because developments in machine-learning mean devices are getting better and better at recognising what a face looks like and can infer depth automatically from an ordinary camera.
So why integrate it at all? Perhaps the TrueDepth infrared projector is not yet strong enough for mapping objects further than a couple of feet away from the phone without sacrificing battery power or getting too hot. Maybe it's a tactic for Apple’s machine-learning algorithms to learn how faces are shaped with more detail. Whilst TrueDepth is certainly more useful for FaceID (to ensure somebody isn’t simply holding up a photograph to trick the device into unlocking), it’s still unclear why a fingerprint sensor had to be removed in the first place and not integrated under the glass (which is admittedly still a technical challenge), or on the rear like on some Android phones.
Technical challenges for AR include: improvements to display and optics, shrinking hardware into something wearable, and improving the way devices perceive the world around them.
As someone working on the AR technologies, I see ARKit is a major step in the road towards an augmented reality future. It was just sad to see this achievement not receive the recognition it deserved.
One meaningful addition to the iPhone X was a front-facing 3D sensor, which Apple refers to as a ‘TrueDepth’ camera. Similar to the technology employed by Google’s Tango phones for the last year, this TrueDepth system projects infrared light outwards into the surroundings, which the phone uses to view the world in 3D. This technology is likely to have been influenced by Apple’s acquisition of a company called PrimeSense back in 2013.
Unlike Google’s Tango system, which uses this technology on the rear of the device for mapping environments, Apple placed its TrueDepth camera on the front, limiting the use cases to 3D-mapping faces rather than locations.
Apple demonstrated this with Animojis — animated emojis controlled by your facial expression. This too, is likely to have been influenced by Apple’s acquisition of another company, FaceShift, which was bought in 2015.
I would have loved a wider-angle camera on the back of iPhone X, allowing the device to display augmented reality content outside in large open areas more accurately and robustly than ARKit currently allows. I would have loved see a TrueDepth camera on the rear of the device, allowing the phone to map environments indoors in greater detail. I would have loved to see a demo using ARKit that shows the true power of the medium. But none of these points are why I was disappointed.
Five years ago, during Google’s 2012 developer conference, we saw a team of base jumpers hovering several hundred meters above San Francisco in a hot air balloon. They were all streaming live video to the conference centre via Google Glass (Google’s first attempt at an augmented reality display).
Having jumped out of the air balloon, the group flew across the skyline in wing suits before landing on the roof of the conference centre. After landing, the Glasses were handed to a team of stunt cyclists, who in turn handed the Glasses to a group of abseilers who ran them down the side of the building. Passing them on to yet another cyclist, the Glasses were finally delivered to the main stage to a rapturous applause.
It was the most exciting product demonstration I had ever seen.
Whilst Google Glass is deemed by many as a failed product, the demo highlighted a large company trying to do something ambitious, inspiring, even dangerous. It showcased a bold vision for the future, led by an incredible team who asked “What if?”. This is what was missing from Apple’s Event.
Apple is an incredible company, with beautiful products, inspiring foundations and a solid future. However, we all want to see Apple replicate what they once did for the mobile industry; that is, create a category-defining product that changes how people view the world. Perhaps next year Apple will show us the way?