Augmented/Mixed Reality: will it stick?

This article is about Augmented Reality (AR) and Mixed Reality (MR). They both have an issue with fidgeting virtual objects. VR might also be affected, but there is no easy way to check.

If technologies like AR, MR and VR confuse you, please consider reading this article.

AR and MR add virtual objects to the real world. A ballerina dancing on your desk, robo-worms coming out of your wall, or a dissected heart on your plate. You only need some gadgets to see these virtual objects. AR uses flat screens like tablets or mobile phones. Pokemon Go is a good example for this. MR provides an immersive experience but you need smart glasses for that.

The problems is that the virtual objects tend to forget their place. They fidget a little. A fraction of an inch at the time, but this can add up to feet over a longer period. This is still cool for the ballerina or a game, but can prevent industrial application of AR and MR. The stool on the below video shows exactly what I am talking about.

I have noticed this issue with the following AR/MR platforms: Apple ARKit, Google ARCore and Microsoft HoloLens. I have no access to Magic Leap yet, but the video below looks promising (although the holograms seem to be trembling).

So why don't virtual objects stick to real world coordinates? Spatial mapping and/or visual tracking seem to be the reason. What is spacial mapping? It is a method for the AR/MR devices for seeing the real world and creating cool videos like the one below.

So AR/MR devices use optical cameras and depth cameras to come up with all those triangles. This is how they see the real world. Then the gadgets are tracking our eyes. "Tracking works out where we are looking, so that graphics can be draw in the right place." Unfortunately, the triangles and the tracking seem to loose some detail, hence the instability. See an interesting discussion here.

So what's next? Will the virtual objects ever stick? Will HoloLens 2 fix this issue? Has Magic Leap already solved it? Please share your thoughts in the comment section!

Disclosure: the author is a Mixed Reality consultant at 360World, an agency specialized in AR and MR solutions.