The full story of Apple’s disruptive mixed reality headset has been revealed

Zoom / Joni Ive speaks on stage during the 2017 New Yorker TechFest in New York City.

A series of reports in The Information paint a detailed picture of the progress, policy, and problems facing Apple’s plan to develop a virtual, augmented, or hybrid headset since the initiative regained momentum in 2015.

Citing several people familiar with the product, including some who have worked on it directly, the reports describe a wills competition about the device’s direction. The confrontation was between Apple’s mixed reality product team (called “Technology Development Group”) and famous Apple designer Jony Ive and his industrial design team. The report highlights Apple’s direction for the device, which Bloomberg recently reported is about to launch.

They also claim that Apple CEO Tim Cook has been relatively far from a product compared to others like the iPhone, and that the Technology Development Group’s location in an office separate from Apple’s headquarters has been a source of problems and frustration.

Information sources say Apple’s mixed reality efforts began almost coincidentally when the company bought a German AR startup called Metaio to use some of its technology in Project Titan, its self-driving car project. There was another key moment when Apple hired AR/VR project team leader Mike Rockwell away from Dolby Laboratories. Beginning in 2015, Rockwell created a team that includes Peter Mayer, co-founder of Metaio and Apple Watch manager, Fletcher Rothkopf.

In 2016, several augmented reality demos were shown to Apple board members. In one, tiny Triceratops grew to life size before the eyes of board members. In another, a room has been transformed into an immersive lush environment. But the board of directors was not Rockwell and the company’s most important drawback. According to The Information, it was Ive who oversaw both the industrial design and human interface teams at Apple.

Eve and his crew have argued against a VR headset because they believe that virtual reality separates users from the people and the world around them, and that VR headsets look unfashionable. But the technology development group gained the support of the industrial design team by introducing a concept: an outward-facing screen on the front of the headset that projects images of the wearer’s facial expressions and eyes to people around them. The wearer can see the people around them through the external camera feed.

Rockwell and his colleagues developed and released ARKit in 2017, an app development kit that allowed developers to create AR apps for iPhone and iPad using technologies and techniques that could later be adapted to a headset.

Initially, Rockwell and the rest of the mixed reality team wanted the headset to be attached to a base station to deliver graphics and performance to the limit, and some on the team envisioned it as a tool primarily used by professionals and creatives in their offices. But I didn’t like any of these ideas and wanted it to be a mass market lifestyle product that consumers can take on the go. Apple’s top leadership has supported Ive’s plan, and Ive continues to play an active role in developing the headset, although he is now working with Apple as a consultant.

The decision to make the headset a standalone device reportedly caused a major headache. For example, some felt that the best way to make it work well on its own was to put more functionality on a single chip. But since the silicon work was already done, they had to find ways to combat the latency inherent in having multiple chips in a device communicating with each other. They were also making programs on the assumption that the base station plan would go ahead.

However, the device is moving to the final stages of development. Bloomberg claimed last week that an advanced version of the product was recently shown on Apple’s board, and that Apple “stepped up” development of the headphone software, an offshoot of iOS called rOS. (The letter R stands for “reality”).

Information reports reveal many details about the upcoming headset. It will feature at least 4K resolution for each eye, which the team believes is the bare minimum for users to not perceive the image as pixelated, unlike most current consumer VR headsets. The integrated processor will be closely related to the M2 processor expected to arrive on Macs and iPads in the coming months.

The headset will also have 14 cameras, some facing outward and some inward. It will allow users to see the outside world and enable people nearby to see a video representation of the user’s eyes. It is possible to track the wearer’s face and body movements directly to draw a 3D avatar (likely similar to the iPhone’s Memoji) that can be used for remote meetings and social gatherings with other remote headphone wearers.

Due to the limited processing capabilities of the M2 chip in a headset without a restricted base station (the scrapped base station was said to have the M1 Ultra’s ultra-performance), the glyphs would be cartoonish. Information sources also say that more photorealistic avatars were attempted when the base station was part of the plan, but Strange Canyon was a problem.

Apple initially planned to release the headset in 2019, but now it appears that it could instead be announced either later this year or in 2023. Additionally, Apple plans to offer more natural-looking augmented reality glasses as a follow-up product , but this device may still be years away from shipping.

Leave a Comment