Behind the Apple Design

Decisions That Bogged Down Its

Mixed-Reality Headset

In 2019, Apple had to make tough choices about the final design
of its augmented and virtual reality headset. In this story—the
second of two on the origins of the project—we reveal how those
decisions have led to huge technical challenges and delays for
Apple's biggest product gamble in years.
(L-R) Jonathan Ive, Johny Srouji, Dan Riccio and Tim Cook. Photos by Bloomberg; Apple. Art by Mike Sullivan

By Wayne Ma
! Share free article

May 20, 2022 6:00 AM PDT

A
pple’s executives had a critical design decision to make about the
company’s riskiest product in years. It was 2019, and a growing team of
Apple engineers had been working for more than three years on a
headset that combined augmented and virtual reality capabilities. Now they had to
figure out whether the mixed-reality headset would be a stand-alone device or
would require a powerful base station to produce the dazzling digital imagery
Apple envisioned for it.
 Apple CEO Tim Cook and then–Chief Design Officer Jonathan Ive were among the
executives who viewed VR demos on prototype headsets that simulated how the
two approaches would differ, according to two people familiar with the demos. The
headset that worked with a base station had superior graphics, including
photorealistic avatars, while the stand-alone version depicted its avatars more like
cartoon characters. Mike Rockwell, the Apple vice president in charge of the
company’s AR/VR team, favored the headset with the base station, believing that
Apple’s top brass wouldn’t accept the stand-alone version’s lower-quality visuals,
according to the two people.

He was wrong. Ive had pushed for the stand-alone version of the headset since the
early days of the project, according to a person familiar with it. Ultimately, Apple’s
senior executives sided with Ive. Despite that, Rockwell still assured them he could
make a great product. The choice has had lasting repercussions for the repeatedly
delayed headset, which goes by the internal code-name of N301.

THE TAKEAWAY

• Apple’s headset project faces technical hurdles related to chips, cameras,

avatars
• Marquee feature is lifelike avatars with accurate facial expressions
• Jonathan Ive remains involved in the headset’s development

In the years following the decision, Rockwell’s team, the Technology Development
Group, would struggle to build a stand-alone headset that balanced the need for
battery life and performance while minimizing the heat generated so people don’t
get singed while wearing the device. Those struggles raise questions about whether
:
 Apple can still meet its target of shipping the headset this year, according to people
familiar with the project.

Bloomberg reported
reported
reported
reportedearlier this year that Apple had canceled a planned
announcement of the headset at its annual Worldwide Developers Conference in
June, due to issues with overheating and challenges with its cameras and software
(Bloomberg also previously
previously
previously
previouslyreported
reported
reported
reportedsome details of Ive’s and Rockwell’s differing
visions for the headset).

Four people familiar with the project say the main reason for the delays is that
Rockwell hasn’t been able to deliver the high-quality mixed-reality experience he
told Apple’s executives he would. Apple’s leaders expect an AR experience far
beyond what
reported
competitors like Meta Platforms, parent company of Facebook, are
offering in terms of graphics, body tracking and latency—the lag between a user’s
movements and what they see on their display, according to three people familiar
with the matter. Apreviously
delay of even
reported
a few tenths of a second between a user’s head
movements and the corresponding perspective changes inside a headset can create
nausea.

Some of those people place the blame on Ive, who they say fundamentally changed
the purpose of the headset from a product that creatives and professionals would
use at a desk to a portable device for consumers. Those people argue that Apple
should have first developed a product for professionals to encourage them to make
content for the headset before releasing one for consumers.

Some people who worked on the device said focusing on a stand-alone version was
the right call but should have been made earlier when there was still time to
change key aspects of its design. By the time the decision was made, the device’s
:
 multiple chips had already been in development for several years, making it
impossible to go back to the drawing board to create, say, a single chip to handle
all the headset’s tasks.

Other challenges, such as incorporating 14 cameras on the headset, have caused

headaches for hardware and algorithm engineers. The cameras include those that
will track the user’s face to ensure virtual avatars accurately represent their
expressions and mouth movements, a marquee feature.

As The Information reported

reported
reported
reportedearlier
earlier
earlier
earlierthis
this
this
thisweek
week
week
weekin a story about the first several
years of Apple’s headset project, the company’s push into AR/VR is its most
important effort in years to define a new computing platform. Members of the team
working on the project believe lightweight AR glasses—which Apple has also
begun developing—could eventually replace iPhones for some users. In the early
days of the project, those team members warned that if Apple didn’t get serious
about AR/VR, Facebook, among others, could end up dominating the category.

reported earlier this week

:
 Jonathan Ive (left) and Tim Cook in 2018. Photo by Bloomberg

But the headset project also hasn’t gotten the same attention from Cook that the
iPhone did from former CEO Steve Jobs, which has added to its challenges,
according to people who have worked on the effort. And Rockwell’s track record of
successfully shipping products isn’t unblemished. When he was a senior executive
at audiovisual technology firm Dolby Laboratories, for example, he oversaw the
development of several products that never shipped, according to a person who
has worked with Rockwell.

Apple’s headset isn’t the only ambitious new project at the company to run into
snafus. Apple’s troubled car effort, known as Project Titan, has similarly struggled
to deliver on its promise of a fully autonomous vehicle after years of development.

A spokeswoman for Apple declined to comment.

Stand-alone Pressures
:
 When Apple’s senior executives decided to kill the headset’s base station in 2019,
some members of Rockwell’s team were in disbelief because they had spent so
much time making software for a system that had one. Prior to 2019, they had
explored using custom chips designed by Apple’s silicon engineering group, led by
Apple Senior Vice President Johny Srouji, that went into its headset prototypes and
a base station.
"

Srouji’s team designed two chips specifically for the headset, an image-processing
chip code-named Bora and a wireless chip code-named Golay, along with a third
chip, part of a powerful family of processors code-named Jade, that was to go into
the base station, said six people familiar with the project. (The base station’s
version of Jade was later announced as the M1 Ultra, Apple’s fastest desktop
processor to date.)

There were benefits to having a base station: The headset could offload most of its
computing functions to the separate device, preserving battery life and staying
cooler, which was important for a wearable computer. But the approach had
drawbacks as well. Relying on a base station meant users had to be mindful of
where to place the station so its signal wouldn’t be obstructed, and it limited the
potential uses for the headset because users couldn’t take it outdoors.

To address those shortcomings, Apple’s AR/VR group in 2018 began

experimenting with a headset prototype that could operate both as a stand-alone
device and be wirelessly tethered to a base station, said two people familiar with
the project. When users moved out of range of their base station, they could still
experience limited AR capabilities, said three of the people.

Still, most of Rockwell’s team believed the base station would offer the best
:
 experience, even though it would be mostly limited to use inside homes and
offices. When they heard that Apple’s leaders had instead opted for the stand-alone
version of the headset, some were concerned that the demo shown to Apple
executives downplayed the drawbacks of using such a device, which would
severely limit the amount of power it could draw and the heat it could safely
generate. That in turn would hurt how the device synced 3D visuals with user
movements, handled hand tracking and reconstructed images of the outside
world.

Even before the decision to jettison the base station, the group had struggled to
build algorithms for tracking and computer vision. That problem would only
become more daunting now that the headset itself had to do all of the work.

Meta Platforms' Quest 2 VR headset in action in May. Photo by Bloomberg

But there were competitive reasons for Apple to embrace a stand-alone headset. In
:
 May 2019, Meta released the Quest, a much anticipated stand-alone VR headset.
The product didn’t have the richer graphical capabilities of Meta’s original Oculus
Rift VR headset, which could operate only when tethered to a PC. In fact, some
managers in Rockwell’s group dismissed the Quest as cheap and underpowered,
and thought it made users look unfashionable.

But the Quest also gave users more freedom of movement, allowing them to easily
take the device on the go and use it on airplanes and in other locations outside
their homes.

More Discipline

Around this time, Apple quietly tapped

tapped
tapped
tappeda longtime trusted manager to join the
headset project, a sign that the company was getting more serious about shipping
the product. Kim Vorrath had previously worked in Apple’s software engineering
group, where she had played a key role in finalizing the first version of the
iPhone’s operating system.

Her intensity was legendary at Apple. In 2007, as Apple was racing to complete the
iPhone OS, Vorrath became so incensed when a colleague left early for home that
she slammed her office door hard enough to trap herself inside, The Information
reported
reported
reported
reportedin
in
in
in2014
2014.
2014
2014 Vorrath’s bosstapped
at the time, Scott Forstall, used a baseball bat to
liberate her from her office.

By the time Vorrath joined the headset project, Apple had pushed back the release
date of the device multiple times from an original target of 2019. She began
making changes to instill more discipline in the group. Prior to 2019, it had a
freewheeling culture, operating almost like a startup within Apple, said four
people familiar with the team. Employees brainstormed features and experimented

reported in 2014
:
 with ideas that might never see the light of day.

Vorrath brought more structure to the group, requiring individual teams to come
up with defining features for the headset’s software to motivate them and create
more accountability. After she joined, engineers were introduced to a concept she
had used in software engineering known as the “six-week sprint,” said two people
familiar with the matter. That schedule required engineers to spend four weeks
working toward a milestone such as making their code 10% more efficient,
followed by two weeks of debugging it.

And she told members of the AR/VR team that they could no longer use
commercially available headsets, such as the HTC Vive, to do software
development and had to instead use Apple’s own prototype headsets. That led to
challenges because the prototypes were bulkier and more prone to crashing.

At times, Apple’s AR/VR team found ways to test its hardware in unusual locations
outside the secrecy of its Silicon Valley offices. For example, members of the group
began taking them into the field disguised as surveying equipment to test their
effectiveness at creating digital maps of 3D environments.

Ive’s Influence

To keep the headset project on track, Rockwell needed money for new hires. One of
his most effective ways of getting it was to wow Apple executives with demos.

One featured a photorealistic Japanese garden in 3D with dancing butterflies,

while another used spatial audio technology to simulate the location of avatars’
voices in a beach house, said four people familiar with the matter.

But the app Rockwell’s team was most bullish on was a FaceTime-like
:
 communications program that represented participants with lifelike avatars and
made the participants feel as if they were in the same physical location. To help
Apple employees create accurate digital facsimiles of themselves to help with the
development of these avatars, Rockwell’s group used a camera rig in a building off
Tantau Avenue in Cupertino, Calif., where they could scan their entire bodies. The
team recruited people from the Hollywood visual effects industry to assist with the
avatar effort.

In some cases, the efforts to create lifelike avatars backfired. In the weeks leading
up to a demo for Apple’s top 100 employees in 2019, the Technology Development
Group attempted to digitally recreate Rockwell’s face, which Rockwell intended to
use to speak live to the audience while he sat in another room. But the resulting
avatar’s face was visually unsettling, marred by a phenomenon known as the
the
the
the
uncanny
uncanny
uncanny
uncannyvalley
valley
valley
valleyin which robots and digital simulations of human beings fall just
short of complete realism. So the team scrapped that part of the demo, three
people familiar with it said.

In October 2019, Rockwell’s team had made enough progress that he was ready to
share a new goal. He gathered the AR/VR team in a building on Apple’s campus
and told them Apple would ship the mixed-reality headset in 2022, followed by a
pair of smart glasses in 2023 (The Information reported
reported
reported
reporteddetails
details
details
detailsof the meeting
shortly after it happened). the
uncanny valley

reported details
:
 Jonathan Ive at an Apple Store in San Francisco in 2016. Photo by Bloomberg

Meanwhile, Ive—who as head of Apple’s industrial design group had exerted huge
influence over the headset project—announced
announced
announced
announced
announcedin
in
in
inJune
June
June
Junethat he would leave Apple
later in 2019, while remaining a consultant for the company.

One person familiar with the matter said Ive’s consulting work for Apple since he
left includes the headset, adding that he is often brought in to help his former team
push through their preferences in areas such as battery, camera placement and
ergonomics over those of engineers. Two people said even after Ive left Apple,
some employees on the headset project were still required to make the trek from
Cupertino to San Francisco, where Ive has a home, to get his approval on changes.
announced in June
Ive has continued to tweak the headset’s design. While earlier prototypes had the
battery in the headband, he prefers a design that would tether the headset to a
battery the user wears, similar to Magic Leap’s headset design. It couldn’t be
:
 learned if this approach will make it into the final design.

Ive, through a spokeswoman, declined to comment for this story.

Camera Complications

Another design decision that has greatly added to the technical challenges for the
Apple headset has been the inclusion of its 14 cameras, which allow it to capture
everything from images of the outside world to facial expressions and body
gestures.

Apple had to build the Bora image signal processor to process the bounty of
imagery. But Apple’s engineers have faced technical challenges getting Bora to
work with the headset’s main processor, code-named Staten. The back-and-forth
communication between the two chips increases latency, which can create nausea
for people wearing the headset.

As a result, Apple’s silicon engineering team had to build yet another piece of
silicon—a streaming codec—to act as a fast conduit between the chips, but that
solution hasn’t fully fixed the problem. Several people familiar with the headset
said this string of makeshift solutions is a reflection of overengineering, too-
complicated solutions that often result from poor planning. The Information
previously
previously
previously
previouslyreported
reported
reported
reportedthat Apple’s upcoming MacBook Airs and iPads will include
Staten and the company will unveil it later this year as its second-generation
desktop-class processor, the M2.

previously reported
:
 Apple senior vice president Johny Srouji speaks at a virtual Apple event in 2020. Photo by Bloomberg

Another technical hurdle for Apple has involved making a key function of the
headset—video pass-through, which depends on the cameras—work properly. That
function will allow people wearing the headset to see video images of their
surroundings on the displays inside the device, a capability intended to reduce the
isolation users experience with other VR headsets, as we previously
previously
previously
previouslyreported
reported.
reported
reported

But because Apple is also putting a display on the outside of the headset—which
will show video images of a user’s eyes and expressions to people around them—it
couldn’t position the outward-facing cameras roughly where users’ eyes will be.
Additionally, Apple’s industrial design team, for aesthetic reasons, insisted on
concealing the outward-facing cameras behind the glass display on the headset’s
front.

As a result, Apple engineers had to develop algorithms to compensate for the

previously reported
:
 awkward placement of the cameras so that users would have a more natural
perspective on their surroundings. Three people briefed on the project’s recent
status said Apple is close to solving this technical issue but is still confronting
some glitches.

Another reason Apple included so many cameras was to make sure it could
accurately capture facial expressions from anyone, regardless of the dimensions of
their faces—which is in line with the broader inclusion goals Apple’s leaders have
made a priority for the company.

Reducing Costs

In early 2021, Apple announced that Dan Riccio, a top executive who had been
overseeing the hardware engineering division housing Rockwell’s group, would
step down from that role and devote his time to an undisclosed project. Multiple
people familiar with the matter said that project was the AR/VR headset.

The shift echoed a similar move Apple had made in 2016, when it lured a longtime
executive, Bob Mansfield, out of retirement to supervise its secretive car project.
Riccio was assigned to the headset because Apple executives wanted to make sure
the project would stay on track, according to two people familiar with Rockwell’s
group.

Those people said Riccio has been focused on reducing the cost of the materials in
the headset to make it more affordable. He has a lot of work to do, given that Apple
is considering pricing the headset around $3,000, The Information previously
previously
previously
previously
reported
reported.
reported
reported

Apple is also taking early steps to encourage outside developers to create software
:
 for the headset. Three people familiar with the project said Apple’s RealityKit
engine will be the only way those developers can build AR apps for the headset,
though there is a plan to allow Unity Technologies to be the first partner to offer
full VR experiences in the headset via its game engine.

A Unity spokesman didn’t respond to a request for comment.

The headset’s operating system—reality OS or rOS, a version of the iOS software for
iPhones—will likely allow other game engines to operate in a more limited 2D
mode on the device.

Four people who have worked on the project also criticized its lack of focus on
gaming, a category of software that appeals to early adopters, which was
important to the success of the iPhone and has been a big priority for Meta’s VR
group. Those people said Rockwell’s group almost never mentioned games in
internal presentations about possible uses for the headset. Apple isn’t developing
game controllers for the device and is aiming to use hand tracking or in
combination with a clothespin-like finger clip as inputs for the device, multiple
people familiar with the project say. (The company is exploring ways to import
simple iPad games to it, one person said.)

A resurgence of Covid-19 in China also has Apple employees concerned that the
company will miss its goal of shipping the headset by the end of this year.
Taiwanese supplier Pegatron’s flagship plant in Kunshan, China, is assembling the
device. But Pegatron, which already assembles iPhones and iPads at the facility,
suspended production last month after local governments imposed strict
lockdowns on the region. Last week, the company said it would need to cut
production of some consumer electronics due to the outbreak.
:
 Inside Apple, Rockwell’s team continues to inch forward. On Thursday, Bloomberg
reported
reported
reported
reportedthat Apple’s board of directors had received another demonstration of the
headset at a meeting last week.

Once Apple ships the headset, the next big question will be when it can do the
same for its next product in the category, a pair of AR eyeglasses that resemble
Ray-Ban wayfarer glasses. While Rockwell has said in the past that the glasses will
likely debut a year after the headset, only a tiny portion of the AR/VR team is
working on the project, known as N421, according to six people familiar with the
effort. The glasses, which require an entirely different type of technology to operate
reported
given the transparent nature of the lenses, are still many years away from release,
those people said.

Read the first part in this two-part series about Apple's mixed-reality headset
project here
here.
here
here

Wayne Ma is a reporter covering U.S. tech in Asia, from Apple's supply chain to
Facebook's and Google's operations in the region. He previously worked for The Wall
Street Journal. He is based in Hong Kong and can be found on Twitter at @waynema
@waynema.
@waynema
@waynema

Subscriber Comments
here

@waynema
: