Here is how Apple’s mixed-reality headset will work

iPhone maker has discussed developing VR content for the platform with about half a dozen media partners

Visitors try Disney's virtual reality experience at the D23 Expo in Anaheim, California. Apple is in talks with Disney about providing content for its headset. AFP

Apple’s long-anticipated mixed-reality headset is an ambitious attempt to create a 3D version of the iPhone’s operating system, with eye and hand-tracking systems that could set the technology apart from rival products.

The roughly $3,000 device, due later this year under the possible name of Reality Pro, will take a novel approach to virtual meetings and immersive video, aiming to shake up a VR industry currently dominated by Meta Platforms.

It’s a high-stakes move for Apple, which is expanding into its first major new product category since releasing a smartwatch in 2015, and the company needs to wow consumers.

Apple is pushing into an uncertain market with a premium-priced product. The company’s 1,000-person-plus Technology Development Group has spent more than seven years on the project, and Apple is counting on it to become a new revenue source — especially with sales growth poised to stall this year.

But virtual reality has proven a challenge for the biggest titans of technology. Though some projections have the industry topping $100 billion by the decade’s end, headsets are still seen as niche items — and Meta has lost billions on its efforts.

Apple’s goal is to bring something new to the table. The eye and hand-tracking capabilities will be a major selling point for the device, according to people familiar with the product, which is expected to cost roughly twice the price of rival devices. Its core features will include advanced FaceTime-based videoconferencing and meeting rooms.

The headset also will be able to show immersive video content, serve as an external display for a connected Mac, and replicate many functions of iPhones and iPads.

Here’s how it will work: The headset will have several external cameras that can analyse a user’s hands, as well as sensors within the gadget’s housing to read eyes. That allows the wearer to control the device by looking at an on-screen item — whether it’s a button, app icon or list entry — to select it.

Users will then pinch their thumb and index finger together to activate the task — without the need to hold anything. The approach differs from other headsets, which typically rely on a hand controller.

Like Meta’s latest headset, Apple’s device will use both virtual and augmented reality. With VR, users see images and content within the goggles. AR, on the other hand, overlays digital content on top of real-world views.

The headset will have two ultra-high-resolution displays — developed with Sony Group Corp. — to handle the VR and a collection of external cameras to enable an AR “pass-through mode.” That means users will see the real world through the cameras positioned on the headset. Apple will offer users with prescription glasses custom lenses that sit within the enclosure itself.

The device will have a so-called Digital Crown — like the Apple Watch — that lets users switch between VR and AR. When in VR, the wearer is fully immersed.

When AR is enabled, the content fades back and becomes surrounded by the user’s real environment. Apple expects this to be a highlight of the product, according to the people, who asked not to be identified because the project is still under wraps.

Given that the headset is still months from being released, some features could still be cancelled or changed, the people added. A spokeswoman for Apple declined to comment.

The headset’s FaceTime software will realistically render a user’s face and full body in virtual reality. Those avatars will allow two people — each with an Apple headset — to communicate and feel like they’re in the same room. The technology differs from virtual meeting rooms on Meta’s headset, which creates a more cartoonlike avatar of the user.

Because of the immense processing power necessary for the feature, the headset will only support realistic avatars during one-on-one video chats. It will still allow for FaceTime sessions with several people, but additional users will be displayed as an icon or Memoji — Apple’s customised emoji.

Apple plans to unveil the device as early as this spring, though the schedule could still shift, according to the people. That would let the company discuss the product at its annual conference for software developers in June and then release it later this year.

As with some of Apple’s earlier big bets, the company plans to start slow. It’s aiming to begin early production of the device as soon as February in China and is considering launching the product in the US only to start.

The price tag also is expected to limit the product’s appeal, and Apple is already working on a cheaper version — for release in late 2024 or early 2025 — that could be closer to $1,500. That’s what Meta charges for its mixed-reality headset.

Apple expects to sell about 1 million units of its new headset in its first year. That compares with more than 200 million units for the iPhone, the Cupertino, California-based company’s biggest moneymaker. In a rare move, it’s also not planning to make a profit on the initial version — even at the high price — indicating that the company is taking a long-term view of the platform.

Immersive video-watching will be a core feature of the new device. Apple has discussed developing VR content for the platform with about half a dozen media partners, including Walt Disney and Dolby Laboratories. And the tech giant is working to update its own Apple TV+ material to work with the headset. As part of the push, Apple bought streaming company NextVR in 2020, aiming to create sports content in VR.

Apple is planning for the headset to have a dedicated video-watching feature that can make viewers feel like they’re seeing a movie on a giant screen in another environment, such as a desert or outer space. But while the headset’s video will be immersive, its speakers will be less powerful. So users will need to wear AirPods earbuds to get full spatial audio — a surround-sound effect.

The device will also have productivity features, including the ability to serve as an external monitor for a Mac. With that feature, users will be able to see their Mac’s display in virtual reality but still control the computer with their trackpad or mouse and physical keyboard.

The headset’s operating system, internally called xrOS, will have many of the same features as an iPhone and iPad but in a 3D environment. That includes the Safari web browser, photos, mail, messages and the calendar app. And it will also have apps for the company’s services, such as the App Store to install third-party software, Apple TV+, music and podcasts. The company is working on health-tracking functions as well.

The experience should feel familiar to Apple users. When they put the headset on, the main interface will be nearly identical with that of the iPhone and iPad, featuring a home screen with a grid of icons that can be reorganised. Users will be able to pin widgets, such as the weather, calendar appointments, email and stock-market performance, among their app icons.

When users need to input text, they can use the Siri voice assistant or rely on an iPhone, Mac or iPad keyboard. Unlike with an Apple Watch, though, an iPhone isn’t required for operation. The company is developing technology that will let users type in mid-air with their hands, but such a feature is unlikely to be ready for the initial launch.

Gaming is expected to be a popular offering from third-party developers, and Apple has created its own underlying engine to power VR games. In 2017, the company released ARKit and other tools to help developers prepare augmented reality experiences on the iPhone. This helped set the stage for programmers to build apps, games and services for the headset.

The Apple device will include a variation of the M2 chip found in the company’s latest Macs, as well as a dedicated processor for graphics and mixed-reality experiences. That second chip will be dubbed the Reality Processor, according to trademark applications filed by the tech giant.

But making the processors powerful enough brought another concern: having the device overheat while it’s on a user’s face. To address that problem, Apple made the decision to offload the battery from inside of the headset to an external pack. It rests in a user’s pocket and connects over a cable. Another tweak is the inclusion of a cooling fan like on high-end Macs.

The headset can last about two hours per battery pack, in line with rival products. The battery, however, is large: roughly the size of two iPhone 14 Pro Maxes stacked on top of each other, or about 6 inches tall and more than half an inch thick. Still, some internal prototypes for software development have a built-in battery and charge over USB-C.

Meta, in contrast, puts its batteries on the back of its headset in a way that helps balance the device on a person’s head. Apple’s approach may ultimately be less comfortable for users, especially if they’re watching a whole movie while wearing the headset. Some testers have complained that the product can be cumbersome, according to the people.

The relatively brief battery life — about 20 hours less than Apple’s latest MacBook Pro — could create its own hassles. If users want to watch multiple movies or play games for hours at a time, they may need to buy multiple batteries and frequently swap them out.

Apple has acknowledged those challenges internally, and it’s been trying to set realistic expectations for the product. One benefit of the device, the company believes, is that it could spur customers to visit Apple retail stores — not necessarily to buy the product, but to try it out. They may then purchase another device, such as an iPad or AirPods.

To show off the new headset, Apple is creating a “store within a store” concept — an area within its retail outlets dedicated to demonstrating the product. The company did something similar when it launched the Apple Watch, which is now central to a $41 billion division.

The initial headset will be made from aluminium, glass and cushions — and be reminiscent of Apple’s $550 AirPods Max headphones. The product will have a curved screen on the front that can outwardly show a wearer’s eyes, with speakers on the sides and a headband that helps fit the device around a user’s head.

That will differ from the mostly plastic design of rival products, which typically strap the device to the wearer with multiple bands.

The eye and hand tracking may end up being the most memorable element of the headset. As with its earlier big bets, Apple likes to include a groundbreaking interface that sets its products apart from competitors. With the iPod, it was the click wheel. With the iPhone and iPad, it was the multitouch approach. And with the Apple Watch, it was the Digital Crown.