A Web Engineer’s Experiment: Reimagining Product Manuals with Apple Vision Pro

I’m Fujiwara from Servithink R&D, a web development company based in Tokyo. Our team has been exploring how Web and XR technologies can shape the future. In this article, I’ll introduce an experimental MR (Mixed Reality) application we developed using Apple Vision Pro to rethink how product manuals might work in the near future. Instead of relying on traditional paper manuals or searching online, we explored a new approach where product information appears directly in the user’s field of view.

What We Built: A Manual App for Apple Vision Pro

We created an app for Apple Vision Pro that displays product manuals in MR, right next to the actual item. The app uses object recognition, so when a registered product comes into view, a user interface automatically appears. This removes the need to store or search for paper manuals, and allows users to view the most up-to-date information at any time.

The app also provides easy access to official websites and contact pages. For instance, if you want to check more details about a product or get in touch with support, you can simply look at the item and select a floating button to open the relevant webpage.

We designed the app with a simple and intuitive user experience. When launched, it automatically recognises the product and highlights it with a yellow glow. Above the product, three buttons appear: one for the product website, one for the manual PDF, and one for the contact form.

As with other Apple Vision Pro apps, both eye tracking and hand tracking are supported. This means users can interact with floating content in the air or control it from a distance using just their gaze.

This made accessing product information very convenient. I often have trouble remembering how to maintain my washing machine and have to dig out the manual each time. With this system, I could check it easily, which would save me time and effort.

In this test, we used a steam iron as the recognised item, but the app’s true value will show when it can support a wide range of household products. I hope to continue testing how physical items can connect more intuitively to the web.

What We Learnt from Building It

Where Should Web Views Appear?

In this app, the web view window is fixed above the product. We also considered placing the window in front of the user or allowing users to move it freely. However, that makes it harder to know which window is linked to which product.

To make the connection between the item and its information clear, we kept the window fixed near the object. But this also introduced a limitation: users had to physically move closer to interact with it. In some cases, this reduced the usefulness of MR.

During development, we considered options such as drawing lines between items and their windows, or making the item glow when its window is selected. However, these might have made the space visually cluttered or difficult to view, so we decided not to implement them.

There are many situations where users may want to view manuals from a distance. Also, if the window is placed too high, children might not be able to interact with it. This is one of the first issues we should address in future updates.

How Product Type Affects Window Placement

We also found that the ideal angle for the window depends on the type of item. For example, wall-mounted products like air conditioners or intercoms are easier to use when the window is fixed parallel to the wall. But for portable items like a steam iron, which can be held in many positions, it feels more natural for the window to face the user directly.

People often look at objects from different angles—not always straight from the front. (And really, what even is the front of a steam iron?) So we may need a more flexible system that adjusts the window’s angle depending on the object.

Designing Web Interfaces for MR

This app displays PDF manuals hosted on the web. But browsing websites in MR is very different from using a computer or smartphone. We found that readability and ease of interaction needed special attention.

For instance, eye tracking and hand tracking make small buttons difficult to use. Apple and Google traditionally recommend a minimum button size of 44–48 pixels for touchscreens. But for spatial user interfaces, Apple recommends buttons of at least 60 pixels. Web designers may need to adapt to this new standard when developing for MR.

Also, simply enlarging the window isn’t always helpful. If the window is too big and fills your field of view, moving your gaze from one side to the other becomes tiring. If it’s too small, the text may be hard to read or select. So designers need to find the right balance between size and clarity.

This raises a key question: what kind of layout works best for MR? Web design may soon need to go beyond responsive layouts and consider MR-specific interface design.

A Future Where Products Connect Directly to the Web

Through this project, we began to imagine a future where all household products are connected to the web. As smart glasses become more common, we might not just read manuals—we might change settings, check maintenance schedules, track food expiry dates, or receive recipe suggestions in real time.

This kind of app also fits well with the growth of AI. Smart glasses that explain what you’re looking at already exist. If we connect these systems to official product information, AI can provide more accurate and reliable answers.

Today’s AI assistants usually search the internet to answer questions like “How do I use this product?” But the information they find isn’t always correct. If each product is linked directly to its official data, AI can give up-to-date and trustworthy responses.

As we rely more on AI to get information, it’s essential that the sources are accurate. Making the web a dependable source of official data is more important than ever.

Conclusion & Next Steps

Looking ahead, it would be exciting to develop features like interactive 3D models that show users how to use a product. But for now, this MR manual app is a first step in exploring how physical items can connect directly to the web.

Through this experiment, we identified many challenges around web view placement, interface design, and information presentation. Some should be addressed on the MR app side, while others may require improvements in web design.

By working through these challenges, we can create web experiences that feel more natural and helpful in MR environments.

This project also gave us a fresh perspective on how the web and AI can work together. As AI becomes a common way to access information, ensuring its connection to reliable sources will be key. Linking products to their official data helps make that possible.

We’re continuing to develop other projects that explore the future of Web × XR. Rather than just imagining what comes next, we aim to build and test it—so we can discover new possibilities, one experience at a time.