Empowering 3D at American University and Cannes

Honing 3D storytelling skills with the help of Sony’s Spatial Reality Display at American University and showcasing professional projects at Cannes.

The challenge

• Overcome the limitations of conventional AR and VR headsets
• Monitor and review student 3D capture projects with the highest possible quality
• Exhibit 3D content at the Cannes XR Marché du Film

The Solution

• Leverage roughly 4 million pixels per eye delivered by the Spatial Reality Display
• Take advantage of the Unity system to render graphics in real time

The Outcome

• Enables highly accurate, impressively detailed 3D visualization
• Stops traffic at trade show exhibits
• Establishes a powerful, new modality for content viewing

A man of many parts
Krzysztof Pietroszek isn’t easily impressed. His professional journey includes a PhD in computer science and an MFA in film, while his personal journeys have taken him to the summits of Mont Blanc and Kilimanjaro. He has produced an award-winning feature film, developed VR video games, and published more than 40 peer reviewed research papers. Now he is incorporating Sony’s Spatial Reality Display into his work.

“While we have reliable although not perfect technologies for capturing what we call ‘holograms,’ displaying them well has been limited to glasses that nobody has,” says Pietroszek. “And even if you have the glasses, you’re limited to maybe 100,000 vertices. That means you’ve got to scale down high-quality capture. The Spatial Reality Display gives you the ability to show much, much higher resolution. And you can make the display interactive. You can put your hand out and interact with the object.”

Impressing the film community at Cannes
On behalf of the start-up he founded, Tabletop Stories, Pietroszek brought Sony’s Spatial Reality Display to an exhibit at the Cannes XR Marché du Film. “When people were passing by, they would catch a glimpse of it and stop! They needed to see what it was and how it happens.”

Tabletop Stories presents movies in AR, typically shown on smartphones and tablets, where the story content is overlayed on the real-time camera image of the room. Pietroszek says, “The difference is huge between showing AR on a tablet – on which the content is supposedly in air – and seeing the same content on the Spatial Reality Display, actually in air without glasses. People are really surprised.”

“At its heart, immersive filmmaking breaks the fourth wall. This display enables you to do it with much, much higher resolution.” -Dr. Krzysztof Pietroszek, American University & Tabletop Stories

Teaching immersive filmmaking at American University
Pietroszek teaches a class in volumetric capture. “We capture with a single camera and Depthkit software, which is affordable and very easy to use. It’s not true volumetric capture because you get one-sided holograms – half a person. But for an introductory class, it’s great. The Spatial Reality Display is very good for this. We can rotate it toward you and it works well.”

Pietroszek finds the Spatial Reality Display more useful than multi-view 3D monitors. “For my applications, the multi-view monitors don’t really work. The resolution is really, really low. You need multiple, multiple images. And the angle of view is so small that they don’t really make sense.”

The Spatial Reality Display and Unity rendering
The Sony Spatial Reality Display enables viewers to see volumetric 3D without special glasses or VR goggles. 1.2 The display works from computer generated images, in the case of American University, using a Windows® PC running the Unity® rendering engine.3 “We train students on Unity,” says Pietroszek. “It’s super easy to get the content on the display. It works without any issues. It just plays.”The screen features a high-precision micro-optical lens to separate the left-eye and right-eye images. In addition, the display tracks eye movement down to the millisecond, pivoting the image in sync as the viewer tilts or moves the head up and down, left and right — even forward and back. The result is an overpowering sensation of a solid 3D object.

Future Prospects
Pietroszek sees multiple future applications for the Spatial Reality Display. “I would definitely recommend this for demos and trade shows and for specialized applications like 3D modeling. And I’m working on a telemedicine project. We have this idea of using volumetrics so that remote experts can assist inexperienced doctors.”