If You Think Instagram Is Intimidating, Wait Until It’s In 3D
Facebook has rolled out limited 3D features, and a new app may push the concept even further
Imagine sharing not just a photo from your next vacation, but an interactive model of the beach itself, one that other people can “walk” through and explore from every angle. That’s the future that Display.Land, a new Android and iOS app, wants to bring to your phone. And it’s one that social media giants like Facebook have been working toward for a while.
Facebook first dipped its toe into the 3D-sharing pond in late February, when it rolled out a feature that allows some users with select Samsung and Google phones to create 3D versions of their photos. It isn’t true 3D because the original photo doesn’t actually contain any depth information. To achieve a 3D-like effect, Facebook instead uses machine learning to estimate how far an object is from the camera and cuts the picture into “slices” that can move independently. It’s similar to how movies that are shot in 2D can be converted to 3D, only done automatically.
Facebook’s solution is a clever way to fake depth on images taken with a single camera sensor. But dual-camera phones — which are becoming more common — can improve the process. Algorithms can examine the small differences in how each of the two lenses captures a photo in order to generate depth data, similar to how human eyes work.
The result from either method is the kind of “3D” photo you might’ve seen on your Facebook feed, one that you can wiggle a bit with your cursor to distort in a way that makes it feel like you’re seeing it from a slightly different angle. It’s a funny novelty and it can make for some amusing memes, but it’s hardly a real 3D representation of an object or scene.
A more accurate re-creation of a scene would involve a process called photogrammetry, which involves taking a large number of photos of a physical object and converting data from those photos into a 3D model. With enough information — and it can take quite a lot of information — any object can be re-created almost perfectly. A variation of this method called videogrammetry does the same kind of analysis using frames of a video. Free software like Meshroom has allowed artists to create 3D models from photos and video for years, but it’s a cumbersome process for the average person.
Display.Land is trying to make true 3D more accessible for everyone. The app asks users to scan an object or scene by pointing their camera at the subject and walking around it to capture as much detail as possible. Once a certain threshold of data is collected, it’s uploaded to Display.Land’s servers, where it’s analyzed and converted into a 3D scene, complete with textures. Put all together, this creates explorable scenes that can be published to a TikTok-style feed that other users can browse.
The app launched less than a year ago, in November 2019, and judging by the amount of activity, it has a sparse user base. Still, the scenes those few users have shared publicly show the potential of sharing experiences in 3D. They include scans of everything from ancient temples to street graffiti to users’ Airbnbs to what they ate that day. In other words, the app features many of the same things you can find on Instagram, but in a form onlookers can explore from every possible angle
Display.Land is far from perfect. Scans are best for capturing the general layout of a scene or a building, rather than fine details or small objects. For example, in the GIF above, the legs of some of the patio chairs get erased entirely. The landscape in the far distance is also garbled, because the user wasn’t able to capture enough data for that particular area while scanning with their phone.
On the other hand, like photos from a camera phone in the early 2000s, the fact that social 3D images can be done at all is more notable than the quality of the result. In the somewhat recent past, this kind of technology would’ve required the kind of specialized equipment you might find in a cutting-edge film studio. Today it’s available on most major smartphones. Display.Land is free and works even with phones that have just one camera lens.
Google has also done work in the photogrammetry field, though with considerably more resources. In late 2019, the company released the results of its highly detailed VR tour of the Château de Versailles — though calling it a “VR tour” is a bit of a misnomer. While it’s possible to explore artifacts and exhibits using headsets like the HTC Vive or Oculus Rift, much of the data is available in a simple web browser. With a phone or laptop, anyone can examine sculptures, artifacts, or even entire rooms in 3D. Exhibits that might not be practical for most people to visit in person can be digitized and made available online for anyone.
It may take time for the quality of 3D scenes that anyone can make to improve. Display.Land users have to upload their scene data to the cloud because the processing power required to create a 3D scene is too great to perform on a phone. And while multiple cameras on a phone could help, perfectly accurate 3D scenes might someday require specialized sensors — potentially like the radar sensor in Google’s latest Pixel phone — that can capture depth data with more resolution.
But despite the hurdles, the technology is likely to get better, if for no other reason than its potential as an artistic tool. The methods used in apps like Display.Land find their roots in special effects and filmmaking, and as new techniques like those used on The Lion King and The Mandalorian continue to blur the line between physical and virtual reality, those techniques will likely filter out into consumer products as well.
It’s still early days for 3D sharing, and even the best technology used to create explorable scenes can lead to clunky models with misaligned textures (see my best effort with Display.Land here). However, as techniques get cheaper, more advanced, and more accessible, it’s feasible to imagine a not-too-distant future where social media users aren’t just able to show off photos or videos of their experiences, but let their followers step into a re-creation of their world.