Nvidia is making great strides in reverse rendering to turn still photos into 3D objects

Nvidia Corp. today present the impressive progress it has made in the field of “reverse rendering”, which is a technique that uses artificial intelligence to reconstruct a series of still photos into a 3D model of an object or scene.

Nvidia’s new reverse rendering method, known as MoMa 3D Nvidia, has potential applications for video game developers, architects, designers, and concept artists. With it, they could quickly and easily import a photo of an object into a graphics engine and edit it in multiple ways, changing its material and texture, adding lighting effects, or changing its scale.

3D MoMa works by taking a photograph of an object or scene and turning it into a kind of triangular mesh, with textured materials, which can be dropped into game engineers, 3D modeling programs, and renderers. of movies. Nvidia explained that advances in neural radiation fields, combined with the processing power of its Tensor Core graphics processing units, allow it to generate these triangular mesh models in an hour or less.

The reconstruction process recreates three key elements of the still photo: a 3D mesh model of the object or scene, materials, and lighting. The mesh can be thought of as a papier-mâché model of the object in question constructed from triangles.

This model can then be modified in various ways by developers to adapt the object to their creative vision. Materials are integrated as 2D textures that can be overlaid on the 3D mesh like a skin. Then the model calculates how the recreated object is lit to maintain lighting accuracy.

Everything is automated, making it much easier than the traditional process of creating 3D objects from scratch using complex photogrammetry techniques, which involve manual effort and can take hours, Nvidia said.

At this week Conference on Computer Vision and Pattern RecognitionNvidia demonstrated the capabilities of Nvidia 3D MoMa by creating 3D objects from a set of images of jazz band instruments, including a trumpet, trombone, saxophone, drums, and clarinet, taken under different angles.

Nvidia 3D MoMa studies these photos and reconstructs the 2D images as 3D representations of each instrument. In this way, he can extract them from their original scene and import them into the Nvidia Omniverse 3D simulation platform to edit them.

From there, it becomes possible to modify the shape or change the material of each instrument. The Nvidia team replaced the original plastic material of the trumpet with various materials, including gold, marble, wood, and cork.

The edited 3D objects can then be dropped into any virtual scene. Nvidia tested this by placing the instruments in a Cornell box, which is a classic graphics test of render quality. The instruments reacted to light as they would in the physical world, the shiny metallic ones reflecting brightly, while the dull drumheads absorbed most of the light.

Finally, Nvidia used the instruments as building blocks to create a complex animated scene of a virtual jazz band.

Holger Mueller of Constellation Research Inc. told SiliconANGLE that it’s good to see Nvidia making progress in creating 3D objects, as it’s an area that involves a lot of tedious and resource-intensive work. This has hindered the creation of rich virtual reality and augmented reality applications, he added.

“Innovations such as transforming 2D materials into 3D objects are key to creating a richer software experience, whether for virtual reality, augmented reality, or the metaverse itself,” Mueller said.

“The Nvidia 3D MoMa rendering pipeline uses modern AI machines and the raw computing power of Nvidia GPUs to quickly produce 3D objects that creators can import, modify, and extend without limitation in existing tools,” said David Luebke, vice president of graphics at Nvidia. to research.

Picture: Nvidia