Point-E

OpenAI has released to open source Point-E, a machine learning system that allows to create a 3D object from a text request. This system uses a text-to-image model and an image-to-3D model to generate the 3D representation of the requested object. The text-to-image model, similar to other generative art systems such as DALL-E 2 and OpenAI’s Stable Diffusion, has been trained with labeled images to understand the associations between words and visual concepts. The image-to-3D model, meanwhile, has been fed a set of images associated with 3D objects so that it learns to effectively translate between the two.

Point-E does not generate 3D objects in a conventional manner, but rather produces point clouds, discrete sets of data in space that represent a 3D shape. Although these point clouds are computationally easier to synthesize, they do not capture the fine shape or texture of an object, which is a current limitation of Point-E. To overcome this limitation, the Point-E team has trained another AI system to convert point clouds into meshes, which do capture the shape and texture of objects. However, sometimes the mesh generation model can fail to understand the image and generate shapes that do not match the text request.

Point-E can produce 3D models in one to two minutes with a single Nvidia V100 GPU and, although its performance is worse than state-of-the-art techniques, it produces samples in a small fraction of the time. This could make it more practical for certain applications or allow higher quality 3D objects to be discovered. Point-E may have applications in 3D printing, game development and animation, among others. OpenAI is not the only company in the field of 3D object generation, as Google launched DreamFusion this year, an expanded version of Dream Fields, a 3D generative system the company introduced in 2021. Unlike Dream Fields, DreamFusion requires no prior training and can generate 3D representations of objects without 3D data. Although the current focus is on 2D art generators, AI model synthesis could be the next big disruptor in the industry.

3D models are widely used in film and television, interior design, architecture, and various scientific fields. For example, architecture firms use them to demonstrate proposed buildings and landscapes, while engineers use them as designs for new devices, vehicles and structures. 3D models typically take time to produce, ranging from a few hours to several days depending on their complexity. AI-enabled 3D model generation could shorten this process significantly and allow companies to create and test more models in less time. In addition, AI-enabled 3D model generation could have applications in education and entertainment, allowing students and content creation enthusiasts to create and modify 3D objects more quickly and easily.