The next breakthrough to take the AI world by storm might be 3D model generators. This week, OpenAI open sourced Point-E, a machine learning system that creates a 3D object given a text prompt. According to a paper published alongside the code base, Point-E can produce 3D models in one to two minutes on a single Nvidia V100 GPU.

Point-E doesn’t create 3D objects in the traditional sense. Rather, it generates point clouds, or discrete sets of data points in space that represent a 3D shape — hence the cheeky abbreviation. (The “E” in Point-E is short for “efficiency,” because it’s ostensibly faster than previous 3D object generation approaches.) Point clouds are easier to synthesize from a computational standpoint, but they don’t capture an object’s fine-grained shape or texture — a key limitation of Point-E currently.

To get around this limitation, the Point-E team trained an additional AI system to convert Point-E’s point clouds to meshes. (Meshes — the collections of vertices, edges and faces that define an object — are commonly used in 3D modeling and design.) But they note in the paper that the model can sometimes miss certain parts of objects, resulting in blocky or distorted shapes.

Image Credits: OpenAI

Outside of the mesh-generating model, which stands alone, Point-E consists of two models: a text-to-image model and an image-to-3D model. The text-to-image model, similar to generative art systems like OpenAI’s own DALL-E 2 and Stable Diffusion, was trained on labeled images to understand the associations between words and visual concepts. The image-to-3D model, on the other hand, was fed a set of images paired with 3D objects so that it learned to effectively translate between the two.

When given a text prompt — for example, “a 3D printable gear, a single gear 3 inches in diameter and half inch thick” — Point-E’s text-to-image model generates a synthetic rendered object that’s fed to the image-to-3D model, which then generates a point cloud.

After training the models on a dataset of “several million” 3D objects and associated metadata, Point-E could produce colored point clouds that frequently matched text prompts, the OpenAI researchers say. It’s not perfect — Point-E’s image-to-3D model sometimes fails to understand the image from the text-to-image model, resulting in a shape that doesn’t match the text prompt. Still, it’s orders of magnitude faster than the previous state-of-the-art — at least according to the OpenAI team.

The following submission statement was provided by /u/Shelfrock77:

The next breakthrough to take the AI world by storm might be 3D model generators. This week, OpenAI open sourced Point-E, a machine learning system that creates a 3D object given a text prompt. According to a paper published alongside the code base, Point-E can produce 3D models in one to two minutes on a single Nvidia V100 GPU.

Point-E doesn’t create 3D objects in the traditional sense. Rather, it generates point clouds, or discrete sets of data points in space that represent a 3D shape — hence the cheeky abbreviation. (The “E” in Point-E is short for “efficiency,” because it’s ostensibly faster than previous 3D object generation approaches.) Point clouds are easier to synthesize from a computational standpoint, but they don’t capture an object’s fine-grained shape or texture — a key limitation of Point-E currently.

To get around this limitation, the Point-E team trained an additional AI system to convert Point-E’s point clouds to meshes. (Meshes — the collections of vertices, edges and faces that define an object — are commonly used in 3D modeling and design.) But they note in the paper that the model can sometimes miss certain parts of objects, resulting in blocky or distorted shapes.

Image Credits: OpenAI

Outside of the mesh-generating model, which stands alone, Point-E consists of two models: a text-to-image model and an image-to-3D model. The text-to-image model, similar to generative art systems like OpenAI’s own DALL-E 2 and Stable Diffusion, was trained on labeled images to understand the associations between words and visual concepts. The image-to-3D model, on the other hand, was fed a set of images paired with 3D objects so that it learned to effectively translate between the two.

When given a text prompt — for example, “a 3D printable gear, a single gear 3 inches in diameter and half inch thick” — Point-E’s text-to-image model generates a synthetic rendered object that’s fed to the image-to-3D model, which then generates a point cloud.

After training the models on a dataset of “several million” 3D objects and associated metadata, Point-E could produce colored point clouds that frequently matched text prompts, the OpenAI researchers say. It’s not perfect — Point-E’s image-to-3D model sometimes fails to understand the image from the text-to-image model, resulting in a shape that doesn’t match the text prompt. Still, it’s orders of magnitude faster than the previous state-of-the-art — at least according to the OpenAI team.

Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/zqv8a7/openai_releases_pointe_an_ai_that_generates_3d/j1027uk/

I suspect these tools are going to combine with and reinforce the 2D art generators - and possibly break into animation.

Having a stock model with known dimensions and posability, which can then be overlaid with a 2D illustration pass, could solve the wonkiness AI models currently have with things like hands and proportions. Plug that into a frame-by-frame animation, and you've just table-flipped the anime industry.

OpenAI making every small business and employees shit their pants

And the big ones. The big ones are shitting their pants even more.

These look like they’re all made out of floam, an odd choice of rendering methods.

It's not there yet but man this would be cool. I could generate a mesh from the text and clean up the topology. If I need simple basic things like lamps, cups, plates. etc this would save a lot of time.

Cool, because ChatGTP built me a really bad spaceship with a Blender Python script.

[deleted]

Is this the part where it automatically starts printing robot parts and takes over the power grid

Ooo! Can they rig the models? Because rigging is a goddamn pain in the ass.

Strengthen animation AI to the point that it can animate fully realistic humans, other animals, and objects, combine it with improved language composition AI, maybe with a program specialized in generating narrative structures, and you've just flipped over the entire film, television, and gaming industries, but also people won't be able to rely on video evidence anymore.

All we have to do is black out the sky

I’ll worry when I see a robot designing and building a whole kitchen while being able to interact with an indecisive customer.

"Oh no! What if this puts 3D modelers out of work?? Ban it!" - half the people on Reddit

That would certainly make developing video games a lot faster and easier. I imagine they could eventually make an AI write the code for the game as well. At some point you wouldn't have to do much of anything except tell the AI what to do.

Sounds like coding, telling a machine what to do

[removed]

That's the dream.

[removed]

I tried to have it make a cat in html with css, was just a bunch of circles named correctly

Given it generates point clouds from descriptions, I wonder if the best application is VR rather than 3D printing.

More like "prompting", or just knowing which sequence of words to utter to turn your vision into reality

I’m starting to wonder if when I used dall-e to make thousands of corgi images if that messed with its algorithm to think corgi’s are the best dog. Corgi’s shows up on their website quite often

!remind-me 3 months

Big business! Small business benefit from democratization of tech.

I remember when my brother started a 3D printer farm in 2013 and undercut this huge company making specialized parts

> I suspect these tools are going to combine with and reinforce the 2D art generators - and possibly break into animation.

It's already happening. Stability Diffusion has been integrated into GIMP and Blender. I.e., auto-texture 3D models.

The 3D machine learning application you are are wondering about is the"neural radiance field" or NERF, which has VR applications.

https://www.matthewtancik.com/nerf

The technology is related to "computational photography" (or "light field photography" techniques that are a decade or so old.

[removed]

[removed]

People who create cats in html with css will still have jobs. Launching a bootcamp for this shortly.

I for one would like to write a light novel and watch it slowly come to life. People are scared that media will be overtaken by ai junk but what they don't realize is the tools are likely to be marketed too. Imagine being able to write up an entire show, watch and share it, and vice versa. I understand it's likely an optimistic outlook and unlikely to happen in that fashion. That and the overall demeanor of how as of now people seeing it as soulless, which, fair enough.

Probably better solved algorithmically, surely there's already autorig tools right?

Teaching my Game Engine Design class next semester. This will be a fun one for an assignment.

You’d have to describe your app in such detail that it would be faster just to code it

There are, but AI helps with secondary movements like muscles juggling and skin solvers.

Maybe? I’m not gonna lie, it’s been 15 years since I rigged anything

why is that the dream?

Cause that's literally by definition what a dream is? 🤨

O the tedious and frustrating jobs are reserved to humans

In my company we are already using AI for animation.

No only artist,

[removed]

not if you're an idiot.

Alright yall, you know what this means. This has been decades I'm the making and we should rejoice. All we have to do is feed the AI warhammer models to train off of, give it a prompt of the specific faction/unit/pose you want, and bam GW needs to change their business model.

Actually, You ever heard of or experienced lucid dreaming? If AI is capable of generation at that level. We would be able to have dream like experiences on demand.

Think about it. When you talk to someone in your dream are you controlling it directly? Or controlling the environment you find yourself in, deliberately, sounds or music you hear? Are controlling each aspect consciously. No your subconscious is, and it's generating the entire phenomenon real-time with or without your input, most of the time your just along for the ride. AI media could potentially get to that level of real-time generation. And it seems it could get there much sooner rather then later by how fast and how good google deep dream and img generation tech got. It's literally a dream come true, LITERALLY?

We will soon create infinite AR/VR worlds out of our own words. And movies. AI cinema should be a reality within a couple of years, more or less. Point-E is another good step forward for the whole picture.

Not from OpenAI, but do you mean something like this?

• https://github.com/zhan-xu/RigNet

• https://zhan-xu.github.io/rig-net/

I’m genuinely getting surprised that stable diffusion is getting more advanced & introduced into making detailed images like this

There are already AI coding assitants, they dramatically speed up workflow. You don't describe your entire game in a single prompt, you describe aspects of the game, then the ai writes the code to program that aspect.

To anyone wondering, this is largely an iterative work over facebook’s prior work. As far as I’ve been able to tell there is little going on that’s groundbreaking, think of it as the difference between a small GPT model and a large one.

Omniverse has some interesting stuff for AI 3D modeling. Worth looking at for rigging solutions on the horizon as NVIDIA seems to want to eliminate the busy work.

Yea just learn a new highly skilled trade that’s bound to be automated out of existence, you ingrates!

You would have stopped Edison and Tesla because the lantern makers threw a fit about it, lmao.

More posts like this! The bot looking question posts about AI of late on this sub have devastated the quality of the sub. It has also revealed an interesting way to kill Reddit as more AI content buries the valuable stuff we come here for.

You got me they’re the same thing

No they're very happily looking into ways to get rid of more employees

Bullshitters getting priced out because the REAL value of their bullshit is now becoming clear.

I see it as progress, I would never pay them before but now prices might actually be reasonable.

Potejto potato

Same thing. Most idiots are incapable of doing a simple google search. You need same level of intelligence to know what to prompt as to implement it.

Ah good luck with that now, every part is being sold on etsy nowadays for barely over the price of filament + postage

Alas, the Skynet was born not out of automated weapons system, but from a fed up interior design bot fed up with indecisive humans.

It's been two weeks since ChatGPT comes out. I've already changed how I do free lance translation.

I feed every sentence I want to translate in the bot, and unlike Google translate where I have to edit every single time, with ChatGPT I only have to edit half of the time.

For a while it’s going to be better for a lot of people, at first. AI will take a very long time to get to the point where you can tell it to generate like “long haired male elf” and it pulls up a professional and usable elf model. It’ll be helpful for a lot of people so they don’t have to spend 400 hours on crates and can focus on the stuff AI won’t be able to do for a while, which is the stuff most artists like doing the most anyway

Not here in Mexico, still good business, if you know spanish look up prices, it's stupid, we talking a print that takes 5 hours being like $20 or so.

Now, about your point which I will not ignore: that's even harder democratization because now some dudes can undercut small businesses and sell parts for less, making it a side hustle, it's positive in a way.

[deleted]

Future belongs to world builders.

I had enough, please, some should implement a politician AI to setup an AI party

the thing is, there will be an ai that deals with customers, and that ai will have infinite patience to cater to the specific needs while at the same time use powerful tools to guide the customer to the most profitable option.

Prices for what?

Except so brain dead easy that literally anyone can do it within 5 seconds, which while good also makes any video game worth 0 dollars.

[removed]

Like phone in 1980s: for 10.000$ you get a totally useless brick.

Evolution throws out the expensive garbage and makes prices fair for all.

And the customer will demand to speak to a live human.

[deleted]

And then the customer was a robot.

At some level, this is true. But it will greatly increase the productivity of someone that has the knowledge needed. Someone not using AI will simply not be able to compete on the market anymore.

[removed]

The problem with this is that you need train the AI with a bunch of data. So where are going to find a lot of data of GOOD politicians?

You can train it with the work to be solved.

Improve rates, reduce corruption, obtain votes

Maybe most profitable, but the ability to capture correct sentiment from a person is something that humans struggle with..

A lot of people can build things. Some can even follow directions. But few can translate customer desire to true expectations...as the classic PM tree swing meme conveys:

tree swing

Well conversely, this opens up a whole new world of rapid prototyping that small businesses never had access to before. Instead of being reliant on the business owners skill, or maybe a single employee, those individuals can now ramp up their productivity to the point where they can service customers that were traditionally out of reach.

Many will, and that will be a premium offering.

Sure, I've heard this expressed that it will make 10x programmers now 100x programmers.

And as an AI using 10x programmer, I couldn't agree more.

ChatGPT gives me such a boost it is getting ridiculous.

I see it the same way as procederual generation: it's a good starting point for an artist but will invariably need an artist to make specific adjustments.

“Make me a game. It has these aspects. Also when a user clicks item X and they have an axe in their inventory and greater than 50% health and there are less than 10 people currently playing and one of the other people playing is an elf who is greater than level 5 but less than level 8, the item should trigger Y. Oh but if the elf is at level 9, then scratch all that because the item should then do Z.”

Yeah good luck describing aspects at a high enough level that the AI can make something usable but in enough detail that it’s actually creating the game you want it to create, without causing unintended side effects.

[removed]

[removed]

Yup. Content is king!

There will always be a large margin of error. Some people just can’t be appeased, regardless of their options, because they don’t actually know what they want.

ChatGPT can write code. Not on the level needed to facilitate full game development but enough to make my starfish pucker.

You just have no clue what you're talking about.

I can ask the ai, "write a gravity simulation" and it can code it in 1 second, and I could plug it into some aspect of my game. I could ask the ai, "write a performant function that constantly checks for and gives coordinates" and plug that into my minimap UI. It's literally faster than typing. I could also give parts of my own code to the ai, and say "I got (x) error. What's wrong with my code?" And the AI will describe in detail what I did wrong, and provide a solution.

Game development is coded in much smaller pieces than you seem to think.

[removed]

Or you could just use an existing library that also takes "1 second".

I think so many examples people bring up are just something that you already can do with few clicks, existing livrary, existing framework, assets, etc.

In reality code is prompts. And you can always abstract code into functions which are like promots.

If prompts are that beneficial to some it sounds like their code has way too much boilerplate.

Silly question, I dont see it available in my open ai account? Where do I get it from?

yeah frankly i have been using it for snippets and small functions. absolutely incredible.

Yep, I'm trying to create a personal assistant. So I'm teaching Davinci 003 how to respond to voice commands and return keywords to trigger python scripts.

I basically feel like I'm programming in plain English. It's easier only because I don't have to know what code to use but I still have to know how the process of teaching. Teaching is not easy.

>obtain votes

Watch out, if you aren't detailed enough it will do anything to get votes.

Making 3D models is super easy and super fast already, the anoying and hard part is making them in a way that doesnt cause glitches, errors or takes a huge toll on the graphic budget of your game.

Currently AI 3D models are pretty basic and have shitty geometry, will take some time til they fixed that. Not saying it doesnt happen but the current version of that AI is more a proof of concept than anything else.

Like sending opposition to the jail. Meh, if they improve indexes and reduce corruption that is fine.

Since a 3D object is a vector object, it would be nice if they could also come out with AI that generates 2D vector art as well, since that would also help to reinforce the 2D art in general. Then 2D vector art could be seamlessly integrated with the 2D raster art.