You can bet dollars to doughnuts that chatGPT is being run against real environments in training.

You know how it gets things wrong, and you need to keep prompting it then eventually it gets the thing correct?

That happening at scale.

Everything being recorded and every test case where it finally generates working code, that's a new piece of training data.

With just the current dataset and ability to feed known good answers back in this could bootstrap itself up in capability.

But of course it's not just using the data that's being ground out internally, it's also going to be training on all the conversations people are having with it right now.