Something I think about sometimes is "the infinite library", where you have infinite books, each book simply containing some sequence of characters.

Every conceivable book is in the infinite library, but finding anything is basically impossible. Somewhere in there is a book with your life story, all the works of Shakespeare, and every math solution, and extraordinary books which no one ever wrote. But there's no known catalog.

Diffusion models feel like a way to filter out most of the garbage(and likely a lot of good items too) and pointing to stuff that isn't pure nonsense.

Makes me wonder if we could train on NES roms or stuff like that.