I think so, yes. In that world the dead internet theory would become true and people will become only more dissociated from reality and society, especially so when AI can generate video and audio. The political repercussions are disastrous.

Also, I really love literature (and art in general) and a future where one cannot differentiate a human writer from AI is, frankly, suicidally bleak to me. I can see a future where publishers use AI to read the market and write the right books for maximum profit completely cutting out human authors from the process. I am an aspiring novelist myself and, while the act of writing is intrinsically motivating there is also a massive social component in terms of having a career and having others read your work that would be completely excised from creativity, so there is also a personal component I suppose. Sharing in the creativity of other humans is the main thing that gives life meaning to me personally and to many others, and to have that stripped from life is extremely depressing.

While this is all very speculative I just can't see the rapid advances in AI leading anywhere expect a lonelier, more isolated and chaotic world if it isn't seriously regulated. But all of this can be fixed if we could just identify AI text. Then nothing would change in terms of the place of human creativity in the world, it would be basically like chess, people still devote their lives to it and the community thrives but only because we can discern AI chess playing from human chess playing. Imagine if there were no anti-cheating policies in chess tournaments, no one would ever play chess seriously ever again.

If we could just identify AI output we would get all of the benefits of LLMs without any of the disastrous drawbacks. To me it is the most important issue right now, but people don't even consider it and are outright hostile to the idea, just see the downvotes to my original reply.