You are assuming AI-generated content doesn't get upvoted. It's entirely possible that they even get more upvotes than real humans producing organic content.

If the AI content does better than the human generated content then why would training on it be a problem?

Much of the success of chatgpt comes from RLHF, which can be viewed as training on its own data, but only the data which has been determined by a human to be especially good.

The AI content gets no input from the world outside of the internet except through humans.

Well, if it got upvoted (by humans) then isn't it reasonable to train on it?

Of course, one challenge is to make sure AI doesn't interfere with moderation. Perhaps this whole situation will spur more work on moderation systems.

It seems like I keep seeing more about AI being used for moderation. I guess that'd be a form of AI being trained on the output of AI (or I guess the output of humans that an AI found acceptable). With AIs training each other it seems like, given enough time, it would only take a couple "poison" AIs in the loop to make things get really maladaptive. While this isn't what I'd call comparable to the huge models (written by real data scientists that might know what they're doing), I've seen a few nudges to data being fed into extremely basic self-reinforcing ML models (written by fake data scientists aka grad students in an intro AI class that didn't know what they were doing or why, but nonetheless fully understood the underlying math needed to write working models) start getting weird outputs on iterations much further down the line that became increasingly problematic, until eventually there really was no longer a relationship to the original data.

That's one heuristic you could use, yes.

If your goal is "engaging content", that's probably a reasonable one. If your goal is "human-generated content", it's not.