I see the penny hasn't dropped yet that: humans are doing (roughly) the same dumb thing these models are doing. Humans are predisposed to not notice that though.
All humans do dumb things, of that we have no doubt. But are the dumb things qualitatively the same as those that AIs do? I don't think so. That's essentially the entire problem. We have pretty good ideas about the ways humans make mistakes. Its pretty much the point of all fiction!
AIs fail in new and unpredictable ways. Nobody is saying humans are infallible.
Finally, because I suspect some people are forming tribalism around this, this doesnt to mind my say AI is Good(tm) or Bad (tm). It literally says its going to be weird.
