Yeah, it's definitely confidently wrong at times. I got in the habbit of asking it "are you sure?" after some iffy responses and often it would correct itself.

I stumbled into a few programming questions where it be confidently wrong and refuse to correct itself and it took me as long to verify it as it would have taken to research it myself.

I think people are still surprised that it's right more often than not.

This is my main concern. It's a good tool if you can afford not to take it too seriously. But when using it seriously, professionally, paid, it better be correct at least most of the time. 40% of the Azure related questions I ask are wrong. This makes me hesitant to spend $20. It's not that I can't afford it. But for that money I'd rather have occasional downtime than wrong answers.

It's analogous to measuring the length of something by using the length of your forearm

Yes, it's less precise than using a tape measure, and it often gives an inaccurate answer (sometimes extremely inaccurate) but it's often an efficient thing to do, as long as you are aware of the limitations.

It's only useful when you can verify the accuracy of its responses - but then it's very useful.

E.g. instead of awkwardly googling for stackoverflow questions that are somewhat related to the problem you have, you can just ask for a complete solution tailored to your needs.

In both cases, you'll have to judge the validity of the answer and you'll likely have to modify the result to some degree.

It's about the presentation, the style, the formatting, the syntax of the responses than the actual content.

That is what a lot of writing is but it's not everything