Oh absolutely, benchmarks are great for measuring your own performance and improving it.
I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks.
The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
It is extremely helpful to be able to estimate performance without building out a whole system. For instance 10ms vs 100ms vs 1s latency makes a world of difference as far as what is possible and what component might be the bottleneck. Though, I agree that the “horse race” between several highly optimized and broadly similar libraries may be far less interesting
I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.