We need data from diverse sets of people. From beginners/noobs to mid levels to advanced. Then, filter that data to find meaningful nuggets.
I run up 200-300M tokens of usage per month with AI coding agents, consider myself technically strong as I'm building a technical platform for industry using a decade of experience as a platform engineer and building all sorts of stuff.
I can quantify about 30% productivity boost using these agents compared to before I started using Cursor and CC. 30% is meaningful, but it isn't 2x my performance.
There are times when the agents do something deranged that actually loses me time. There are times when the agents do something well and save me time.
I personally dismiss most of the "spectacular" feedback from noobs because it is not helpful. We have always had easier barriers to entry in SWE, and I'd argue that like 80% of people are naturally filtered out (laid off, can't find work, go do something else) because they never learn how the computer (memory, network, etc.) _actually_ works. Like automatic trans made driving more accessible, but it didn't necessarily make drivers better because there is more to driving than just controlling the car.
I also dismiss the feedback from "super seniors" aka people who never grew in their careers. Of the 20% who don't get filtered out, 80% are basically on Autopilot. These are the employees who just do their jobs, are reliable enough, and won't cry that they don't get a raise because they know they will get destroyed interviewing somewhere else. Again, opinion rejected mostly.
Now the average team (say it has 10 people) will have 2 outstanding engineers, and 8 line item expenses. The 2 outstanding engineers are probably doing 80% of the work because they're operating at 130% against baseline.
The worst will get worse, the best will get better. And we'll be back to where we started until we have better tooling for the best of the best. We will cut some expenses, and then things will eventually normalize again until the next cycle.
I run up 200-300M tokens of usage per month with AI coding agents, consider myself technically strong as I'm building a technical platform for industry using a decade of experience as a platform engineer and building all sorts of stuff.
I can quantify about 30% productivity boost using these agents compared to before I started using Cursor and CC. 30% is meaningful, but it isn't 2x my performance.
There are times when the agents do something deranged that actually loses me time. There are times when the agents do something well and save me time.
I personally dismiss most of the "spectacular" feedback from noobs because it is not helpful. We have always had easier barriers to entry in SWE, and I'd argue that like 80% of people are naturally filtered out (laid off, can't find work, go do something else) because they never learn how the computer (memory, network, etc.) _actually_ works. Like automatic trans made driving more accessible, but it didn't necessarily make drivers better because there is more to driving than just controlling the car.
I also dismiss the feedback from "super seniors" aka people who never grew in their careers. Of the 20% who don't get filtered out, 80% are basically on Autopilot. These are the employees who just do their jobs, are reliable enough, and won't cry that they don't get a raise because they know they will get destroyed interviewing somewhere else. Again, opinion rejected mostly.
Now the average team (say it has 10 people) will have 2 outstanding engineers, and 8 line item expenses. The 2 outstanding engineers are probably doing 80% of the work because they're operating at 130% against baseline.
The worst will get worse, the best will get better. And we'll be back to where we started until we have better tooling for the best of the best. We will cut some expenses, and then things will eventually normalize again until the next cycle.