> I'd much rather sketch some diagrams on paper and try to implement something that works but is a little ugly and then make a second pass to clean it up and properly architect it.
Fred Brooks said "plan to throw one away, you will anyhow". He was mostly talking about the value of prototyping or what in some circles is called spiking.
> I suppose if you write a lot of the same kinds of software you can estimate up front pretty accurately though.
Not in my experience, and I'm a pro-estimate loony. A lot of products and problems do have fairly predictable elements and it's possible to give useful estimates (eg, add new form element, needs a new field in the endpoint, new column on the table, some validations, tests for the above, no changes to docs ...).
Other problems are closer to pure research and you just accept someone might come back with something some day.
What we argue about in software circles is how wide valley in the middle is.
I think an engineer needs freedom to be able to exercise judgement say "I don't understand this task well enough to estimate it yet."
Why? Speaking from personal experience, when I was a recent grad and got asked for an estimate, said "I don't know" and then got pressed for one, my thought process was "okay... I guess I'll just come up with a number and double it" because thats the advice I'd been given multiple times. This resulted in an estimate that had no relationship with reality and which I knew had no relationship with reality. This was not helpful for communication because...well because it was a lie. Thats the key: unless you give someone the freedom to say "I cannot yet give you an estimate", you can't know if the estimate you get is a lie.
Establishing this freedom is hard:
* One reason that it can be hard to give estimates of simple tasks like adding a new form element/adding a new field in an endpoint/adding a new column on a table is if you don't yet know your tools well. But it can be rather embarrassing to admit this. So the engineer can be reluctant to exercise the backbone to claim this freedom.
* There is a lot of moralizing around estimates. The article On Being a Senior Engineer[1] says "Avoiding responsibility for estimates is another way of saying, 'I’m not ready to be relied upon for building critical pieces of infrastructure.'" so admitting that you don't yet know how to estimate might be admitting that you should be fired.
* This is hopefully an uncommon case, but an engineer might have been unwittingly hired as a senior engineer right out of university because their recruiter told the business he'd known the engineer for years. This exacerbates the previous point.
* A business stakeholder really wants to have an estimate, even if the reality of the world is that engineer can't give them one.
* There aren't any resources for new engineers to learn task-estimation[2]. So if one of your engineers doesn't know how to estimate tasks, you might worry that they'll never learn how, since coaching that engineer to learn estimation is too time-intensive an option for most companies.
* When an engineer is asked "how long will this take?", it isn't obvious from that question that the estimation process requires first identifying the key pieces of the task and the riskiest pieces of this. That relies on knowledge of how to produce estimates: understanding that they key piece of estimation is not the number you get at the end but is the mile-markers you produce along the way.
All of these difficulties hurt the key thing required for a good conversation around estimates: trust. Trust that the person hearing the estimate knows that there is uncertainty. Trust that the engineer will flag up if they run into a pile of complexity.
> I think an engineer needs freedom to be able to exercise judgement say "I don't understand this task well enough to estimate it yet."
Absolutely. This is itself a valuable estimate. Very occasionally engineers will give a very high points value to indicate that the proposed story is simply too complex as written and needs to be decomposed further.
> Establishing this freedom is hard:
It absolutely is. Everything we do at Pivotal that works well is actually hard. Our cheat code for bootstrapping these practices is that we already bootstrapped them years ago. But maintaining a culture is harder than building one, which is a trillion times harder than improving one.
Fred Brooks said "plan to throw one away, you will anyhow". He was mostly talking about the value of prototyping or what in some circles is called spiking.
> I suppose if you write a lot of the same kinds of software you can estimate up front pretty accurately though.
Not in my experience, and I'm a pro-estimate loony. A lot of products and problems do have fairly predictable elements and it's possible to give useful estimates (eg, add new form element, needs a new field in the endpoint, new column on the table, some validations, tests for the above, no changes to docs ...).
Other problems are closer to pure research and you just accept someone might come back with something some day.
What we argue about in software circles is how wide valley in the middle is.