Estimate: Guess and Commitment

Guess

So this is a typical situation. Manager comes to you, and says "Good day today, we've just got this new cool feature request... blabla ... This is extremely important feature, our bottom line greatly depends on how quickly we can deliver it. How much will it take you?".

 Being a nice day indeed, with a fresh head in the morning, excited about the great news, you feel noone can stop you. So you say "two weeks" (Thinking you could pull this off in a couple of all-nighters, but of course I will be clever and take reserve to cope with all those risks which of course will not affect Me, the Superhero. Just for the ease of mind.).

 "Great, so two weeks it is", says the manager rubbing his hands. "Please do your best. This is very important. It would be great if you could do this in one week. Now, the details...". And so, now you take a look at all the details. Needless to say, they are not what you expected at all.

 By this time, you start realizing how stupid you were. You barely know the requirements yet, having spent only a few moments to consider them. And still, you have managed to fully commit to do this, in a week (well, 2 weeks actually, but you know the expectation is 1 week, 2 being the worst case). You know you should have asked for time to consider and clarify requirements. But at the moment, you thought you fully understand what business means with this feature, and there is no need to clarify anything. Turns out you were wrong. They have very different view on what this feature means. And this view is at odds with your current architecture. And noone wants to hear about any compromises you are offering.

 And so, bit by bit, your motivation starts dropping. You start remembering how awfully you are underpaid and how your manager had no moral right to ask you to do this oh-so-important feature virtually for free, in the first place. To the level that you take full two months to complete this goddamn feature. It is buggy as hell. But of course it's not your fault. It's the fault of your manager, who was so stupid not to understand how you had to compromise your system's architecture to fit this abomination in. "How could anyone be so short-sighted", you wonder.

Commitment

 The next time you are much more careful. You are asked to estimate this apparently trivial feature. You ask for time to consider requirements. You clarify everything you can. Yes, this is indeed as easy as it looks, you conclude. Having spent 2 hrs estimating, you finally say "this will take me 4hrs, no more". Manager wonders "What took you so long to estimate this? I thought this was a no-brainer you should be able to do in a few hours".

 And indeed, by the 2nd hour, you are already done. Of course, the manager is not happy again. He accuses you of how was that extremely wasteful of you to spend 2 hrs estimating a feature, when you could have actually gotten it done by that time.

You are extremely frustrated by his assertion. Hadn't you put effort in understanding the problem, you'd probably had slipped on one of those edge cases. How can anyone be so blind?

Estimate

The two examples above are edge cases of what estimate could mean. The first is your rough guess. The second is your full commitment. They are very different things, and yet, when one is asked to give estimate, these two are often used interchangeably. One time, you just pull a number off the top of your head. Next time, you are very confident you have all cases covered; you know every little step that has to be made.

It is no wonder managers tend to look at estimates as commitments, while developers would like to take them as guesses. In ideal case, estimate should be a sweet spot somewhere in between. It is a probability distribution. It is your task as a developer to work with manager and find this sweet spot, so that you didn't spend too much time considering all the unlikely cases, and still have a reasonable prediction business can rely on.

As a developer, I am very guilty of doing this vile thing, being very inconsistent with my estimates. One time they are pure guesses based on my mood, the other time they are inevitable fact, based on deep knowledge of the problem. Managers hate that. They like to look at the numbers you give them in spreadsheet, and be confident those numbers are correct. The worst thing that could happen is when your "1" is sometime "2". Oh no, wait, it's more like it's sometimes "20".

In reality, you shouldn't give all your estimates with the same level of confidence. That would be wasteful. You need to find the sweet spot for each problem. And be very explicit where it lies. If you had seen this problem before and you know how to solve it, you should say so. Everyone will be happy. On the other hand, if you have no idea, you should tell your manager this could take years. He will not be happy and ask you to be more specific. You then take a few hours researching the problem, find a blog post describing solution to similar problem, and now you are confident this will take 1-2 weeks with prob. 90%. Manager is happy with such estimate, he asks you to stop researching and start coding.

Epilogue


Estimates are necessary when you want to have predictable results. In that case, we should ask how much predictable these results should be, and base our estimates accordingly. Estimate is probability distribution. It will probably be normal-distribution shaped for non-trivial problems. That means, there will be non-zero probability the task could take 100 years to complete.

If you think that's a nonsense, think about NP-completeness. Let's say the problem you need to solve is in NP. And you need efficient solution, for a general case. It is not known at the time whether the problem is in P. But who knows, maybe in 100 years someone will discover it is indeed in P. So what you do? You go to business and say "Here is and approximation, I am confident it will fit our case". If they say "No, no, we need it exactly as we specified. Continue searching. And please hurry, this thing is taking you too long" - I'm sorry for you mate. As someone said, "change company, or change company".

As you minimize risk, you minimize failure. But so you minimize success. The catch is: the more experience you have, the more you are minimizing failure rather than success when minimizing risk. Why? Because minimizing risk means making choices. The more experience you have, the better choices you can make. It doesn't mean experience will lead you to better choices always. But it is a vector, your North Star. It's like playing chess if you know nothing about openings vs someone who knows his openings. It is very unlikely you will avoid all opening traps and pull it off just because of your "lucky star", "genius mind", or anything of that sort.

As we see in software world today, we are strongly lacking in experience to minimize failure efficiently, while at the same time not to limit our success chances severely. Many successful software companies just made a random thing that worked. There are as many companies who made a random thing that failed. While we are at this level, there is no need for estimates, they just waste time. Build your MVP in a day, if people like it, start iterating based on feedback. and here you have it, a random product that works.