Estimating development tasks, projects and effort

How long will…

“… it take to build this new feature for our client? What do you say? 3 days? A week? Two weeks? A month? Come on, you have to be able to give me an estimate! That long? No, surely it can be done faster! I have entire Excel spreadsheets color coded that tell me otherwise. I have seen thousands of developers finish this task on time in extremely complex projects! I think it can be done in 2 days and you can wrap up the documentation while the client takes it for a spin. Testing? Oh don’t worry about it and if you don’t have time for documentation, that’s ok, it’s only a small feature anyway. I look forward to your response. Kind regards, someone”

Some links in the text ahead lead to tracker infested pages. Feel free to find these resources elsewhere if you’re interested in the topic and don’t wish to visit the links I provide.

Illustration of silverworkers in factory

Morin Edmond Silversmith Workshop Illustration

I spent the past few weeks working on a number of different tasks at work and in my free time. I’ve been trying to setup a small home server and use it as a chance to learn more about server management, security and Linux in general. It has been a great experience so far and I’ve documented it extensively and once it is running I might even convert everything I have learned into a guide that would cover the setup process for a home media/cloud server. I also wanted to improve my JS skills (they are non-existent) by completing a Udemy course and reading some books. Meanwhile at work, it has been quite busy as well, with incidents, new feature requests and new projects.

The reason I am providing this expose is to paint a picture of how I currently divide my time. The problem that all of these share though is the problem of estimating time. How long will it take me to complete the JS course? When will I finish that new feature at work? How long does it take to secure an SSH connection to the server? How long will it take to document these things and take notes?

Estimating personal tasks

It has been observed that when working on a new task people will tend to overestimate how long it will take them as a form of defensive pessimism. This will allow them to complete the task ahead of time and feel better about themselves. This isn’t to say that this happens all the time, but it could explain why some people might greatly overestimate time requirements for different tasks. Another reason might be their lack of experience, which in turn will mean that they will try to account for that when giving an estimate. If I don’t know how to build a small, 1-user CRUD app in .Net, I might say that it will take me a month to do so, whereas someone confident with experience would ask additional probing questions and provide a more realistic estimate.

When you are still learning things, another issue that might arise is that you will find yourself engrossed in the task at hand, which in turn might limit your ability to estimate the actual time costs associated with your task. Time estimates are derived from the output of temporal and non-temporal information processing mechanisms in the brain which will be competing for attention. This means that you will be so preoccupied with what you’re doing that you won’t realize how long it actually took you to complete the task. You might have spent a week and you will give that as an estimate at work, but that week might have been 10-12 hours a day as you learned all day long trying to complete the given task.

Previous experience will help to a certain extent, but even in the case of previous experience, if monetary or other external factors are added, people will have a tendency to underestimate the time needed to complete a task. How many boxes can you stack an hour? How many boxes can you stack an hour if I double your pay? How many will you actually stack? The example is simple, but we can see how external motivating factors can influence performance, so these factors can also motivate someone to respond with lower estimates. How long will it take to have feature X implemented? Oh, and btw, if feature X is implemented you get a bonus. Answer? Whatever seems like the shortest feasible period of time.

Personal tasks might not be as influenced by external elements, but even here you might end up fudging the numbers. What if you have good chances for an interview at a new company your friend setup for you, but for that you’ll have to finish learning something. You might say to yourself - “I’ll surely finish this in 3 days” and 2 weeks later you’re still grinding. You might have also failed the interview since this was setup sooner than expected based on that estimate you gave.

I think we need to reflect and understand why we’re undertaking a task to gain perspective and estimate the difficulty and the time associated with the task better. It can also realign your goals further down the line - I want to learn this so that I can do that later on. You might also realize that the task or the process itself will take quite a bit of time, but you motivate yourself to try and push for a shorter turnaround time.

The point I’m trying to make here, is that even when we’re estimating things that are largely within our control, a number of external factors can influence our estimates. In my case, I really felt I advanced well through some of the different lessons in the JS course which made me estimate that I will be done in 4 weeks. 6 weeks later and I’m only halfway through, but I understood that you can’t really control life and things outside of your learning bubble and the same is applicable with any work projects.

Estimations that are forced on you

The above focused more on what I was thinking about personally and how to deal with personal projects and the timelines we build for ourselves. But what happens when unrealistic estimates are generated outside of our control and then we need to “hit” those targets? Well, it can be frustrating - I know that from personal experience. It can be annoying to be told that “this can be done in X-time!”. Who is the person that can do it in that time frame? Or is that the time frame you arbitrarily decided as being close to your goals and your targets, so now I somehow need to achieve it?

This sort of behaviour is seen all too often with companies such as Revolut pushing ridiculous working hours in an attempt to make money, maximize profits and hit those unrealistic targets someone imposed. Even companies such as CD Projekt Red push staff to work overtime to finish a game after saying for years that the would not do that. Why are we here? Because someone said it will be done on this date or within this budget and now we’re all here… working overtime.

I’ve seen plenty of people pulling up their Excel spreadsheets with neatly color code blocks of work, but it’s rare to have someone explain why a given color block represents 2 weeks; or one week; or a day. It’s simply a measure that someone decided to use. I have yet to encounter correct estimates given in a workplace that relied on previous experience, that relied on the total amount of time it would take to deploy a feature, a fix or a new application altogether. To give an example, I fixed an issue with a LinQ query for a project. The project was hosted on Azure and in order to deploy it, the changes would be deployed to a deployment slot and then slots would be swapped once everyone confirmed the issue was fixed. Finding and fixing the issue was around 10-15 minutes. The time it took to arrange someone to actually check and sign off on the change, the time it took to complete change request forms, look for a suitable time to deploy and ensure I’m not interfering with anyone else… well, that is counted in days. I wasn’t under any pressure in regards to this project, however, most people will incorrectly use 10 minutes as the estimate if a similar task appears in the future. We need to become better at accounting for the large number of externalities tied to a project.

So what happens then? People work overtime. Documentation is not written, tests aren’t completed and best practices are not followed. Code breaks and more 10-15 minute fixes come along later down the road. Sometimes we fight the estimates, but more often than not, we’re just put in a position to try and hit these arbitrary targets.

I am lucky enough to work with some great people (and have worked with some great people!) who understand and appreciate a lot of these things. People who are sensible enough to understand these limitations and restrictions. But most of us aren’t. Most people in this field (as far as I have experienced) aren’t part of companies that are using software to reduce estimates to within 5% error margin from 10% margins. Most people work for companies with error margins of 100% when they give out estimates - “It will be done this year!”; two years later the application/feature/fix is still not deployed.

I hope that more people will use those Excel spreadsheets to come up with realistic estimates and adjust future estimates based on previous project timelines to come up with more realistic measures. I hope that more people will speak up and not take jobs at these companies to reinforce a culture of exhaustion and overworking. I understand that not everyone has the priviledge to say no to a job, but unless we start somewhere, unless we push back, this will keep getting worse.

We need to be more vocal about these things and we need to be more generous with ourselves. We also need to learn from our mistakes and understand that different abilities will yield different results. The same measure cannot be applied in a vacuum to all projects and everyone working on those projects.

I might bore you to death in 10 minutes, but someone unskilled will take longer.

Catalin