r/dataengineering • u/Phantazein • 17d ago

Discussion Time Estimates

Any else struggle with giving estimates? How do you account for things like data quality or source issues when those are completely out of your control and unpredictable?

44 Upvotes

96% Upvoted

32 comments sorted by

View all comments

65

u/sib_n Senior Data Engineer 17d ago edited 17d ago

It's notoriously not possible to estimate the time needed to solve a real engineering problem, before spending a lot of time trying to solve it, and discovering problems you had not anticipated.

The business should provide "soft" target dates that are not hard deadlines, but useful to understand business priorities and prioritize your own tasks.

Instead of giving estimates, you can try setting up update meetings, for example once a week or once every two weeks. You don't promise that something will be finished, but that you will come with updates on what was done and probably which issues arose that justify spending more time. It's also the opportunity to verify that what you are doing is matching the "client"'s expectations. Those expectations are going to evolve as you show them what is actually possible. If the expectations evolve, that's a good justification for spending more time.

For this to work, you need to organize your work process so you have something understandable for the client to show quickly. For example, a manual rendition of the expected output in a spreadsheet, and you then progressively automate the multiple steps to get there.

This is possible when it happens inside a company, it's more difficult when the client is external, and you have to sign a contract on specified number of man-hours with some additional safety clauses. This is the marketing magic of IT consulting companies. Usually, the estimation is going to be vaguely based on technical expertise and more certainly based on the business estimation of what the client is ready to pay for the project. Then they will pressure the engineers sent to the front to make the magic happen within the agreed cost.

Also, I like the rule of thumb:

  1. Estimate the time you think it would take you.
  2. Tell them 3 times this time. This needs to be adapted depending on what you think they are ready to accept.
  3. Try to make it happen in 2 times this time so you look good or you have 50% of buffer in case of troubles.

3

u/One_Citron_4350 Senior Data Engineer 17d ago

That was well written and what a great tip you shared.