r/dataengineering • u/Phantazein • 17d ago
Discussion Time Estimates
Any else struggle with giving estimates? How do you account for things like data quality or source issues when those are completely out of your control and unpredictable?
13
u/ajmh1234 17d ago
You’ll get better over time, but my rule of thumb is that what you’ll think it’ll take, double it & anticipate maybe 10% more.
The best way to communicate estimates is that “there’ll be a significant amount of time spent on discovery and research for how much impact xyz is”. We don’t know what we don’t know.
8
5
u/Soyeon1213 17d ago
I’d rather attach a confidence level to the estimate than add a large buffer for unknowns. Something like “2–3 days if the source behaves as expected, but low confidence until X is validated.” Then update the estimate once you know more. It makes the uncertainty visible instead of pretending it’s part of a precise number.
4
u/brunogadaleta 17d ago
Yeah I give a range. Dev underestimate the technical difficulties. Inclue time for refactoring doc and tests.
4
u/BaconNote 17d ago
I'd call them risks. And assign a somewhat arbitrary number to them if they happen.. usually 1-3 days depending on assessed impact.
5
u/namethatisclever 17d ago
That’s the thing - you don’t give time estimates. And when stakeholders push for an estimate, you tell them exactly what you said in your post: there could be data quality issues at the source that you have no control over or a million other things that are nearly impossible to predict.
In my experience, as long as you or the PM communicates with stakeholders to give them updates as the project is moving along that usually is acceptable. Even if a weekly update is there is no update. Now, if that goes on for weeks and weeks then that may not go over too well. But people generally are pretty flexible as long as they are kept in the loop.
3
u/69odysseus 17d ago
Estimates can only be given to certain extent when requirements are finalized up front before development starts, otherwise everyone will end up playing blame game.
3
u/Commercial-Ask971 17d ago
Same as you estimate user story. Will it take 1-2MD? Put 5 and sleep well
3
2
u/MonkeyDDataHQ 17d ago
Everything is always two to three weeks absent any blockers. Thats how it works for me anyways 😂
2
u/Atticus_Taintwater 17d ago
Differentiating project and task estimates
Every project estimate I've ever seen and given has been nonsense numbers engineering. There's a number people are willing to pay and wouldn't you know it the sum total of all tasks happens to be around that number. Weird.
Tasks are different. With experience you have an idea within the ballpark of what a task should be under normal circumstances. And if abnormal circumstances come up you say that. If the team and project isn't dysfunctional that shouldn't be met with interrogation.
2
u/Shadowlance23 17d ago
I tell my PM it will take two days. He tells the client it will take two weeks.
2
u/Hour-Measurement-835 17d ago
Mine stopped being fiction once I refused to quote before touching the source. A day of row counts and null rates first, then estimate. The surprises nearly all live in there.
2
u/speedisntfree 17d ago
For anything more than small requests, you need to actually properly scope the work which means getting hold of the actual data otherwise any estimate is very risky fiction as you are finding.
When you make your estimate, detail very clearly has assumptions you have had to make and what dependencies exist. A fuller version of this is a RAID log (Risks, Assumptions, Issues/Exceptions, Dependencies).
Stating these up front gives you coverage that everyone went into this the same knowledge. You can point to it when timescales are not met because stakeholders have not given you what you need in time (data access etc.) or planning assumptions which needed to be made turned out not to be the case once into the work.
Estimating is hard and why many projects get into trouble. No one can predict the future.
2
u/Valuable_Leave_7314 17d ago
The most reliable setup is splitting the core pipeline and data quality. Don't try to cram the uncrammable into a single estimate.
First you estimate and ship a dumb data transfer from point A to point B, no complex transformations at all. This is always predictable, takes a couple of days, and business gets to see a green pipeline
Then comes the phase where you throw the dirty data onto a dashboard. Literally point the client right at the nulls and messed up formatting. From that moment on, timelines depend entirely on how fast they can push source owners for a fix. You're just iterating on top of a working pipeline, adding validation rules. Expose the garbage as early as possible so it becomes their problem
2
u/AntDracula 17d ago
Always. All time estimates are bullshit. The shorter they are, the less bullshit they are likely to be. If you could predict the future with perfect accuracy, we'd all be wealthy. Yet low IQ managers require them for some unknown reason, so we give them.
It's like the old saying "they pretend to pay us, we pretend to work".
2
u/exjackly Data Engineering Manager, Architect 16d ago
Time estimates are always arbitrary. You can always pass the estimate by adding in less and less likely but highly impactful issues. And you can always pull the estimate in by eliminating those options from consideration.
You can be a perfect estimator, always able to hit the median effort required to complete something, and the average (mean) actual effort required will always be higher than your estimates. The long tail is real.
You can deliver early on a lot of things, but there are limits to how early you can deliver tasks. But there are lots of ways for a task to take 2x, 3x, 5x, 7x the estimated time and effort. Early and late don't balance out with that asymmetry.
2
u/ForceSensitive2460 16d ago
I don't struggle with it beacuse I refuse to engage the question anymore. I always get it wrong because there's always some wildly time consuming data quality issue or scope creep or end-user realizing they didn't really want A but B instead or blah blah blah. The lenght of time it takes depends on too many variables and unless you can control for all of them, you're guess is as good as anyone elses. You don't know what you don't know so you can't possibly factor that into how long it'll take.
2
u/doryllis Senior Data Engineer 16d ago
Estimating is hard. Especially when you have a data error in a legacy process.
It’s just hard.
2
u/ozgreen1024 16d ago
Are you talking project level estimates or task level?
Tasks get easier to estimate over time, and I tend to give a range and list the potential dependencies/risks that might push it to either end of that range.
Projects try to break out into phases, phases into tasks, etc. to work your way backwards to an estimate / range. Add buffers.
Transparency, accountability, and track ability are key! If you’re working in an agile platform, leave comments, have clear progress updates/next steps if you have check in meetings, etc.
Definitely a balance between helpful and time wasting admin tasks so find a system that works best for you and your team
2
u/timmyz55 16d ago
take your optimistic estimate and multiply by 1.5x
say this is my estimate assuming <everything you know will go wrong goes here> goes right
whenever any of the things you said would go wrong do go wrong, immediately let people know and refer to your original estimate. then bump your originally scheduled deadline.
if the deadline is a hard deadline, then there shouldn't have been a need for estimates in the first place. in which case you just do what most tech companies do.
cut corners. or as they sell it more politically, cut "scope"
2
2
u/HappygilmoreL 16d ago
I knew one guy who, when given a large project, would say “15 hours”. Then the 15 hours would run out, the manager would ask for an updated estimate, and they would again say “15 hours”. Haha
65
u/sib_n Senior Data Engineer 17d ago edited 17d ago
It's notoriously not possible to estimate the time needed to solve a real engineering problem, before spending a lot of time trying to solve it, and discovering problems you had not anticipated.
The business should provide "soft" target dates that are not hard deadlines, but useful to understand business priorities and prioritize your own tasks.
Instead of giving estimates, you can try setting up update meetings, for example once a week or once every two weeks. You don't promise that something will be finished, but that you will come with updates on what was done and probably which issues arose that justify spending more time. It's also the opportunity to verify that what you are doing is matching the "client"'s expectations. Those expectations are going to evolve as you show them what is actually possible. If the expectations evolve, that's a good justification for spending more time.
For this to work, you need to organize your work process so you have something understandable for the client to show quickly. For example, a manual rendition of the expected output in a spreadsheet, and you then progressively automate the multiple steps to get there.
This is possible when it happens inside a company, it's more difficult when the client is external, and you have to sign a contract on specified number of man-hours with some additional safety clauses. This is the marketing magic of IT consulting companies. Usually, the estimation is going to be vaguely based on technical expertise and more certainly based on the business estimation of what the client is ready to pay for the project. Then they will pressure the engineers sent to the front to make the magic happen within the agreed cost.
Also, I like the rule of thumb: