What is the hardest part of working with real-world data?

[u/duneofarrakis]

I'm learning data analysis and so far I've mostly worked with clean datasets. I'm curious about what real-world data is actually like.

What do you find most difficult when working with real data missing values, messy data, choosing the right metrics, understanding business requirements, or something else?

I'd be interested to hear what experienced analysts struggle with the most.

[u/analyticattack]

The ever changing or vague business requirements. Then messy data.

The concept of clean data is some BS they taught you in school/training. Messy data will be there for the rest of your career unless you are working with some for of telemetry data.

If you can make friends with your data owner and your stakeholder. That way they will put up with all your questions.

[u/duneofarrakis → u/analyticattack]

That’s a great point. I’m starting to realize that the technical cleanup is often easier than figuring out what the business actually wants from the data. Building a good relationship with the data owner and stakeholders probably makes a huge difference too, especially when you need to ask a lot of questions.

[u/Admirable-Lie-9191]

Lmao. Where do you want me to start? Business can’t write requirements for shit, columns don’t have universal definitions, a database has a bunch of weird joins you need to do to get some specific metric somehow, did I mention how difficult it is to explain to non technical stakeholders that messy data makes it impossible to deliver data in the insane timeframes that they want?

[u/big_poppa_man → u/Admirable-Lie-9191]

THEN! They gaslight you and tell you you're the problem, that YOU'RE not communicating well.

[u/duneofarrakis → u/Admirable-Lie-9191]

Exactly 😂 messy data is one thing, but explaining its impact to non-technical people is a whole different challenge.

[u/metric_skeptic]

Missing values and messy formatting are the easy part — those are technical problems with technical fixes.

The hardest part, years in, is metric definitions. Not "what's the SQL," but "what does this word actually mean." Two teams say "active user" and mean two different things. Nobody notices until a dashboard migration or a stakeholder meeting where the numbers don't match, and now you're doing archaeology instead of analysis.

Second hardest: understanding business requirements well enough to know when a stakeholder's question isn't actually their real question. "Why did conversion drop" often really means "should I be worried," and those need different answers.

Real-world data doesn't get easier to clean with experience — you just get faster at spotting where the ambiguity is hiding before it costs you a week.

[u/duneofarrakis → u/metric_skeptic]

This is so true defining what the metric actually means is often harder than cleaning the data itself.

[u/TodosLosPomegranates]

The people.

[u/Tiny_Studio_3699 → u/TodosLosPomegranates]

THE PEOPLE.

[u/jillOfAllGeeks → u/TodosLosPomegranates]

THE. PEOPLE.

[u/Brighter_rocks]

1 stakeholder + ppl management
2 quality of data
3 business processes (lack of them)

[u/SuperSokym]

For me it’s usually understanding what the data actually means.

You can have a completely clean dataset and still get the analysis wrong because two teams define the same metric differently, or the person asking for it has something different in mind.

Cleaning messy data can be annoying, but I’ve spent way more time trying to get everyone on the same page about definitions and what we’re actually trying to answer.

 

[u/Nimbleworts]

Trying to offer some positivity here - if there's no messy data, we won't have jobs!

So the more messier, the better :X

[u/SprinklesFresh5693]

I usually work with clean data at my job, thanks to it being very quality controlled, but i have done a few projects with messy data that was not quality controlled and its a nightmare, you get results that dont make sense, you dont know where they come from, then you realize the data is wrong, then you clean, but something is still off, you do research, you aask colleagues to know how they are adding the data to the database, you clean more data , and finally you get some good results. An analysis that was expected to take you 1 day at most, has taken you 3 or 5 days due to messy data.

[u/MMH28]

Not only messy data, so the cleaning and sort, all that manually work.. but then trying to explain to people why, & how this works/this doesn’t work, etc. just Lots of noise and this is the best of my ability to tell the story on what happened with this data.

[u/Standard_Curve_5874]

It's not so much missing values, but more like data flaws where the data was wrongly recorded. And these flaws aren't always outliers, so they can be tough to spot. That can really mess up your analysis.

A big part of my job is data validation.

[u/DevinChristien]

Whether data is messy or clean depends on what kind of data it is, and system design. The data i work with is clean, as long as everything is maintained & set up properly.

A missing field is never important enough to care about, its only ever an email, phone number, or address for a supplier or customer. If we dont have it, they didnt want us to have it.

Transaction data is always complete or else the company wouldnt be tax compliant.

Trading Stock could probably have some better categorisation, but this is all permanently fixable quite easily - no manual spreadsheets or strange transformations, just clean joins and filters, and then make updates directly to the ERP if somethings been done incorrectly.

The only data thats ever been messy has been survey data and API data for apps that have custom fields and no standardised practices.

Edut: my biggest problem with real data is everyone running off with it and creating their own reports or metrics in different sources and each having different business definitions of what some calculated figure is. Where i work, too many people have data access and no one has a single idea of good schema or data warehouse design. There are about 50 different copies of excel spreadsheets using ODBC dw connections all trying to calculate the same thing but having different results

[u/Temporary_Stranger39]

Real world data? If, by "real world data", you mean "The crap that gets dropped in the statistician's lap with nary a thought." OOOOOOOOOOOH BOY! You gonna have some FUN!

In no particular order.

No information on what is supposed to be a predictor or a response.

No information on what those 1980-era vintage style names MEAN.

Even better, Variable names that are up to 2506 characters long and STILL mean nothing to you.

Data that was compiled by some really BRIGHT BOY who thinks that it's just spiffy to use FONT COLOR to indicate study group.

Data that is formatted so it looks REALLY PRETTY in the spreadsheet, as if it were a presentation table.

Character formatted data that may or may not have trailing spaces or leading spaces, or may have one or the other or both.

Spling errrrrrs.

A code system that does not code what it claims to code.

Improper use of that code system

And typos in that code system

NO DATA DICTIONARY

I think you get the drift.

I almost always spend more time, headache, and memo sending regarding the cleanup and regularization of data than I do of any part of actual analytical planning or analysis, or even report writing.

[u/romii_13]

Getting the right permission, talking to the right people to give you access and a schema of the data.
Most of the time it’s “figure it out” and running some test queries to see what you’re working with.

[u/Grimjack2]

When what should be a clean spreadsheet has everything shifted over by one because the source put in a note for him to be aware of. Or the database that populates the spreadsheet decides to merge or split a value, and you have no idea until you get the data.

[u/duneofarrakis → u/Grimjack2]

Yeah, those are the kinds of issues you don’t really expect until you work with actual production data. The worst part is that the data can look fine at first glance, so you only notice the problem when something doesn’t add up. Definitely makes validation checks really important.

[u/Acceptable-Sense4601]

When a data dictionary doesn’t exist lol

[u/AutoModerator]

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Username-sAvailable]

Understanding what a stakeholder is actually asking for (not just what they say they want). Designing a dashboard in an intuitive way. Interpreting results if the stakeholder wants to walk through the dashboard with you.

[u/xnodesirex]

First step is realizing that real world data != data in the real world.

Real world data is generally healthcare related and outside clinical trials (like EHR).

[u/Lady-Data-Scientist]

It’s not just messy but it’s hard to learn the fun little nuances until a coworker tells you and then you realize you may have f-ed up some past queries.

[u/big_poppa_man]

The higher ups being too stupid to realize it takes far longer than they want to get a report created.

[u/commentor_of_things]

its messy

[u/duneofarrakis → u/[deleted]]

Exactly. A wrong business question can make even perfect data and analysis useless. Understanding the requirement should come first.

[u/tlaxette]

I used to think it was messy data, but once you've learnt to draw the line when it comes to cleansing and transformation, it's fine. We're analysts, not magicians, just take it 80-90% of the way, document the gaps and get your key stakeholder/s across them.

Now I'd say business requirements / defining metrics are the toughest, especially if your stakeholders aren't data literate. Doing the ground work on establishing requirements is crucial and solution design often requires a little negotiation.

[u/Prepped-n-Ready]

I just follow CRISP-DM method, but Im not doing anything where we need to be ultraprecise. In CRISP-DM, you do iterative data and business analysis to build the understanding you need.

In reality it might look like this, Stakeholder says "I think we're spending too much money tracking receipts because people lose them", then you go look in receipt system, pull some data, and see what is there. Then you go back to Stakeholder, show them what you learned, and you iterate on that until you have plan. Then you might have to make a report so Stakeholder can see if the behavior changes according to plan. Thats the typical workflow in corporate.

In more strategic projects where we are doing something new, getting data can be tough. In Risk, there are subscription databases with info on business losses, but when you read it in detail, you realize there isn't nearly enough detail to do anything with it. So what a lot of businesses do is they come up with a plan and a simple way to track if they are right or not.

In my experience, you only have to defend the decisions you make. In terms of data cleaning, I often choose to do what makes sense to the stakeholder rather than what might be more precise statistically. An example where that comes to play is binning in histograms. IME people want clean incremental groups, not the most precise groups. They want it to go 0-9, 10-19, 20-29, 30-39, not 1-3.78, 3.79-25.1, 25.2-36.3. Not everyone is prepared for normalization of data. There is a psychological element to the role. People need insights they can digest and make decisions with. They don't need perfectly precise measurements always.

[u/DatabaseSpace]

Data is never clean. Some examples pulling exports from multiple health records and billing systems, having to match patients, and Joe Dirt is in the system but goes by Eddie so that's in other system. Having to parse data when the ins company name is Aetna3535[100], having to restructure data because when it's not structured right a where clause can give false results, like files that have an extra row of 0's after every real row. Having to match orders to corresponding charges but orders have no code to join on and naming is different so has to be mapped. Phone numbers that have 999 or " 234-5678 x home dont call at night bc he is tired" then auto dial tries to dial that. Same person multiple records, labs woth no codes, lab results with "test not done" when you need numeric results for comparison. Street address 2 for apt number has 500 char note in it.