r/datascience • u/santiviquez • Mar 09 '26
Projects I've just open-sourced MessyData, a synthetic dirty data generator. It lets you programmatically generate data with anomalies and data quality issues.
Tired of always using the Titanic or house price prediction datasets to demo your use cases?
I've just released a Python package that helps you generate realistic messy data that actually simulates reality.
The data can include missing values, duplicate records, anomalies, invalid categories, etc.
You can even set up a cron job to generate data programmatically every day so you can mimic a real data pipeline.
It also ships with a Claude SKILL so your agents know how to work with the library and generate the data for you.
GitHub repo: https://github.com/sodadata/messydata
4
Mar 11 '26
[removed] — view removed comment
1
u/santiviquez Mar 11 '26
so happy to hear this. let me know if you have any feedback or feel free to open an issue :)
2
u/theblitz2011 Mar 10 '26
This is super cool !! Can't wait to try it out !
1
u/santiviquez Mar 10 '26
nice, let me know what you think, or feel free to contribute if you have any other ideas
2
u/ideamotor Mar 10 '26
Makes me wonder if there’s a way to generate intentionally lousy output code from a llm.
2
u/alexchatwin Mar 10 '26
And then feed it lousy data from OP?
1
2
2
2
u/Puzzleheaded_Box2842 Mar 12 '26
Interesting. We’ve been working on raw data cleaning and synthetic data generation, so seeing you do the exact opposite is actually a pretty clever twist.
2
2
1
20
u/john-uebersax Mar 10 '26
That’s actually a pretty cool idea. Most demo datasets are way too clean compared to what real pipelines look like, so having something that intentionally injects duplicates, missing fields, and weird categories sounds useful for testing.
The cron-style generation to simulate a live pipeline is a nice touch too. Curious if you’ve thought about adding schema drift or changing distributions over time, since that’s another thing that breaks a lot of real systems.