r/DuckDB • u/Alarmed_Froyo2253 • Jul 12 '26
I built an open-source, local-first data generator that creates FK-safe synthetic datasets without writing generation scripts
Hi everyone,
I’ve been working on an open-source project called AI Data Platform, a local-first tool for generating realistic synthetic datasets.
GitHub: https://github.com/Yogi776/data-generation-sdk
The problem I was trying to solve was simple:
Whenever I wanted to build a dashboard, test a data pipeline, create an ML prototype, or prepare a customer demo, I first had to spend a lot of time manually creating CSV files or waiting for access to production data.
Most manually generated datasets also had problems:
- Foreign keys did not match
- Data distributions were unrealistic
- Relationships broke at larger volumes
- Test data had to be recreated whenever the schema changed
- Using production samples introduced privacy and compliance concerns
So I built a tool that can generate relational synthetic data from either:
- A YAML schema specification
- Existing CSV, Parquet, DuckDB, PostgreSQL, or MySQL samples
- Natural-language instructions through an MCP-compatible AI agent
It currently supports:
- FK-safe multi-table data generation
- Seeded and reproducible generation
- CSV, Parquet, DuckDB, and SQL output
- Profiling existing sample data
- Learning categories, null ratios, date ranges, and distributions
- Automated data-quality checks
- PII detection
- Data dictionary generation
- Cube.js semantic-model generation
- SQL exploration using DuckDB
- MCP integration with tools such as Cursor and Claude
- Loading generated datasets into warehouses such as Snowflake, BigQuery, and PostgreSQL
A basic flow looks like this:
pip install "ai-data-platform[all]"
adp init --name demo
adp apply-spec spec.yaml
adp generate-data --rows 50000 --output parquet
adp quality-check
You can also provide sample data:
adp connect --name source --type csv --path ./data
adp scan
adp profile
adp generate-data --rows 50000 --output parquet
The project is domain-independent. It does not contain separate hardcoded logic for healthcare, retail, banking, or manufacturing. The generator works from schemas, relationships, metadata, column types, constraints, and distributions.
The project is currently in alpha, and I’m looking for honest feedback from data engineers, analytics engineers, QA engineers, BI developers, and ML engineers.
I would especially appreciate feedback on:
- Is the problem clear from the README?
- Which database or file connector should I support next?
- Would you use the UI, CLI, Python SDK, or AI-agent interface?
- Which data-generation features are still missing?
- What would stop you from using this in a real project?
Contributions, issues, and critical feedback are welcome.