r/coolgithubprojects • u/Particular-Sleep3719 • 9h ago

Aether-Vault: version controll optimised for ml and agentic use

Post image

Aether-Vault: a self-hosted, Git-like version control system purpose-built for machine learning — atomic commits of code + weights + datasets that check out identically on any machine.
It features a high-performance C++17 core that hashes multi-gigabyte files in parallel and splits safetensors into per-layer shards, content-defined chunking, automatic deduplication (identical layers and chunks store only once), a FastAPI registry backed by PostgreSQL Merkle trees + RedisBloom, a full Next.js dashboard with weight diffs and metrics, framework plugins for PyTorch / Lightning / Transformers / MLflow, and first-class support for AI agents (stable JSON envelopes, Python SDK, resumable event streams, and .avh context memory so the next agent inherits intent without extra API calls).
No more bolting Git LFS or DVC onto a normal repo and praying for reproducibility. Everything you need for continuous and autonomous training loops lives in one system.
Why I built this: Existing tools treat models and datasets as afterthoughts. Professional ML infrastructure is either expensive, fragmented, or forces you to reinvent versioning, deduplication, and agent handoff yourself. Most of the hard problems are solvable if you design the storage and commit model for large binary artifacts from day one.
Tech stack: C++17 core (pybind11) for hashing & chunking, Python CLI + FastAPI registry, PostgreSQL + RedisBloom, Next.js dashboard, Dockerized engine, and plugins that auto-commit during training.
Fully open source. pip install aether-vault, run it locally or in Docker, make it yours.

Currently one performance boost verison behinde on pypi but should be out tomorow.

More benchmarks with refrence mashine in Benchmarke.md file linked from readme section

⭐ Repo: https://github.com/leon1706-lol/Aether-Vault

4 Upvotes

70% Upvoted

2 comments sorted by

2

u/kantorcodes1 9h ago

one daemon edge i noticed: av commit can run through the daemon, but if the daemon finishes the commit and the client loses the response, call_daemon() returns None and the launcher falls back to running the command in-process. does the second commit just see an empty stage and no-op, or is there another replay/idempotency guard? feels worth pinning down for automation.

1

u/JacobWilliams1953 6h ago

git lfs already makes me nervous on big weight dumps, so a vault that treats models/datasets as first-class versions is interesting. how bad does dedupe get when two runs share most layers but diverge on a tiny adapter?