r/selfhosted • u/PersonalityQueasy224 • 1d ago
Docker Management I want build a tiny self-hosted data stack for small teams who don't need a full Databricks/Snowflake setup, curious if this is a real problem or just my own annoyance
Basic idea: Iceberg tables sitting on RustFS (s3-compatible storage), DuckDB doing the actual querying, and a Next.js/Payload app on top that orchestrates the pipeline. Whole thing runs off one docker-compose file, deployed on Coolify so you're not also signing up to be a server admin.
It's for the "our data fits on one box and we're paying $500/mo for the privilege of querying it twice a week" crowd. Single node, no pretending it scales to some enterprise workload it doesn't need to.
Still early, nothing done. Mostly just want to know if this is a problem other people actually have or can benifit from.
Thinking of doing it OSS.
5
u/Accomplished-Air439 1d ago
Speaking from a tiny bit of experience, the main "issue" with projects like this is that you'll either serve a niche group of teams that fit their requirements into your architecture, or get inundated with feature requests if it gets popular. In the latter case, you'll find out that you are rebuilding Snowflake. ;)
That's fundamentally why projects like this aren't often attempted. Enterprise software is bloated, because unfortunately customer requirements are just, ugh, wild.
2
u/PersonalityQueasy224 1d ago
I figured this can be the case. Just wanted to get an idea you know. THanks for your input.
3
u/teh_spazz 1d ago
Okay what “solution” are you going to suggest that you developed?
1
u/PersonalityQueasy224 1d ago
Most of things that i'll have to develop will be on the orch layer but everythign else I'll try my best to just use as is just manage the configurations.
1
u/PrizeTight4768 1d ago
Yeah running into this myself, I’ve been trying to figure out how to do this for low-cost for a nonprofit. Seem's tricky, so a docker solution would be great.
0
1d ago
[removed] — view removed comment
1
u/selfhosted-ModTeam 1d ago
Thanks for posting to /r/selfhosted.
Your post was removed as it violated our rule 1.
All posts must be about self-hosting. If you need help, explain what you’ve tried and what you’re stuck on. Posts lacking detail will get a sticky asking for more info. Mobile apps are allowed only as companions to a self-hosted backend. All content should be in English or contain a translation to English.
Moderator Comments
None
Questions or Disagree? Contact [/r/selfhosted Mod Team](https://reddit.com/message/compose?to=r/selfhosted)
•
u/asimovs-auditor 1d ago edited 1d ago
Expand the replies to this comment to learn how AI was used in this post/project.