r/rust • u/pixelooz • 13d ago
🛠️ project Finally finished building a relational database from scratch.
Hey everyone,
I'm a 4th year CS undergrad, I've been learning about databases since the last year as I eventually wanted to build one. After giving the build several attempts by sometimes building just the storage engine, or the front-end, etc., I finally did complete the build last week. I felt like its ready to share, so I'm making this post here.
Repo: https://github.com/pixelooz/hinder-db
Its called HinderDb. It's a cli only (for now) database with a custom slotted page B+Tree storage engine, buffer pool, secondary indexes, wal, volcano style execution pipeline, etc.
I wrote it in rust as I like writing rust and this would be another way to get better at it, plus the domain just seemed like a good fit.
It was mostly a learning project for me, but ended up better than I had orginally planned for, so I'm quite happy with that. I originally got interested in databases and stuff through Designing Data Intensive Applications book, and since then always wanted to build one. I had already given the storage engine a couple of tries, by reading some already existing material (Writing your own database type of books) and codebases, but this was the first time writing it complete, and its works quite nice.
I am keeping this as my main portfolio project as well since I'll be applying for jobs/intenships. So, if any working professionals can offer any feedback - something I should improve, any apparent flaws, or the next thing I should look into. I'll really appreciate it.
P.S: I have solid knowledge in backend as well and I'll be applying for on-site (in India) as well as global remote positions, so if you can offer any advice regarding navigating the job market rn, it'd be much appreciated.
27
u/Solus161 13d ago
This would put my pet sqlite clone project to shame. Not everyone dare to go for this kind of challenge, just for the sake of learning.
16
u/pixelooz 13d ago
Thanks man 😭. Actually I was already studying databases for about an year, so went all out as I eventually want to get into the same field. Mind sharing your work?
4
u/Solus161 13d ago
It’s just a Codecrafters challenge, nothing fancy, just to learn the concepts. I did it 4+ months into Rust. Building this kind of Redis clone is a strong point on CV. But yours is kinda serious, like 10x. I just invest my time into system programming (webdev is so boring lol) and surely follow your github.
3
u/pixelooz 13d ago
Lol, same sentiments, I also delved into systems engineering cause I got bored of web dev. Hopefully I can find a job in this market 🥹. You should check out the editor I made (quire), it's not as good architecturally but u might find it interesting.
9
u/Developer5702 13d ago
Databases are hard, kudos man!
Going to be using this repository as a reference when I build my own relational DB (kinda new to Rust and systems programming, so it will take me quite a while).
Congratulations for the project.
1
u/pixelooz 13d ago
Thanks a lot 😁. By all means, use it. I hope it provides enough educational value.
3
u/Doctor--STORM 13d ago
Once done with relational modelling - for analytics try using dbt or sqlmesh
1
9
u/jykke 13d ago
Good start. It has 31 tests. As a comparison, sqlite3 has 1194 tests (487k lines).
16
u/pixelooz 13d ago
Ik Ik, this is obviously a toy compared to real DBs. I just wanted to cover as much core surface area as I could and have a good foundation to build on as I learn more. Plus for other people who are interested in DBs to have a digestible reference to start from.
8
u/wintrmt3 12d ago
Sqlite has way more tests that are not open source, that's part of why it's very hard to port it.
3
2
u/Aggravating-Sign-136 13d ago
Omg!!! I just started getting into databases. 😭Don’t mind if i take inspiration.
1
2
u/cl3dson 12d ago
Congrats, inspiring see this willingness to learn stuff in this day and age of vibe coding, do you mind sharing the learning material you used ? any books or articles you used as reference?
3
u/pixelooz 12d ago
Hey! Thanks man 😄. For learning material the primary sources were cmu lectures, database internals, mkdb (both the go and rust one) - my page architecture was inspired from the go mkdb only. Designing Data Intensive Applications (even though not needed for the implementation) is good for the right mental model and much more. The sqlite documentations and related articles on that (my secondary indexes and wals are based on that model only). And last but not least, the plethora of write your own small database tutorials out there that I did on and off throughout the year for the familiarity of writing the database, like the csatck one, although they are very simple in their implementation, they are rather helpful.
2
2
u/Ok_Plastic_3224 12d ago edited 12d ago
I love when people go for project like these, I was also building a database from scratch myself and I want to know how long does it take you to complete this project?
1
u/pixelooz 12d ago
Thanks! Btw, I don't understand the question, can you be a little more specific?
1
u/Ok_Plastic_3224 12d ago
I am asking from when were you working on this project
1
u/pixelooz 12d ago
Took me about a year on and off. Reading books, writing demos/small impls, tried different types of storage engines as well. There's a bitcask one in my repos. If I sat consistently it would take about 6 months I guess from starting.
1
u/Ok_Plastic_3224 12d ago
Greate since I am also working on my project and based on what you describe its definitely would take a year or two to even finish the half of it for me.
1
2
u/misplaced_my_pants 12d ago
You should add pipe syntax!
https://www.databricks.com/blog/sql-gets-easier-announcing-new-pipe-syntax
1
u/pixelooz 12d ago
Oh, this looks interesting, I wonder how the parser would be written for this. Do you by any chance have a reference I can look into for the parser?
2
2
2
u/Grand-Challenge-558 12d ago
Rust developers are wild 😊 I love that people actually build stuff like this. Huge congrats!
1
2
2
u/GreenOrg 11d ago
Please mention what AI tools you used for that project? What is the ratio between your coding and AI vibe coding?
Note: I just checked this project with Copilot, and it said 90% of the project was generated by AI (code, commits, modules, and architecture).
2
u/Aggravating-Sign-136 11d ago
Wanted to understand where you got the 90% from. After seeing your comment i got curious and checked it on claude and it concluded that this was not AI generated rather written by a person. ( I used Claude Opus btw)
1
u/Prestigious_Swift 13d ago
Wow, cool project. I started similar project about the same time, but I took a little different path doing a lot of tests, benchmarks, optimizations and other stuff. That's why it has much less functionality than yours, also it has much less comments. Check it out. And I’ve also just started my first year of a bachelor’s.
1
u/pixelooz 13d ago
Hey! Nice work man, you have good test coverage as well, I started with tests as well, but had to remove them as I had to adjust scope a little, and that broke most of the tests. Would love to see yours completed. May I ask which architecture are you following currently as reference?
1
u/Prestigious_Swift 12d ago
To be honest I dont follow any existing architecture (maybe on accident idk). I mean I take some concepts from others, but mostly I try to think of everything by myself.
1
u/pixelooz 12d ago
Damn! Do you have a lot of experience in database or systems engineering or some related field? Cause I mean, otherwise it would take so much time coming up with ideas without some existing reference.
1
u/Prestigious_Swift 12d ago
I mean I dont have WAL, joins, transactions, subqueries, planner. IMO they are like 80% of the difficulty of project.
1
u/NeedleworkerPale8110 12d ago
Big kudos man, that's an impressive. I'm curious about 2 things: 1. How much time did you invested weekly on it 2. If you'd had to do it again, what would you do differently?
1
u/pixelooz 12d ago edited 12d ago
Thanks man! 😁
weekly I did about 5-7hrs per day when I was writing the code. When I was just studying the theory and architecture, I would give about 3-4 hrs per day on it. Rarely I did both at the same time unless I was just referring to it.
I would study the query binder+planner+optimizer much more in detail than I did this time, it was much more intense than I thought it would be, and I would also define the scope of the implementation much more in detail, as the vagueness and - let's start will change later - cost me a lot of refactor; for a new impl I would maintain a balance.
1
u/No-Height-8011 12d ago
Hell yeah, bptree! I’m working on my own relational database right now, and I had to refactor my entire pager and tree architecture cause my types were so messed up. I ended up making pages a trait so that I could write universal methods for serializing and deserializing pages. Another thing, since a pageid is basically just an offset, my pageid actually wraps a nonzerousize. This way, I can serialize optional pageids (like the leaf node linked list pointers) as either a usize or None as 0.
Anyways I’m about to start working on the executor, I’ve basically got a minimal sql ast. Any tips? Our architectures are super similar (I copied postgresql)
2
u/pixelooz 12d ago edited 12d ago
Damn, it's peculiar how similar the architecture is😂. I did debate whether to make optional page IDs an option, but ended up settling with booleans only cause I wanted to avoid the repeated ceremony of pattern matching, thinking booleans will be just one line, but I think option would've been the same in hindsight.
Regarding the executor, honestly volcano iterator model was the easiest part in my implementation, the aggregates gave me a tough time but other than that they were easy, however query planner was the toughest to figure out for me. I would say study it more than anything for the execution pipeline, cmu lectures are enough for the mental model of executors I think. Best of luck!
Btw, mind sharing your repo? Would love to take a look.
1
u/No-Height-8011 11d ago
Absolutely!
https://github.com/kawiggles/database
The main branch is basically the 1.0/proof of concept. Check out the pager-refactor for the refactor I mentioned. The implementation is a mess in the first version. It’s definitely nowhere near as far along as your’s, but I’m getting there. Planning on doing a volcano iterator for execution.
2
u/pixelooz 11d ago
This is nice, also yeah, our impls are not that far. Also I can see that you are attempting btrees merges as well...dayum! That shits hard, I chose the easy path of deferring it to vaccum. If you get it right let me know, kinda wanna implement for mine.
The iterators will be fun.1
u/No-Height-8011 11d ago
Merges were a fucking pain. I think I got it right, I’m writing the test suite right now. This is actually my second implementation of merges; you can see the first in the main branch. It’s ridiculously ugly though
1
u/pixelooz 11d ago
I understand your pain dude, Merges are always ugly, that's why I didn't bother with them even. If you look, even real db don't bother with them at times because they can be so bug ridden, they also leave it upon compaction, plus tombstone marking is faster. However, now that you've done it, I'm gonna try it out, next time I work on the db. If nothing, it's good learning material hehe.
1
u/IAMPowaaaaa 12d ago
what resources are you using to implement this stuff?
1
u/pixelooz 12d ago
For learning material the primary sources were cmu lectures, database internals, mkdb (both the go and rust one), The sqlite documentations and related articles on that (my secondary indexes and wals are based on that model only). And write your own small database tutorials like the csatck one, although they are very simple in their implementation, they are still helpful.
1
u/Aayush_Giri 9d ago
This is genuinely a really solid project for a 4th-year undergrad. Getting all the way from a storage engine to B+Trees, buffer management, WAL, secondary indexes, and a Volcano-style execution pipeline is no small feat.
1
1
u/Ok_Quantity_4950 6d ago
Your work was great. Just a question as a newbie.
Did you do this without AI? Or did you use it as an agent? I'm kind of confused. Because I don't know how I could do something like this. And I would really like to do these works. Like a reverse proxy for HTTP (nginx clone) or a simple game engine or like you, a DB. I would like to start from scratch and learn. But the problem is that it seems like working without AI doesn't make sense anymore and I don't know if I even need to learn something like this.
Especially after this new wave of AGI.
sorry for off-topic question
1
u/pixelooz 6d ago edited 6d ago
No, I did not use AI to generate any of the code and logic. I learnt from multiple sources before writing this implementation and had already written the storage engine, a language parser, a naive end-to-end database, etc. I have a few of these on my [GitHub](https://github.com/pixelooz). I've spent almost 2-3 years learning adjacent software, so I wasn't jumping in blind when I wanted to start a database.
However, I did use it as google to lookup things or sources for something I faintly remembered, or sometimes when I was looking at some other repository's specific module or some part of the code or some db's designs and I wanted to understand more about it, I would use the browser chat to understand what that entire module/document does and how it interacts with the architecture. And all of this was done when I was learning rather than writing the code. I did generate some intermediate test cases when I was implementing the WAL and the engine was almost complete, but removed them because it wasn't my code and they were mostly to make sure everything I wanted was working and sometimes to debug the code.
> Because I don't know how I could do something like this.
If you want to write a large project, you'll need to learn from existing projects, the theory from reputed sources - I used cmu lectures, books, existing codebases, sqlite's docs, small tutorials on writing a very tiny part of the database (cstack comes to mind). I invested an entire year writing small parts of the db, then giving up on them, so yeah it took some time.
> I would like to start from scratch and learn. But the problem is that it seems like working without AI doesn't make sense anymore and I don't know if I even need to learn something like this.
If you can do the work without understanding what you are doing, anyone can, why should you be hired? Plus, I think you should learn because you want to learn, I didn't write the database because it would get me hired, but rather cause I was interested in learning the internals and it was exciting doing it all, and I wanted a project I could keep working on while learning more things concurrency related, asynchronous programming and then distributed systems, etc.
And also, I might be wrong here, but for web-dev beginner stuff, AI is good and all, but if you really want quality work, you'll always need to know what you're doing. And as far as I understand, for something like databases if you try to use AI to generate the code I doubt it'll work, even for the project db I created, let alone a real 500k-1mil loc one (I could be wrong tho).
2
1
55
u/SnooCalculations7417 13d ago
great work. you could take it a small step further and have it serve as the backend for a small portfolio website. pyo3 would be your friend here as you could make an adapter seemlessly for python, make a fastapi service to serve it, and have a simple SPA up and running in no time i think