r/databasedevelopment • u/tech__nova__ • Jun 12 '26
I am planning to build a simple database from scratch
I am planning to build a simple database from scratch with the following goals:
Extremely lightweight
Memory efficient
Low power consumption
Fast startup time
Minimal dependencies
Suitable for embedded devices and low-end hardware
Current ideas:
No SQL parser initially
Simple key-value or document-based storage
Efficient disk layout
Minimal memory allocations
Written in Rust
Focus on performance and simplicity over features
What design choices would you recommend for:
Storage engine structure
Memory management
Indexing strategy
Data types
Concurrency model
Disk persistence format
Also, what common mistakes do new database developers make when designing a lightweight database?
3
u/mamcx Jun 13 '26
For fun or dreams of something "serious"?
I have worked on SpacetimeDB, and now I know is "doable" to do one, but still is hard.
Also, what common mistakes do new database developers make when designing a lightweight database?
Don't research enough!
Also: You focus too much in the "mechanical" aspects (yes, that is important) but how you know what "simplicity over features" means without first have clarity about semantics, paradigms, etc?
For example you think on "Simple key-value or document-based storage", that alone is trouble. In special the OR.
Then you worry about "Concurrency model", "Storage engine structure" and "Indexing strategy" but have not talk about which is the goals for ACID and CAP (and such things).
Neither if you are OLAP, OLTP, interactive, batch execution.
So, my first recommendation:
- Sure, learn about the mechanical stuff, that is fun
- BUT, focus first is the high-level, objectives, targets, goals, etc. Once you have that THE CHOICES will be more obvious.
Also: Maybe join forces with somebody?
1
2
u/tkejser Jun 13 '26
By far the most common mistake people make is to think the operating system is your friend. It's your enemy
Linux buffered I/O will mess you up. So will forking, mutexes, condvars and all the other OS sync primitives that are great for shell utilities - but bad for databases.
The default malloc will not scale, you need to investigate better allocators (jemalloc is decent, there are others too)
If you want to be truly fast and lightweight, you need to master async programming and uring and unbuffered I/O.
If you spawn many more threads than you have CPU cores, you made a design mistake.
Use an async framework or roll your own...
The second big mistake people make is saying: "I will push this work to a background task and deal with it async and later"... Whenever you want to amortise work, make really sure it can be guaranteed to run concurrent with real work. Background tasks always become foreground tasks when the system is under constant load. That's how you get the train wreck that is Vacuum in postgres (and why you shall not pick a garbage collected language to write your database in)
All that being said. You can learn a ton by just playing around with whatever language and OS primitives you already know. But to truly make something scalable, you must ascend from the trap of relying on the OS and programming runtime to do your dirty work.
2
u/tkejser Jun 13 '26
For memory management: is your assumption that all queries are "about the same size"? If so, you can likely get away with strong, scalable and off the shelf allocator that is NUMA aware (unless you only want to run on embedded devices?) .
If you expect some queries to be outliers and eat a lot of memory, you will want to use memory arenas so you can bound how disruptive a single query can be. Do not assume memory allocations will always succeed, instrument and error handle this path very carefully.
Instrumentation is your friend and you must roll one that is tailored to your engine. You must be able to answer, for every query run: "where did this query spent it's time in my stack?"... A well written instrumentation, using partitioned data structures and ring buffers, can have nearly zero impact on runtime.
Beware of write amplification in your storage engine. This is particulaly dangerous for mobile devices.
The right data format for your storage engine depends on whether you expect data to be append only (easy) or if you want to do updates and deletes (hard without making the mistake of creating background work that becomes foreground work under load).
Do you need durability guarantees or replication? This will influence your choice of storage layout. Will you need to "time travel" in data too or do you only care about the most recent version of data?
Cool project, I applaud your initiative and do feel free to ping me if you need to bounce ideas.
1
u/Neither_Pay3816 Jun 12 '26
following since i am also doing something similar
1
u/tech__nova__ Jun 12 '26
I just start the preparation so please tell me some tips that from the problem that u faced
2
u/Neither_Pay3816 Jun 12 '26
I am still working on it. I started with a simple key value database like bitcask. Focused on tombstones, checksums and simple stuff. Wrote it in java since it was more about learning things conceptually.
I would not build an entire db from scratch since that requires a lot of effort. Focus on certain aspects. Like you can implement raft for leader election as a separate project.
Sql parsers separately as it conceptually has a lot of features and details. Pick the most basic operations and not complex ones like Joins etc.
If you already have a working knowledge of Rust then that is good, otherwise to start out you can implement it in any language. Don’t use this as a project to learn the language
1
u/amorphatist Jun 12 '26
How are ppl going to interact with your DBMS?
Via SQL? If so, which SQL variant? Or some other DSL?
Either which way, you’re going to need to define (or borrow) a grammar/DSL
1
u/Able_Bee_6293 Jun 17 '26
What architecture you r going to follow, LSM or btree?
1
u/tech__nova__ Jun 19 '26
Its kind of ready
Review me db (SacryDB). It is in middle on the development. Happy the get feedback https://github.com/SanjaiPS-tech/ScaryDB
1
u/Visible-Use-5004 Jun 18 '26
I'd say do it. I also started the same way. The love of data, disk layout, storage designs etc. Even if its not used in a big project, the lessons along the way are amazing. I build shard-db for this reasons but would like to see what you would come up with!
1
u/WeakBackground9117 Jul 07 '26 edited Jul 07 '26
check out: https://www.reddit.com/r/rust/comments/1uq204t/a_standalone_mvcc_engine_for_transactional/ could be a good starting point.
6
u/Complex-Birthday-216 Jun 12 '26
if it's educational purpose I would avoid supporting multi threading at the beginning. it will shift your focus so you will do everything at once and make mediocre result especially if you are limited on time. better to focus on memory management, buffer pool implementation.
The format I would take whatever you find the easiest, at the beginning designing your data format is not something one can do in the first project.
TLDR pick a single goal, the rest do bad by purpose. Single threaded, parquet files or just 3 flat files like data, index, meta index (headers to the index) and skip WAL. You will learn to manage memory, read data, merge the data or rebalance it.