Goals
The goal of this project is purely learning, I’m a database geek and it’s time I built my own storage engine to learn more and geek out better (especially to understand folks who know about these systems a lot better than I do now).
As such, this will probably never be something that can be put into prod for any serious usecase.
The things I want out of this project are, in no particular order:
- A deeper understanding of storage, memory and compaction
- A deeper understanding on the levels of durability
- Understanding of syscalls (
fsyncandio_uringare priorities) - Understanding of the actor pattern
- Understanding of concurrency, and a deeper look into Go (mastery is a long ways away)
- Understanding of how SSDs work and how modern databases can use that to improve things
- Becoming somewhat decent at performance and algorithms
This project will have minimal (preferably zero) use of AI, in any parts - research, code or any other process. I intend to do this the hard way in order to learn as much as possible.
What I’m building
I’m not really sure of what the end goal is here, but I have a few directions I want to explore. That said, there’s a few things I want to keep the same:
- This will be a key value store, with key attention to durability
- Will be using LSM trees
- Programming language is Go (despite some issues with lower level stuff)
The directions I want to explore (besides io_uring and the SSD direction, which are a bit more advanced):
- Using the actor pattern instead of multithreading on a core with multiple writers
- Trying thread pinning to use a single thread per cpu core - to eke out the max perf without having any context switches (this may be counterintuitive with the way modern CPUs are designed, but I want to try this anyways)
- Durability first. This means more exploration on the persistence and crash/error recovery parts.
Since these are pretty wide areas, I plan to do this in phases, with each major version focused on a specific paradigm. This will likely span into a multi year project, which I’m fine with.
Knowledge gaps
I know practically nothing about implementing storage engines, except for algorithms and high level details (the frustration with this is a major reason for starting this project).
That said, I’m looking at the WAL and strategies used in implementing WALs and subsequent error recovery methods first. Transactions aren’t something I’m looking to enforce, so I’m going to delegate that to later.
Resources
I’m going to be following a lot of resources on this, since it’s a project with a lot of things to learn. Primarily I’ll be going through books, papers, blogs and implementations (leveldb, rocksdb, badger, sqlite).
Still, I’m looking at two books prominently to understand this problem space better:
- DDIA by Kleppmann
- Database Internals by Petrov
If you’ve got some resources or suggestions, kindly reach out on email or any of the other places I’m available, I really need all the help I can get for this.