Blog

Why Databases 2024-09-11Permalink

My strong interest in databases emerged around 2016 when I watched a talk by Martin Kleppman entitled "Turning the database inside out with Apache Samza" where he discussed the various challenges of existing database technologies. I fondly recall one slide of his presentation in particular where he begins to talk about the data flow between web browsers and databases.

He highlights a critical problem with modern RDBMS systems: they are not real-time. Rather, one must poll the database to fetch state updates, or one can build some intricate system to track when relevant updates occur. Martin suggests that we should replace all request / response models (i.e. REST APIs) with a subscribe / notify model.

He also discusses event sourcing, replication, and more. Inspired by some of these ideas, I have set out to develop a persistent, efficient, programmable, and scalable data structure store called MinerDB.

My goal is to post more information about my journey to develop this database on this blog at least once per month.

Blog Post Index