In order to support multiple backends, LumoSQL needs to have a more general way of matching capabilities to what is available, whether a superset or a subset of what SQLite currently does. This needs to be done in such a way that it remains easy to track upstream SQLite.
The SQLite architecture has the SQL virtual machine in the middle of everything:
vdbeapi.c has all the functions called by the parser
vdbe.c is the implementation of the virtual machine, and and it is
from here that calls are made into btree.c
All changes to SQLite storage code will be in vdbe.c , to insert an API shim layer for arbitary backends. All BtreeXX function calls will be replaced with backendXX calls.
lumo-backend.c will contain:
- a switch between different backends
- a virtual method table of function calls that can be stacked, for layering some generic functionality on any backends that need it as follows
lumo-index-handler.c is for backends that need help with index
and/or key handling. For example some cannot have arbitary length
keys, like LMDB. RocksDB and others do not suffer from this.
lumo-transaction-handler.c is for backends that do not have full
transaction support. RocksDB for example is not MVCC, and this will
add that layer. Similarly this is where we can implement functionality
to upgrade RO transactions to RW with a commit counter.
lumo-crypto.c provides encryption services transparently backends
depending on a decision made in lumo-backend.c, which will cover
everything except backend-specific metadata. Full disk encryption of
everything has to happen at a much lower layer, like SQLite's idea of
a VFS. The VFS concept will not translate entirely, because the very first
alternative backend is based on mmap, and which will need special handling. So we are for now expecting to implement a lumo-vfs-mmap.c and a lumo-vfs.c .
lumo-vfs.c provides VFS services to backends, and is invoked by
lumo-vfs.c may call lumo-crypto for full file encryption
including backend metadata depending on the VFS being implemented.
Backend implementations will be in files such as
This new architecture means:
- Features such as WALs or paging or network paging etc are specific to the backend, and invisible to any other LumoSQL or SQLite code.
- Bug-for-bug compatibility with the orginal SQLite btree.c can be maintained (except in the case of encryption, which no open source users have access to anyway.)
- New backends with novel features (and LMDB is novel enough, for a first example!) can be introduced without disturbing other code, and being able to be benchmarked and tested safely.