Notes on the Port of SQLite to BDB 18.1.32

Motivation

The work described in this document is part of an ongoing project to test, document and consolidate source code for numerous back-end storage systems that have been ported to SQLite in various ways. There are 4 key-value stores used at scale and widely-ported that have the property of being MVCC. BDB is one of them.

The original port of SQLite to BDB was funded by Oracle, who have ceased development on it as of July 2020. LumoSQL plans to continue this, using the Not-Forking Upstream Source Code Tracker . Once the BDB non-fork configuration is created and tested, and once the first forward-port of BDB 18.1.32 has been completed, then:

  1. This document will be of historical interest only, and
  2. LumoSQL will be where updated versions of SQLite+BDB can be found

At that point, it will be possible to compare the performance and other attributes of the BDB KV store to SQLite's default Btree store and to the LMDB backend.

Versions and Availability

BDB is freely available under the GNU AGPL license. It was formerly Sleepycat BDB, before Sleepycat was purchased by Oracle.

BDB version 18.1.32 was the last to have a port of SQLite to BDB included. From the release notes for BDB 18.1.40:

Changes between version 18.1.32 and version 18.1.40 [..] The SQL API is no longer supported. If you require SQL support you must use Berkeley DB 18.1.32 or earlier.

BDB 18.1.32 contains scripts that patch SQLite 3.18.2 to have BDB support. The scripts are intended to assist with forward-porting to later versions of SQLite.

A convenient place to get the source for BDB 18.1.32 is the LumoSQL mirror . It is also available at an Oracle site that requires a login.

This document will refer to the port as BDB-SQLite, because SQLite was modified to fit BDB, not the other way around. This is the how nearly all of the many projects that provide a different storage backend for SQLite have approached the task.

Licensing Gotchas

Oracle applied the AGPL to the BDB code so that it could no longer be used as an embedded library without all code linked to it also becoming subject to the AGPL. That is the viral nature of a Copyleft License. The GNU Lesser GPL would have avoided this, or indeed the original BDB license. Nevertheless, BDB is truly open source under GNU licensing without any tricks; it is just unhelpful for a library to be licensed as if it were an application.

This definitely means:

  • All code linked to SQLite with a BDB backend is subject to the AGPL
  • Source code present in the same tree as the BDB backend is not subject to the AGPL, so there is nothing to worry about distributing BDB source code
  • LumoSQL contributions to BDB code are covered by the AGPL, as well as the LumoSQL license.

Without offering legal advice in any way, this probably means:

  • Users who have paid Oracle for a commercial licence to BDB code are probably exempt from the AGPL requirement, but do check with your open source lawyer

Running Out of the Box

Comments and documentation for BDB-SQLite suggest it should compile and run on many platforms including some quite niche embedded operating systems. However we have only tested BDB-SQLite on common and current Linux distributions, where the process is:

tar zxvf berkeley-db-18.1.32.tar.gz
cd db-18.1.32/build_unix
../dist/configure --enable-sql_compat
make

This will create:

  1. libdb_sqlXX.{so|la} - A C API library, that exactly mirrors the SQLite C API.
  2. dbsql - A command line SQL interpreter, that exactly matches the default sqlite3 command line interpreter.
  3. libsqlite.{so|la} - A C API library, that exactly mirrors the SQLite C API, and has the same name as the library generated by a SQLite build.
  4. sqlite3 - A command line SQL interpreter, that exactly matches the semantics and name of the default sqlite3 command line interpreter.

Design of the Port

Files and Directories

db-18.1.32/lang/sql/sqlite has the unmodified source for SQLite 3.18.2

db-18.1.32/lang/sql/adaptor has all of the replaced SQLite source files, such as btree.c

db-18.1.32/lang/sql/adaptor/sqlite-patches has 43 patches that modify the SQLite sources. Most of them are very small patches, often just a line or two:

  • inserting DB-specific files into the Makefile (eg the DB-specific pragmas)
  • branding (eg 07_shell_prompt.patch)
  • build fixes for less common platforms (Solaris, VS, Cygwin etc)
  • 11 modifications to the test code that comes with SQLite

db-18.1.32/dist/s_sql_upgrade is a script to help with upgrading the SQLite version

Code Changes

In lang/sql/adaptor the significant files replaced are:

btree.c backup.c btreeInt.h vacuum.c userauth.c userauth.h

There are also placeholder files, containing only function definitons and a tiny amount of code. But most of these are not needed and a cleaner port would probably eliminate them altogether:

backup.h backup.c pager.c pager.h btmutex.c pcache.h (stubs only, no code) pcache.c (stubs only, no code) wal.h (stubs only, no code) pcache1.c (stubs only, no code) wal.c (stubs only, no code)

In addition there are files added to SQLite for entirely BDB-specific functionality. These could probably be removed from the build without causing problems elsewhere:

db_encrypt.c db_pragma.c db_sequence.c db_shell.c