Investigate SQLite- and RocksDB-based storage backends

[cdata]

In very recent history we have run into a number of stability issues with Sled, our resident K/V storage backend that is used in all non-web-based clients today. Specifically: we have found ourselves stuck with a corrupted database that is unable to be opened, and at this time there is no recovery path.

It's possible that these stability issues are related to a known issue in Sled:

• either always use fsync on btrfs or use chattr +C on heap and log files spacejam/sled#1351

Of course, we love using Sled for two important reasons:

1. It's a "pure Rust" dependency
2. It's very fast (possibly ~50% speed up compared to SQLite)
   • Related commentary from the Sled author on performance trade-offs

In the end, no amount of performance gain is worth the high-likelihood of database corruption, so we should begin exploring alternatives.

The two best candidates at this time are:

• RocksDB, a Sled-like K/V store written in C++ (Rust bindings)
• SQLite (Rust bindings)

In the case of both of these alternatives, we will be taking on a major non-Rust dependency in the stack. And, in the case of SQLite in particular, we will be facing performance trade-offs (mainly to the downside) in exchange for apparent stability.

At any rate, let's implement storage for both of these backends along with subjective and objective tests to measure performance (and, if possible, reliability) of them compared to Sled. It is probably ideal for us to switch to RocksDB if it can be shown to be more stable, because it is almost certainly faster than SQLite. But, if the subjective performance of SQLite is indistinguishable from other backends, then the extra stability would give us some peace of mind.

Aside

Honorable mention for potential alternatives goes to https://github.com/mozilla/rkv, which attempts to offer different backends in the same way that we do. They have a "safe mode" backend which perhaps betrays that Mozilla deals with some of the same challenges that we have faced with this work area overall.

[cdata]

Related work from @jsantell : #549