rocksdb-haskell-jprupp

Haskell bindings to RocksDB, the embedded persistent key-value store. The package on Hackage is rocksdb-haskell-jprupp; the module to import is Database.RocksDB.

The bindings follow the RocksDB C API closely. On top of it they add bracket-style resource management, so that databases, snapshots and iterators are always released, even when an exception is thrown.

• API documentation
• Changelog
• Report a bug

Contents

• Why these bindings
• Design principles
• Features
• Installation
• Quick start
• Tutorial: an event log
• Concepts
• Usage guide
• Configuration
• Module overview
• Things to watch out for
• Development
• Related projects
• License

Why these bindings

RocksDB is a storage engine: a library that keeps sorted keys and values in files on local disk, with fast writes, fast point lookups and fast ordered scans. There is no server to run. It is what you reach for when SQLite's relational model is more than you need, and an in-memory map is not durable or big enough.

These bindings exist because the Haskoin Bitcoin indexer needed them. It keeps the whole Bitcoin block chain indexed by address, transaction and block, which is hundreds of gigabytes, and serves queries from many threads while new blocks are written. That use shaped the library:

• it must be safe to share one database between many threads;
• resources must never leak, because the process runs for a long time;
• memory outside the Haskell heap must be boundable, because RocksDB's defaults let it grow with the size of the database;
• the hot paths, such as reading many keys at once, must cost as little as the C API allows.

The library is a fork of leveldb-haskell and of earlier RocksDB bindings, maintained as rocksdb-haskell-jprupp.

Design principles

• Stay close to the C API. Functions map nearly one to one to RocksDB's C functions, so RocksDB's own documentation and tuning guides apply directly. Conveniences such as typed keys live in separate libraries (rocksdb-query).
• Scope every resource. Databases, snapshots and iterators are C objects that must be freed once and never touched again. withDB, withSnapshot and withIter free them when their block ends, even on an exception.
• Plain IO. No monad transformer constraints. Lift into your own monad with liftIO.
• Bytes in, bytes out. Keys and values are strict ByteStrings, which is what RocksDB stores. How you serialize them is your choice.
• Don't crash in C. RocksDB crashes the process if you misuse an iterator. The Haskell side checks first and turns those cases into no-ops or Nothing.
• Expose what bounds memory. The few settings that turn RocksDB's memory use into a fixed budget are first-class fields of Config.

Features

Area	What you get
Basic access	get, put, delete on strict ByteString keys and values
Batches	Atomic write of many puts and deletes, across column families
Multi-get	multiGet reads many keys in one call to RocksDB's MultiGet
Column families	Open a database with several independent key spaces; add new ones on open
Snapshots	Consistent point-in-time reads across many queries and iterators
Iterators	Ordered scans, forwards and backwards, optionally pinned to a snapshot
Bounds	Iterators that stop at a key or at the end of a prefix, without reading past it
Tuning	Bloom filters, fixed-length prefix extractor, shared LRU block cache
Limits	Memtable and WAL size limits, open file limit
Administration	Properties (memory use, statistics, SST files), approximate sizes, destroy, repair

What is not included

• No transactions (TransactionDB, OptimisticTransactionDB), merge operators, TTL databases, backups or checkpoints.
• No manual compaction, DeleteRange or SingleDelete.
• No custom comparators: keys always sort by their bytes.

Pull requests that add bindings for these are welcome.

Installation

System library

You need the RocksDB C library and its headers.

# Debian and Ubuntu
sudo apt-get install librocksdb-dev

# Fedora
sudo dnf install rocksdb-devel

# macOS with Homebrew
brew install rocksdb

# Nix
nix-shell -p rocksdb

The library links against -lrocksdb. If RocksDB is installed in a non-standard place, pass its directories with --extra-include-dirs and --extra-lib-dirs (Cabal or Stack).

The headers are needed to build, not only the library: the binding of the Bloom filter policy is compiled against rocksdb/c.h, because RocksDB changed the type of its argument in 6.24 and the header says which one is installed.

Check the library is found

ls /usr/include/rocksdb/c.h        # or $(brew --prefix)/include/rocksdb/c.h
pkg-config --modversion rocksdb    # prints the installed version, if registered

Step by step with Stack

1. Install Stack and the RocksDB library (above).

2. Create a project:

   stack new rocks-demo simple
   cd rocks-demo
   

3. Add rocksdb-haskell-jprupp, bytestring and data-default to build-depends in rocks-demo.cabal.

4. Put the quick start program in src/Main.hs and run it:

   stack run
   

   It creates a database in /tmp/example and prints what it reads back.

If the build fails with cannot find -lrocksdb or rocksdb/c.h: No such file, the C library is not where the compiler looks. Install the development package, or pass --extra-include-dirs and --extra-lib-dirs.

With Cabal

build-depends:
    base
  , bytestring
  , data-default
  , rocksdb-haskell-jprupp

data-default gives you def for the default Config.

With Stack

The package is in Stackage nightly snapshots. If your resolver does not have it, or has an older version, add it to stack.yaml:

extra-deps:
  - rocksdb-haskell-jprupp-2.7.1

With stack --nix, add RocksDB to the Nix packages:

nix:
  packages:
    - rocksdb

Quick start

{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE OverloadedStrings #-}

import Data.Default (def)
import Database.RocksDB

main :: IO ()
main = do
  let cfg = def {createIfMissing = True, bloomFilter = True}
  withDB "/tmp/example" cfg $ \db -> do
    put db "hello" "world"
    get db "hello" >>= print -- Just "world"

    -- atomic batch
    write db [Put "a" "1", Put "b" "2", Del "hello"]

    -- many keys at once
    multiGet db ["a", "b", "c"] >>= print -- [Just "1",Just "2",Nothing]

    -- every entry, in key order
    withIter db $ \it -> do
      iterFirst it
      let loop =
            iterEntry it >>= \case
              Nothing -> return ()
              Just kv -> print kv >> iterNext it >> loop
      loop

Tutorial: an event log

This builds a small append-only event log, to show how a RocksDB schema is designed around sorted keys.

1. Design the keys

RocksDB sorts keys by their bytes. Records read together should therefore sort together. Events are read per user and in time order, so the key is the user followed by a big-endian timestamp, which sorts numerically:

{-# LANGUAGE OverloadedStrings #-}

import Data.ByteString (ByteString)
import qualified Data.ByteString as B
import qualified Data.ByteString.Builder as BB
import qualified Data.ByteString.Lazy as BL
import Data.Word (Word64)

eventKey :: ByteString -> Word64 -> ByteString
eventKey user time =
  BL.toStrict . BB.toLazyByteString $
    BB.byteString user <> BB.char7 '/' <> BB.word64BE time

2. Write atomically

An event and a per-user counter must stay consistent, so they go in one batch:

import Database.RocksDB

logEvent :: DB -> ByteString -> Word64 -> ByteString -> Int -> IO ()
logEvent db user time payload count =
  write db
    [ Put (eventKey user time) payload,
      Put ("count/" <> user) (BL.toStrict (BB.toLazyByteString (BB.intDec count)))
    ]

If the program dies halfway, either both writes are there or neither is.

3. Read a range with an iterator

All events of a user sort together, so one seek and a short scan read them:

eventsOf :: DB -> ByteString -> IO [ByteString]
eventsOf db user = withIter db $ \it -> do
  let prefix = user <> "/"
  iterSeek it prefix
  let go acc = do
        mkv <- iterEntry it
        case mkv of
          Just (k, v) | prefix `B.isPrefixOf` k -> iterNext it >> go (v : acc)
          _ -> return (reverse acc)
  go []

4. Read consistently with a snapshot

If another thread logs events while you read the counter and then the events, the two can disagree. Read both through a snapshot:

report :: DB -> ByteString -> IO (Maybe ByteString, [ByteString])
report db user = withSnapshot db $ \snap ->
  (,) <$> get snap ("count/" <> user) <*> eventsOf snap user

5. Tune for size

Once the log holds millions of events, open it with a memory budget and a Bloom filter for faster point lookups:

let cfg =
      def
        { createIfMissing = True,
          bloomFilter = True,
          maxFiles = Just 1000,
          blockCacheSize = Just (512 * 1024 * 1024),
          cacheIndexAndFilterBlocks = True
        }

Concepts

The database handle

withDB and withDBCF open a database, pass a DB to your function, and close it when the function returns or throws. The DB must not escape that function. It is safe to use from many threads at the same time.

Keys and values

Keys and values are strict ByteStrings. RocksDB sorts keys by their bytes, which is the order iterators walk them in. To store Haskell types, serialize them yourself, or use rocksdb-query.

Column families

A database can hold several key spaces called column families, each with its own configuration. Functions ending in CF act on a given column family; the others act on the default one. The handles are in columnFamilies db, in the order you listed them when opening.

Snapshots

withSnapshot db f passes f a DB that reads from a frozen view of the database. Reads and iterators through it do not see later writes. Writes through it still go to the live database.

Errors

Failures reported by RocksDB are thrown as IOError (built with userError). The message starts with the operation, such as put: ... or open: ....

Usage guide

Column families

withDBCF "/tmp/example-cf" cfg [("users", cfg), ("orders", cfg)] $ \db -> do
  let [users, orders] = columnFamilies db
  putCF db users "alice" "..."
  getCF db orders "1234" >>= print
  write db [PutCF users "bob" "...", DelCF orders "1234"]

On an empty directory, withDBCF creates the database and all the column families. On an existing database, RocksDB needs every column family that exists to be listed. To add a new one, list it and set createMissingColumnFamilies = True in the database Config.

Snapshots

withSnapshot db $ \snap -> do
  a <- get snap "a"
  b <- get snap "b" -- same point in time as "a"
  withIter snap $ \it -> iterFirst it >> iterEntry it >>= print

When the snapshot's lifetime does not fit a scope, use createSnapshot and releaseSnapshot. withIterSnap and withIterSnapCF iterate over a raw Snapshot.

Iterators

An iterator starts out unpositioned. Move it with iterFirst, iterLast or iterSeek, then step with iterNext and iterPrev. iterKey, iterValue and iterEntry return Nothing once it moves past either end.

-- every key that starts with "user:"
withIter db $ \it -> do
  iterSeek it "user:"
  let go acc =
        iterEntry it >>= \case
          Just (k, v) | "user:" `B.isPrefixOf` k -> iterNext it >> go ((k, v) : acc)
          _ -> return (reverse acc)
  users <- go []
  iterGetError it >>= mapM_ (fail . show) -- tell the end apart from an error
  return users

createIterator, createIteratorSnap and destroyIterator manage iterators by hand.

Bounded iterators

The scan above stops when it reads a key that does not start with user:. RocksDB has to find that key first, and that can cost far more than the scan itself. A deleted key is not gone: it leaves a marker that stays until compaction drops it, and an iterator steps over every marker on its way to the next key that is alive. If the keys after user: were a queue that has been emptied, the scan reads the marker of every job that ever was in it.

withIterConfig creates an iterator that RocksDB stops at a bound, without looking past it:

-- every key that starts with "user:", and not one step further
withIterConfig db Nothing def {upperBound = prefixUpperBound "user:"} $ \it -> do
  iterSeek it "user:"
  let go acc =
        iterEntry it >>= \case
          Just kv -> iterNext it >> go (kv : acc)
          Nothing -> return (reverse acc)
  go []

The second argument is the column family, or Nothing for the default one. The IterConfig has four fields, all off in def:

Field	Type	What it does
upperBound	Maybe ByteString	The iterator stays below this key; the key itself is outside
lowerBound	Maybe ByteString	The iterator does not go below this key; the key itself is inside
prefixSameAsStart	Bool	The iterator stays within the prefix of the key it was sent to
totalOrderSeek	Bool	See every key in order, in a database that has a prefixLength

Which one to use:

• upperBound, with prefixUpperBound, when each range gets its own iterator. It needs nothing from the configuration of the database, and the range can be any.
• prefixSameAsStart when one iterator is sent to many ranges, seek after seek, which saves creating an iterator for each: the bound moves with every iterSeek. It needs a prefixLength in the Config, and each range has to be a whole prefix of that length. It also lets RocksDB use its prefix filters to skip the files that have nothing for the prefix. Every key, sought or stored, has to be at least as long as the prefix: see Things to watch out for.
• totalOrderSeek to walk a database that has a prefixLength from end to end. Without it, an iterator there only promises the keys that share the prefix of the key it was sent to.
• withIter when what follows the range is alive, or when there is no range. A bound costs a little when the iterator is created, and gains nothing there.

How much it matters, in this package's own test: one live key, then 20,000 keys that were written and deleted, then another live key. A scan from just after the first one, which finds nothing, in microseconds:

Iterator	Time
withIter	2,600
withIterConfig with an upperBound	2.5

createIteratorConfig manages such an iterator by hand. It returns the iterator together with the action that destroys it, which also frees the bounds: RocksDB keeps pointers to them for as long as the iterator lives.

Properties and maintenance

getProperty db Stats >>= mapM_ BS.putStr
getProperty db BlockCacheUsage
getPropertyCF db cf CurSizeAllMemTables
approximateSize db ("a", "z")

destroy deletes a closed database and repair tries to recover a damaged one. Both take a raw Options pointer, which the public API does not let you build yet, so in practice they are only usable from inside the library.

Configuration

Config has a Default instance: all flags off and all sizes left to RocksDB.

Field	Scope	Meaning
createIfMissing	database	Create the database if it does not exist
errorIfExists	database	Fail if the database already exists
paranoidChecks	database	Check data aggressively and stop on corruption
createMissingColumnFamilies	database	Create column families given to withDBCF that are missing
maxFiles	database	Maximum number of open files (default unlimited)
dbWriteBufferSize	database	Bytes of all memtables together (default unlimited)
blockCacheSize	database	Bytes of an LRU block cache shared by all column families
maxTotalWalSize	database	Bytes of write-ahead log that trigger flushes
writeBufferSize	column family	Bytes per memtable (RocksDB default 64 MiB)
bloomFilter	column family	Full Bloom filter with 10 bits per key
prefixLength	column family	Length of a fixed key prefix for the prefix extractor
cacheIndexAndFilterBlocks	column family	Keep index and filter blocks in the block cache

Database-wide fields are ignored in the configuration of a column family.

Choosing a configuration

RocksDB's defaults are tuned for small databases. Its memory use outside the Haskell heap comes from three places:

1. Memtables, where recent writes collect before being flushed to disk. Capped per column family by writeBufferSize and in total by dbWriteBufferSize.
2. Index and filter blocks of every open table file. Without cacheIndexAndFilterBlocks they live outside any cache and grow with the number of files; maxFiles limits how many are open.
3. The block cache of recently read data. blockCacheSize makes one cache shared by all column families, so memory goes where reads are.

A reasonable starting point for a large database is a block cache of a few hundred megabytes to a few gigabytes, cacheIndexAndFilterBlocks = True, maxFiles of a few thousand, and dbWriteBufferSize of a few hundred megabytes. Enable bloomFilter when most reads are point lookups of keys that may not exist. A filter takes about 1.25 bytes per key, held with the index blocks: in the block cache with cacheIndexAndFilterBlocks, in memory outside it otherwise. A filter belongs to a table file and is written with it, so turning bloomFilter on for an existing database only covers the files written from then on; the others get a filter when compaction rewrites them.

For a large database, setting maxFiles, blockCacheSize, cacheIndexAndFilterBlocks and dbWriteBufferSize puts a ceiling on the memory RocksDB uses outside the Haskell heap. Without them, index and filter blocks of every open file and every memtable can grow without bound.

Module overview

Database.RocksDB is the only exposed module. It re-exports:

Part	Contents
Types	DB, Config, BatchOp, ColumnFamily, Snapshot, Iterator, Property, Range
Opening	withDB, withDBCF
Reading and writing	get, getCF, multiGet, multiGetCF, put, putCF, delete, deleteCF, write
Snapshots	withSnapshot, createSnapshot, releaseSnapshot
Creating iterators	withIter, withIterCF, withIterSnap, withIterSnapCF
Bounded iterators	IterConfig, withIterConfig, createIteratorConfig, prefixUpperBound
Moving iterators	iterSeek, iterFirst, iterLast, iterNext, iterPrev
Reading from iterators	iterKey, iterValue, iterEntry, iterValid, iterGetError
Administration	getProperty, getPropertyCF, approximateSize, destroy, repair

The FFI bindings (Database.RocksDB.C) and helpers (Database.RocksDB.Internal) are hidden modules.

Things to watch out for

• Do not use a DB, snapshot or iterator after the function that gave it to you has returned. The C object is gone and RocksDB will crash.
• Only one process can open a database at a time. A second open fails with a lock error.
• Moving an invalid iterator does nothing. Reposition it with iterSeek, iterFirst or iterLast.
• An iterator that becomes invalid because of corruption looks like one that reached the end. Check iterGetError.
• A scan that stops by looking at the key it has reached steps over every deleted key that follows its range. Where keys come and go, give the iterator a bound with withIterConfig.
• In a database with a prefixLength, an iterator that moves past the prefix it was sent to, or that starts with iterFirst or iterLast, may skip keys. Ask for totalOrderSeek, or stay within the prefix with prefixSameAsStart.
• With prefixSameAsStart, no key may be shorter than the prefixLength: not the keys you seek, and not the keys in the database. RocksDB takes the prefix of a key without checking its length and reads the memory that follows a shorter one, so the iterator stops at an arbitrary place. The binding cannot check this, because an iterator does not know how it was made. Where keys can be shorter, use an upperBound.
• With prefixSameAsStart, iterFirst and iterLast give the first and the last prefix of the database, and no more: the iterator stays within the prefix of the key it finds.
• Destroy an iterator from createIteratorConfig with the action that came with it, not with destroyIterator, which would leave its bounds and read options allocated.
• Calls into RocksDB are safe foreign calls, so they do not block other Haskell threads, but each one costs a little more than an unsafe call.
• Releases before 2.6.2 built no Bloom filters on RocksDB 6.24 or later, whatever bloomFilter said: they passed the bits per key as an integer to a function that had come to take a double. A database written by one of them has no filters in its table files. After upgrading, files get them as they are written, and the memory the filters need comes with them. The size of the filters of a column family is in the rocksdb.aggregated-table-properties property, as filter block size.
• The compiler does not check a foreign import against RocksDB's header. A binding with the wrong type builds and runs, with garbage in place of the argument; the two fixed in 2.6.2 went unnoticed for years.

Development

Stack

git clone https://github.com/jprupp/rocksdb-haskell.git
cd rocksdb-haskell

stack build                        # build the library
stack test                         # run the tests
stack haddock --no-haddock-deps    # build the API documentation

With stack --nix, the nix section of stack.yaml provides RocksDB.

Nix

nix develop
cabal build all
cabal test all

The shell includes GHC 9.8.4, cabal, haskell-language-server, fourmolu, hlint and hoogle.

package.yaml is the source of the package description. Change it rather than rocksdb-haskell-jprupp.cabal, which hpack generates.

The tests are in test/Spec.hs and use hspec and QuickCheck. Each test runs against a fresh database in a temporary directory, so they need RocksDB installed and nothing else. There are two kinds:

• Examples pin down one behaviour each: what a function returns, when it throws, what a setting changes in the files RocksDB writes. A test of a setting checks an effect that can be observed, such as the size of the filter blocks, because a wrong binding can do nothing without failing.
• Properties apply random writes to the database and to a Map, and require that every way of reading the database agrees with the map: point reads, multiGet, iteration in both directions, seeks, snapshots and column families. The keys are short and drawn from the bytes a binding is most likely to mishandle, zero and 255, so that operations often meet on the same key.

destroy and repair have no tests: they take the Options of the C API, which this package gives no way to create.

Contributing

Pull requests are welcome.

• Format new code with fourmolu, which the Nix shell provides.
• Add tests for the behaviour you change.
• When you add or change a binding in Database.RocksDB.C, read the prototype in rocksdb/c.h argument by argument: nothing else checks the Haskell type against it. If the type of an argument differs between versions of RocksDB, import the function with capi.
• Describe the change in CHANGELOG.md. Versions follow the Haskell Package Versioning Policy.

Related projects

• rocksdb-query stores and queries Haskell types with these bindings.
• haskoin-store is a Bitcoin block chain indexer built on them.

This library started as a fork of leveldb-haskell and of earlier RocksDB bindings. See AUTHORS.

License

rocksdb-haskell-jprupp is released under the BSD 3-Clause License.