Storage architecture

One storage engine

The current engine is single-node and single-shard. Embedded applications and the CLI use the same Go package. Collections are namespaces, not authorization boundaries; independent directories represent separate databases.

Go application / CLI
         |
   Serialized commit
         |
   WAL append + sync
         |
   Memtable publication
         |
   SSTable checkpoint + compaction

Write path

1. Validate and copy the complete batch and calculate final per-key budgets.
2. Checkpoint existing memory state if buffer limits require it.
3. Assign a sequence and append the batch as one checksummed WAL frame.
4. Sync the WAL in durable mode.
5. Publish every mutation under the exclusive database lock, then acknowledge.

Readers use a shared lock, so they cannot see an uncommitted or partially published batch.

Read path

Point lookups check the memtable first. If the key is absent there, a Bloom filter can rule out table access; otherwise a sorted in-memory index identifies the record’s file offset. Disk blocks are checksummed again on reads.

Scans merge matching memtable keys with the ordered table index, reconcile overwrites and tombstones, and enforce result limits. Each call observes one committed state.

Disk layout

data/
  LOCK
  CURRENT
  MANIFEST-00000000000000000042.json
  000001.log
  L1-00000000000000000042.sst

This MVP uses a flat directory and one consolidated table. Despite its .json extension, the manifest contains a checksummed binary frame holding JSON metadata; it is not a plain JSON document to edit by hand.

Format 1

Integers are little-endian. Frames contain a 32-bit payload length, a 32-bit IEEE CRC32 checksum, and the payload. The format caps payloads at 64 MiB; configured write limits are normally smaller.

• WAL: ZEEWAL01, then complete batch frames.
• SSTable: ZEESST01, then one live record per frame, sorted by collection/key.
• Manifest: format version, checkpoint sequence, generated table basename, exact byte length, and record count.
• CURRENT: generated manifest basename and newline.

CRC32 protects against accidental corruption, not adversarial changes. Files and keys are unencrypted.

Checkpointing

The engine merges the memtable with the entire existing table, streams records to a new synced table, and publishes synced manifest/CURRENT replacements via rename. Only then does it retain just the WAL header and reclaim obsolete files.

Tombstones can be dropped because the checkpoint contains all older state and covers older WAL operations. This is not a replication tombstone-retention policy.

Checkpointing is synchronous and holds the write lock. There is no background compaction, block cache, compression, persisted sparse index, or scalable leveled-table design in this version. Startup is O(data size) because it verifies every block.

For the full file-format protocol, see the repository storage specification.