Format status
Format 1 is experimental and versioned. This reference explains the implementation for debugging/review; it is not permission to edit internal files or a stable third-party write protocol. There is no format migration tool.
All multi-byte integers below are little-endian. Keys still sort lexicographically by their bytes; integer endianness inside frames is independent of an application’s key encoding.
Shared frame
| Offset | Width | Field |
|---|---|---|
| 0 | 4 bytes | Unsigned payload length |
| 4 | 4 bytes | IEEE CRC32 of payload |
| 8 | payload_length bytes | Payload |
Zero-length or over-64-MiB payloads are invalid. CRC32 detects accidental corruption; it is not cryptographic authentication.
The reader distinguishes EOF/incomplete final framing from a complete frame with an invalid checksum. A damaged length resembling a torn final frame cannot always be distinguished from a genuine incomplete write.
Batch payload
| Field | Width |
|---|---|
| Commit sequence | 8 bytes, unsigned, nonzero |
| Operation count | 4 bytes, unsigned, nonzero |
| Each delete flag | 1 byte, 0 or 1 |
| Collection length + bytes | 4-byte length, then bytes |
| Key length + bytes | 4-byte length, then bytes |
| Value length + bytes | 4-byte length, then bytes |
Every operation in a decoded batch receives the batch sequence. The decoder checks lengths against remaining payload, rejects empty collection/key and NUL-containing collections, and requires exact payload exhaustion. A delete must have an empty value.
Public write validation additionally checks configured limits before encoding. Recovery checks relevant configured record limits too.
WAL
The fixed eight-byte header is ASCII ZEEWAL01. Complete batch frames follow it in commit order.
One frame is the atomic recovery unit; there is no separately written commit marker. Recovery validates sequences, skips batches covered by the published checkpoint, and replays newer complete consecutive batches.
The active file is 000001.log. Checkpoint reclamation keeps its existing header and truncates covered frames; it is not a multi-segment WAL rotation design.
SSTable
The header is ASCII ZEESST01. Each following frame contains one live record using the batch payload encoding with count 1. Its sequence retains the original record commit version.
Records sort by the internal identity collection + NUL + key. Because collection names cannot contain NUL, this separator does not collide with arbitrary key bytes. Compaction drops tombstones after covering all older state.
There is no persisted Bloom filter, sparse-index footer, compression dictionary, or table block cache. On open, the engine verifies blocks and builds:
- A sorted index with identity, offset, framed length, accounting size, and sequence.
- A Bloom filter sized around ten bits per table record, with seven derived probes.
Lookups use the Bloom filter and binary search, then recheck the selected frame’s checksum and identity/version. Startup reads the entire table; point reads need not load every value into the heap.
Manifest and CURRENT
MANIFEST-<20-digit-sequence>.json contains a checksummed frame of JSON, despite its suffix:
{
"format": 1,
"sequence": 42,
"table": "L1-00000000000000000042.sst",
"table_bytes": 4096,
"records": 12
}The numbers above are illustrative metadata, not the encoded size of a specific example dataset. Exact table bytes and record count protect against truncation at a complete-record boundary. Extra manifest bytes and unexpected format metadata are rejected.
CURRENT points to a strict generated manifest basename. Checkpoint publication syncs new files before CURRENT replacement and reclaims the WAL only after the new checkpoint is published. See recovery for crash boundaries.
Ownership and cleanup
Use a dedicated directory. Successful recovery/compaction can remove obsolete L1-*, MANIFEST-*, and CURRENT.tmp files. Never place unrelated files under these generated prefixes or remove LOCK from an open database.
If a newer engine changes formats incompatibly, it must reject or migrate explicitly. The current implementation does not promise forward reads of unknown formats.