Ordered keys & pagination

Run the example

go run ./examples/pagination

page 1: 1 2 3 4
page 2: 5 6 7 8
page 3: 9 10 11 12

The example encodes unsigned IDs as eight-byte big-endian keys, sets MaxKeyBytes: 8, writes twelve records atomically, and reads pages of four. The complete source below is embedded from examples/pagination/main.go during the site build.

Resume after a returned key

Scans have an inclusive Start. Reusing the last returned key would repeat that record. In a bounded byte-key domain, use this application helper:

func KeyAfter(key []byte, maxBytes int) ([]byte, bool) {
    next := append([]byte(nil), key...)
    if len(next) < maxBytes {
        return append(next, 0), true
    }
    for i := len(next) - 1; i >= 0; i-- {
        if next[i] != 0xff {
            next[i]++
            return next[:i+1], true
        }
    }
    return nil, false
}

Call it only with a valid returned record key and the same maximum key length used to open the DB. false means there is no greater key in that bounded domain. This helper is not an SDK method.

For a shorter key, appending zero is the smallest byte-string boundary greater than it. At maximum length, increment-and-truncate avoids constructing an invalid oversized boundary.

Page loop

1. Scan from the saved inclusive start with an explicit positive limit.
2. On success, process the records in order.
3. If no records are returned, stop.
4. Calculate a boundary after the last returned key.
5. Stop if there is no successor; otherwise request the next page.

Do not access records[len(records)-1] on an empty page. Keep the collection and any original upper bound unchanged across requests.

Prefix upper bounds

The exclusive end of a prefix range is different from the next key after a record:

func PrefixEnd(prefix []byte) []byte {
    end := append([]byte(nil), prefix...)
    for i := len(end) - 1; i >= 0; i-- {
        if end[i] != 0xff {
            end[i]++
            return end[:i+1]
        }
    }
    return nil
}

prefix := []byte("account/42/")
records, err := db.Scan(ctx, "events", zeedb.ScanOptions{
    Start: prefix,
    End:   PrefixEnd(prefix),
    Limit: 100,
})

For an empty or all-0xff prefix, nil means an unbounded end. For a nonempty prefix, keep the prefix itself as the start. Appending 0xff is not a universal upper bound because keys may contain that byte and more bytes afterward.

Byte pressure and concurrent writers

ErrLimit on a scan returns no partial page. Reduce the page size and retry from the same start. If even one record exceeds MaxScanBytes, increasing that budget or using an explicitly chosen point-read strategy is required.

Each page is a separate snapshot. Inserts, deletes, and replacements between pages affect later results. A pagination cursor is an application continuation boundary, not a persistent database snapshot or sync cursor.

Complete source

The build reads the runnable repository file directly, including its key encoding, bounded-domain continuation helper, page loop, and cleanup.

// Run from the repository root: go run ./examples/pagination
package main

import (
	"context"
	"encoding/binary"
	"errors"
	"fmt"
	"os"

	zeedb "github.com/Berlindraft/ZeeDB"
)

// KeyAfter returns the next inclusive boundary in a bounded byte-key domain.
// It is application code, not a ZeeDB SDK method. Call it on a valid record key.
func KeyAfter(key []byte, maxBytes int) ([]byte, bool) {
	next := append([]byte(nil), key...)
	if len(next) < maxBytes {
		return append(next, 0), true
	}
	for i := len(next) - 1; i >= 0; i-- {
		if next[i] != 0xff {
			next[i]++
			return next[:i+1], true
		}
	}
	return nil, false
}
func numericKey(id uint64) []byte {
	key := make([]byte, 8)
	binary.BigEndian.PutUint64(key, id)
	return key
}
func main() {
	if err := run(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}
func run() (err error) {
	dir, err := os.MkdirTemp("", "zeedb-pages-")
	if err != nil {
		return err
	}
	defer func() { err = errors.Join(err, os.RemoveAll(dir)) }()
	db, err := zeedb.Open(dir, zeedb.Options{MaxKeyBytes: 8})
	if err != nil {
		return err
	}
	defer func() { err = errors.Join(err, db.Close()) }()
	ctx := context.Background()
	mutations := make([]zeedb.Mutation, 0, 12)
	for id := uint64(1); id <= 12; id++ {
		mutations = append(mutations, zeedb.Mutation{Collection: "events", Key: numericKey(id), Value: []byte(fmt.Sprintf("event-%02d", id))})
	}
	if err := db.WriteBatch(ctx, mutations); err != nil {
		return err
	}
	var start []byte
	for page := 1; ; page++ {
		records, err := db.Scan(ctx, "events", zeedb.ScanOptions{Start: start, Limit: 4})
		if err != nil {
			return err
		}
		if len(records) == 0 {
			break
		}
		fmt.Printf("page %d:", page)
		for _, r := range records {
			if len(r.Key) != 8 {
				return errors.New("invalid event key encoding")
			}
			fmt.Printf(" %d", binary.BigEndian.Uint64(r.Key))
		}
		fmt.Println()
		var ok bool
		start, ok = KeyAfter(records[len(records)-1].Key, 8)
		if !ok {
			break
		}
	}
	return nil
}