PackLog: git packs appended into a numbered log

A pack log is Keeper's sole object storage: a numbered append-only file holding git packs back-to-back, never mutated in place. The goal is a store trivially git-exchangeable that still never rewrites packs, so bytes copy verbatim and a dropped branch is just a refs tombstone. The method: keep git's pack encoding but strip the per-pack header and checksum, append object records to the project's tail log, and index every object and pack boundary in the Indices so a valid packfile reframes on the fly.

Numbered append-only logs

A log is named NNNNN.keeper with a 10-char RON64 store-wide sequential seqno, so a file_id names a log uniquely; the file sits in the project shard that wrote it.

One shard per project: each .be/<project>/ flat pool holds all the project's NNNNN.keeper logs, .idx runs, and one refs.
Store-wide seqno: numbering is global, so a file_id names a log; it lives in the single project shard, no per-branch dirs.
Never mutated: objects are only appended and offsets stay stable, so the Indices point at them by (file_id, offset).

Many packs per log, stripped framing

One log holds many packs back-to-back; a new log starts only past a size threshold, so a lone local commit appends to the tail rather than opening a file.

Stripped framing: only object records land in the log; the PACK header and SHA-1 trailer are dropped except at file start.
Cheaply reframed: the wire encoder writes a fresh PACK header, sendfiles the records, and appends a fresh SHA-1 trailer.
No self-sync markers: pack boundaries are found only via the index bookmarks below, a git limitation kept for compatibility.

Intra-pack object order

Within a pack, objects MUST follow git's type order — commits, trees, blobs, tags; keeper makes the implicit git repack grouping an explicit invariant writers enforce and readers trust.

Forward-reference free: a tree never cites a commit, a blob never a tree; delta bases sit earlier in this or a prior pack.
Commits by type index: commits are found via the kv64 type4-keyed index, not a pack-prefix scan — a pack may be blob-only.
Non-decreasing type: strict_order in KEEPPackFeed enforces only non-decreasing type; import copies git's stream, no Feed.

Delta-dependency rules

Two git delta encodings get two base rules; together they keep every project shard self-contained, so a shard copies or recompacts in isolation.

OFS_DELTA is pack-local: the base sits earlier in the same pack, so a pack copies verbatim between stores with no rewrite.
REF_DELTA resolves by hashlet within the single project pool: any earlier object in the shard, no per-dir walk-up.
Cross-project forbidden: a base MUST live in the same project shard, never in another project — every shard stays movable.

Pack bookmarks

Every appended pack also gets a bookmark index entry recording its extent, so per-pack reconstruction for the wire is O(1) — no varint scan, no inflation.

Bookmark: key=file_id20|offset40|type4(PACK), val=obj_count32|byte_len32; the reserved PACK type avoids object collisions.
obj_count sums into the fresh PACK header; byte_len is the record length to sendfile from offset in file_id.
No per-pack hashlet: the pkt-line wire negotiates over object sha1s; per-object hashlets in the Indices still remain.

Epoch recompaction replaces GC

Keeper never rewrites or GCs packs live; dropping a branch is a refs tombstone and its objects linger until an out-of-band epoch.

Drop a branch: write a refs tombstone; objects stay in the shard, no log is deleted or repacked, no live deletion.
Epoch recompaction: a release copies the reachable closure to a fresh project id (beagle→beagle2), leaving dead objects.
Live readers survive: an mmap of a log works until close; the single pool means new lookups never route around a missing dir.