Pack files

Without pack files, each repository chunk is stored as a separate borgstore object. For large repositories this means millions of individual objects, each requiring its own I/O round trip to read or write. On high-latency backends (SFTP, cloud object storage) this overhead dominates backup and restore times.

Pack files address this by grouping multiple chunks into a single store object. A reader that needs one chunk does a partial read (range request) at a known offset instead of fetching a separate file. Store object count drops from one-per-chunk to one-per-pack.

Pack File Format

There is no separate file header. Each blob starts with the 8-byte OBJ_MAGIC (BORG_OBJ), so a forward scanner can locate blob boundaries and identify each chunk using only the pack file bytes with no external index.

Per-blob layout

Each blob is a self-contained unit:

Offset (relative to blob start)  Size              Type     Field
--------------------------------  ----------------  -------  -----
0                                 len(OBJ_MAGIC)    bytes    OBJ_MAGIC = ASCII b"BORG_OBJ"
8                                 1                 uint8    Format version: 0x02
9                                 32                bytes    chunk_id
41                                4                 uint32le meta_size
45                                4                 uint32le data_size
49                                meta_size         bytes    encrypted_meta
49 + meta_size                    data_size         bytes    encrypted_data

chunk_id is the ID hash of the plaintext data (id_hash(plaintext_data)). Storing it in the unencrypted header lets a scanner rebuild the chunk_id → location index without decrypting any blob.

chunk_id is not duplicated into encrypted_meta: the header is its only place in the object. RepoObj.format() puts the object type and the compression bookkeeping into the meta dict (type, ctype, clevel, csize, size, plus psize/olevel when the obfuscate pseudo compressor is used), nothing else. What keeps the plaintext header copy honest is that it is bound into the authentication of both encrypted slots, see below.

The fixed part of each blob header is 49 bytes (REPOOBJ_HEADER_SIZE): len(OBJ_MAGIC) + 1 version + 32 chunk_id + 4 meta_size + 4 data_size. REPOOBJ_HEADER_SIZE = len(OBJ_MAGIC) + 1 + 32 + 4 + 4 = 49

The format version is 0x02 (OBJ_VERSION_HEADER_AAD), the only version RepoObj.format() writes and parse()/parse_meta() accept. It binds the header’s first 41 bytes (OBJ_MAGIC + version + chunk_id -- REPOOBJ_HEADER_AAD_SIZE) into the authentication of encrypted_meta and encrypted_data as additional authenticated data (AAD: data that is authenticated together with the ciphertext, but not itself encrypted). This applies to all borg 2 modes: the AEAD encryption modes (AES-256-OCB, ChaCha20-Poly1305) authenticate it with their AEAD tag, the authenticated-* modes with their MAC, see Modes without encryption. meta_size and data_size are excluded from the AAD. RepoObj.parse() reads both slots, so tampering with either size still fails the check, by changing the length of the slice being read. parse_meta() reads the metadata slot alone: it catches a changed meta_size the same way, but not a changed data_size, which the repair walk described below pins separately. A forged chunk_id, version, or magic byte fails authentication in both.

encrypted_meta and encrypted_data each add a one-byte slot tag on top of the shared header AAD -- b"M" for encrypted_meta, b"D" for encrypted_data -- binding each ciphertext to its slot. This stops an attacker controlling repo storage from swapping the two ciphertexts (adjusting meta_size/data_size to match): decrypting a ciphertext under the wrong slot’s AAD fails authentication.

iter_headers() (used for pack recovery/compaction, see below) reads the header without decrypting, so it does not check header AAD authentication. The repair walk described below is the exception: given a validator it reads and decrypts each metadata slot, and thus does check it.

The fixed 49-byte blob header. meta_size and data_size drive traversal; integrity comes from the content-addressed pack name and the per-blob tag, which authenticates magic/version/chunk_id as additional authenticated data.

A reader locates the next blob by advancing:

next_blob_offset = current_blob_offset + REPOOBJ_HEADER_SIZE + meta_size + data_size

iter_headers() checks every header it walks: it must have OBJ_MAGIC, a supported version, and sizes that keep the blob inside the pack and within MAX_DATA_SIZE. A header that fails these checks means a corrupt pack, and IntegrityError is raised, naming which check it failed. A chunks index rebuild without a repair walk (see below) would be incomplete from that point on, so it turns that into CorruptPack, telling the user to run borg check --repair.

The per-blob magic limits the blast radius of corrupted length fields. The repair walk (iter_headers(validate=...), used when borg check --repair rebuilds the chunks index from the packs) validates every header it walks, reading the metadata slot along with it: the slot’s tag covers the slot itself and the chunk id, and at version 0x02 the header AAD described above, so a corrupted chunk id or meta_size fails it; a corrupted magic fails the magic check, and a corrupted version fails because the version decides which AAD the slot is parsed with. data_size - the one header field outside the tag at either version - must equal csize (the data payload size recorded in the tagged metadata) plus the key’s fixed envelope overhead. A header that fails makes the walk scan for the next blob that validates and resume there, so the blobs after the damaged one are still found; the damaged blob itself is dropped, it can not be read back.

The walk rebuilds the index from the pack as it is: the damaged bytes stay where they are, as a gap no index entry covers. A pack is named by the store hash of its content, so a pack damaged in the store keeps failing the store-level check that borg check runs over packs/, also after borg check --repair has rebuilt the index from it. Rewriting such a pack is repository-level repair, see #10026.

OBJ_MAGIC occurs inside the payloads as well, so the scan accepts a candidate only when it validates like any walked header. Validating needs the key, which borg check always loads: it aborts if it can not.

In the authenticated-* modes the tag binds a blob to its chunk id and nothing else (see Structural Authentication), so validating does not establish there that this repository wrote the blob. These modes also store payloads as they are, so a backed up file can contain something that validates - the blobs of a repository sharing the key, for instance. Such a blob reads back as itself, adding a chunk nothing references, but the extent its data_size claims covers whatever follows it at that offset, which the walk then skips. The scan reaches a payload only after the blob owning it failed to validate, so a corrupt header is what makes this reachable.

Bit flips in the data are caught when the blob is read, on that blob alone.

Blobs follow one another contiguously with no padding:

OBJ_MAGIC | version=0x02 | chunk_id_0 | meta_size_0 | data_size_0 | encrypted_meta_0 | encrypted_data_0
OBJ_MAGIC | version=0x02 | chunk_id_1 | meta_size_1 | data_size_1 | encrypted_meta_1 | encrypted_data_1
...

A pack file: self-describing objects concatenated back to back. Object boundaries are found by walking each 49-byte header (offset += 49 + meta_size + data_size).

Pack ID

The pack ID is the store hash of the pack file’s bytes:

pack_id = store_hash(pack_bytes)

Content-addressing the file by its own bytes makes the name commit to the content, so borgstore can verify and cache it and borg check can detect silent corruption of the stored file.

Namespace

Pack files are stored under the packs/ namespace in borgstore, using a single directory level keyed on the first byte of the pack ID (hex-encoded):

packs/
  00/ .. ff/
    <pack_id_hex>

Pack Index Entry

A pack usually holds many blobs, so locating a chunk needs which pack it is in, where inside that pack its blob starts, and how long the blob is. The ChunkIndex maps each chunk to a full pack location:

chunk_id  →  (..., pack_id, obj_offset, obj_size)

obj_offset is the byte offset of the blob from the start of the pack file and obj_size is the total blob length (header + encrypted_meta + encrypted_data). A reader fetches a single chunk with one range request:

read packs/<hex(pack_id)> at [obj_offset, obj_offset + obj_size)

The full ChunkIndex entry is (flags, size, pack_id, obj_offset, obj_size) (ChunkIndexEntry in borg.hashindex), where size is the plaintext chunk size. While a chunk is buffered in the pack writer but not yet flushed, its entry carries the F_PENDING flag and its pack location is unresolved. When an operation aborts (an exception unwinds out of the repository context), chunks still buffered in the pack writer were never stored: they are discarded together with their pending index entries, while a pack already handed to the store is still recorded if its store succeeded.

The archive pointer write (archives/<archive_id>) is the commit point; a crash before it leaves only unreferenced objects that borg compact reclaims.

Pack data must be stored before any archive pointer references it. The required write order is:

1. Store the pack files to packs/<pack_id> via borgstore. The archive metadata object goes into a (usually tiny) pack of its own, stored last.

2. Store index fragment(s) covering all objects the session stored -- the archive metadata object included -- to index/<index_id> (see Index Namespace).

3. Write the archive pointer archives/<hex(archive_id)>. This pointer write is the sole commit point.

A crash between steps 1 and 2 leaves orphan pack files in packs/. No archive references these chunks; borg compact removes them on the next run.

A crash between steps 2 and 3 leaves a partial index file covering packs not yet committed to any archive. The extra index entries point to valid, fully-written pack data; they are harmless and will be cleaned up by the next borg compact.

A crash after step 3 cannot leave the repository in an inconsistent state. The archive pointer write is the commit point: archives are listed from the archives/ namespace, so data not referenced by any archive pointer is unreachable and treated as garbage by borg compact.

Pack files are removed by borg compact (dropping packs whose indexed objects are all unused, rewriting packs above --threshold and merging tiny packs), borg check --repair (when it drops a defective object), borg repo-compress (Repository.transform_pack stores the re-compressed pack under its new content-addressed name and deletes the old one) and borg debug delete-obj. A single blob cannot be removed from a pack in place: all of these paths write a new pack file without it and then delete the old one, so store-level deletion always operates at pack granularity. borg compact (rewriting, merging) and borg repo-compress skip packs recorded corrupt in cache/checked-packs: the rewritten pack would get a new content-addressed name that passes borg check, hiding the corruption.

Gap bytes

The gaps of a pack are its byte ranges that no chunks index entry covers. They hold chunk copies that were stored again in another pack, and blobs of a backup that crashed before writing its index. A gap blob is superseded when the index maps its chunk id to another location. Equal chunk ids mean equal plaintext, so a superseded blob is redundant, whatever the stored size of the indexed copy (compression and obfuscation padding change it).

Rewriting a pack (compact_pack, transform_pack) drops the superseded gap blobs whose header and metadata slot validate, checked as in the repair walk above (repoobj.object_validator), and copies all other gap bytes into the new pack. Validation covers meta_size and data_size, so a dropped range is exactly one blob. Without a validator (validate=None), no gap bytes are dropped.

The walk over a gap steps from header to header by the blob size each header states. It ends at a header that does not parse or that reaches past the gap. The rest of that gap is kept, and so is a superseded blob that does not validate; both are logged as a warning with the pack id and the offset.

Index Namespace

Chunk-to-location mappings are stored as a separate set of objects under the index/ namespace, called index fragments.

A fragment is a serialized ChunkIndex (a borghash HashTableNT keyed on chunk_id) holding only the pack location; the flags and the plaintext size of each entry are zeroed before serializing. The fragment is stored in the key’s store object envelope: encrypted and authenticated in the encrypting modes, authenticated only in the authenticated-* modes. A fragment’s name is the store hash of the stored envelope, so borg check and borgstore can verify it without the key, like any other content-addressed object:

index/
  <store_hash_of_envelope_hex>

An ordinary backup writes only the entries that are new in that session; a full rewrite (e.g. by borg compact) writes all of them. In both cases the write is split into fragments of at most CHUNKINDEX_FRAGMENT_ENTRIES_MAX (400000 entries, roughly 32MB), so no single fragment gets too large -- not even the one large write a first backup of a big dataset produces. The split selects and sorts the keys one leading-key-bits partition at a time, so the same set of entries always yields the same fragments, no matter in which order the entries were inserted.

Content-addressed naming makes each fragment self-verifying. In the authenticated-* modes, the envelope is deterministic, so the same entries produce the same name. In the encrypting modes, the envelope is randomized, so the same entries produce a differently named fragment each time they are stored. So that writing the same index data twice does not store it twice, the client remembers the plaintext store hash of every fragment it read in the session, and a write (unless forced) skips a fragment whose content is already present in the repository. Duplicate fragments that are left anyway (e.g. two clients consolidating the same small fragments at the same time) are harmless, as the merge (see below) is idempotent; the next borg compact removes them.

Index fragments are write-once; an existing fragment is never modified. The in-memory ChunkIndex is built lazily, on the first access to Repository.chunks: everything under index/ is listed, loaded, authenticated and merged (build_chunkindex_from_repo). Loading does not hash a fragment to verify its name: the authentication of the envelope already proves its content (borg check verifies the names). The merge is commutative and idempotent; order does not matter. It has to succeed for all fragments or not at all, because a partially merged index would be missing chunks that do exist in the repository: a fragment that vanishes mid-merge (a concurrent consolidation replaced it) restarts the merge, and a corrupt one (it fails the authentication or does not deserialize) aborts the command: run borg check --repair to rebuild the index from the pack files. Only the commands that rewrite the whole index anyway, under an exclusive lock (borg compact, borg repo-compress), rebuild it from the pack files instead of aborting.

Because every backup appends a fragment, small fragments would pile up over time. repack_chunkindex() (run at cache close, and by anything that loads the index and persists it, e.g. borg compact) merges the fragments below CHUNKINDEX_FRAGMENT_ENTRIES_MIN (100000 entries, roughly 8MB) into fragments of up to CHUNKINDEX_FRAGMENT_ENTRIES_MAX entries and deletes the small sources. Fragments already within that range are left untouched, so they stay immutable -- and, once index/ is cache-backed, stay cached for every client, instead of being invalidated by an all-in-one consolidation. The merge is deferred until it can seal at least one full fragment, or until more than CHUNKINDEX_SMALL_FRAGMENT_CAP (15) small fragments have accumulated, so a slowly growing fragment is not rewritten on every backup.

borg compact rewrites the index/ namespace as a whole: it determines the live chunks via mark-and-sweep, writes the complete surviving index as bounded fragments, and deletes all the fragments it supersedes.

A deletion that could drop entries -- dropping the index entirely, or the full rewrite above -- is guarded by a marker object, cache/chunkindex-invalid, written before the first deletion and removed after the last one. A single-object delete writes the marker just before it removes the old pack, and borg check --repair writes it after storing packs, before it re-reads them and stores the index; both remove it once the index is stored. While the marker is present, the fragments may be missing entries or point at deleted packs, so they are not merged; the index is rebuilt from the pack files on the next load instead. A consolidation needs no marker: the entries of the small fragments it deletes are already contained in the merged fragments it wrote before deleting them.

If the entire index/ namespace is lost or corrupt, the ChunkIndex can be rebuilt by scanning pack files directly; see Recovery Path.

Recovery Path

The ChunkIndex can always be reconstructed by forward-scanning all pack files in packs/. The archives phase of borg check --repair does that unconditionally (it has to work from the real packs, so it can find archives referencing chunks whose pack has gone missing), and the same rebuild is the fallback whenever the index/ fragments cannot be loaded completely (see Index Namespace).

Each blob’s unencrypted header supplies the OBJ_MAGIC (for re-sync after corruption), the chunk_id, and the size fields needed to locate the next blob. The scan produces a complete chunk_id → (pack_id, offset, length) mapping without decrypting any blob and without the repository key.

Repository Version

Repositories using pack files require repository version 5 or later, and the version is the only gate for the pack format.

Repository.save_config() stores the version in the repository config (see Repository config; currently 5, which also introduced the config object itself). Repository.open() reads it back and, if it is not in Repository.acceptable_repo_versions (currently (5,)), closes the store again and raises InvalidRepositoryConfig -- before any repository data is read.