diff options
Diffstat (limited to 'docs/FOG-STORAGE.md')
| -rw-r--r-- | docs/FOG-STORAGE.md | 1507 |
1 files changed, 1507 insertions, 0 deletions
diff --git a/docs/FOG-STORAGE.md b/docs/FOG-STORAGE.md new file mode 100644 index 0000000..00498ac --- /dev/null +++ b/docs/FOG-STORAGE.md @@ -0,0 +1,1507 @@ +# FOG Storage + +Status: Draft 0.1 + +Date: 2026-08-08 + +## 1. Purpose + +This document defines `FOG-STORAGE`, the capability-addressed scattered +storage contract used by FOG native asynchronous services. + +It fixes the storage trust boundaries, pairwise stream model, rotating box +capabilities, storage-epoch manifests, courier envelopes, replica selection, +single-box read and write operations, authenticated replica receipts, +idempotency, request retry and deduplication, empty-read behavior, retention, +tombstones, repair, resource bounds, and conformance gates. + +It also records a non-active integration candidate named +`FOG-STORAGE-CANDIDATE-BACAP-PIGEONHOLE-1`. The candidate evaluates the +published BACAP construction and the single-box portion of the published +Pigeonhole protocol while preserving FOG role separation and packet profiles. +It is not `FOG-STORAGE-1`, has no numeric profile identifier, is not directly +wire-compatible with Katzenpost, is not authorized for a public network, and +does not establish a deployed security claim. + +The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, and MAY describe +normative requirements in the sense of BCP 14 when they appear in uppercase. + +## 2. Scope + +`FOG-STORAGE` owns: + +- storage stream, read-capability, write-capability, and per-box state; +- private per-contact directional mailbox streams; +- immutable public storage-replica manifests; +- storage epochs and replica envelope-key periods; +- fixed Composer-to-courier storage envelopes and replies; +- fixed courier-to-replica and replica-to-replica command bodies; +- deterministic final-shard selection and disjoint intermediate selection; +- single-box reads, data writes, tombstone writes, and authenticated results; +- courier request deduplication and bounded request state; +- idempotent final-replica behavior and replica repair; +- retention, garbage collection, backup deletion, and non-resurrection rules; +- fixed geometry compatibility with FOG-MESSAGING, KEMSphinx, and FOG-WIRE; +- storage overload, flooding, and non-amplification requirements; +- parser limits, failure behavior, lifecycle, and conformance evidence. + +This document does not own: + +- message identities, ratchets, application frames, or message ACK semantics; +- KEMSphinx packet construction, SURB cryptography, or packet replay state; +- adjacent-link Noise framing or connection scheduling; +- Composer local-database encryption and rollback detection; +- the entry capsule, return rendezvous, or offline transfer bundle; +- concrete cover, polling, retry-delay, or traffic-rate distributions; +- anonymous client admission or a general anti-Sybil solution; +- multi-box atomic copy, group delivery, multi-device synchronization, or + permanent archival storage. + +Those contracts belong to `FOG-MESSAGING`, `FOG-SPHINX-PROFILES`, `FOG-WIRE`, +`FOG-COMPOSER`, the entry and return specifications, `FOG-SX`, the cover +profile, future admission work, and future group or bulk-delivery protocols. + +## 3. Security Boundary and Information Exposure + +### 3.1 Composer + +The networkless Composer is the only role that holds stream root capabilities, +derives successive box identifiers and box keys, encrypts or decrypts box +payloads, verifies box authenticity, constructs storage envelopes, and makes +the final decision to advance a stream index. + +The Composer learns its contact relationship, stream direction, box sequence, +message envelope, retry state, replica selection, and detailed local result. +None of those data classes may be delegated to the blind relay. + +### 3.2 Courier + +The courier terminates the final KEMSphinx hop and sees: + +- one fixed storage profile and storage epoch; +- the exact storage-manifest hash; +- two intermediate replica identifiers and envelope-key identifiers; +- a fixed client ephemeral public key, two fixed encapsulations, and one + fixed opaque ciphertext; +- a short-lived hash used only for request deduplication and reply matching; +- coarse acceptance, timeout, and local overload state. + +The courier MUST NOT learn the box ID, stream capability, stream index, +operation type, final shard pair, message ID, contact, application type, +plaintext, data-versus-tombstone state, or empty-versus-hit result. + +### 3.3 Intermediate replica + +Each intermediate replica opens one addressed envelope and learns one fixed +inner storage request, including the box ID, operation type, storage epoch, +operation nonce, and deterministic final shard pair. It does not learn the +Composer network location, contact identity, stream root, stream index, +application type, or message plaintext. + +An intermediate replica MUST NOT persist a mailbox, capability root, or +unbounded request history. It keeps only bounded dispatch and reply state. + +### 3.4 Final replica + +Each final replica learns the box ID, operation type, storage epoch, signed +opaque box record, and the intermediate replica that forwarded the request. +It stores only the fixed authenticated record or tombstone in the selected +storage-epoch namespace. + +A final replica cannot identify the stream to which a box belongs unless the +capability construction, client behavior, collusion, or side information +breaks that unlinkability objective. Repeated reads of the same still-empty +box are necessarily linkable at a final replica and remain a documented +residual risk. + +### 3.5 Blind relay, entry, and mixes + +These roles see only their existing fixed transfer, entry, KEMSphinx, and +FOG-WIRE objects. They do not parse a FOG-STORAGE field. Storage protection +does not replace KEMSphinx or Noise, and those layers do not replace box and +envelope protection. + +## 4. Protocol Invariants + +### STORE-INV-01: No stable public mailbox + +A FOG mailbox is a private stream capability and an evolving sequence of +pseudorandom box IDs. The protocol has no public mailbox name, user lookup, +stable network account, or direct Composer-to-replica endpoint. + +### STORE-INV-02: One writer per stream + +One stream has exactly one active write-capability holder. Bidirectional +messaging uses two independent streams, one per direction. Concurrent copies +of one write state or multiple writers are invalid. + +### STORE-INV-03: Capability authority is explicit + +A read capability can derive future box locations, verify records, and decrypt +the corresponding payloads. A write capability can additionally derive +signing and encryption material, create data records, and create tombstones. +If the selected construction lets a writer derive the read capability, FOG +MUST state that fact and MUST NOT present write-only confidentiality as a +property. + +### STORE-INV-04: Box locations rotate + +Every stream index derives a new pseudorandom box ID and per-box key state. +Online roles MUST NOT receive a stream root or a stable capability from which +they can enumerate the sequence. + +### STORE-INV-05: Four independent replicas minimum + +A claim-bearing deployment has at least four eligible storage replicas. Two +final replicas are selected deterministically for a box, and the Composer +selects two distinct intermediate replicas outside that final pair. Fewer +than four disables the intermediate/final disjointness claim rather than +triggering silent fallback. + +### STORE-INV-06: Immutable data, authoritative tombstone + +The first valid data record at an empty box is immutable. An exact duplicate +is idempotent success. A different data record at the same box is conflict. +Only a valid tombstone from the box writer can replace data, and data can never +replace a committed tombstone within that storage-epoch namespace. + +### STORE-INV-07: Persist before export or advancement + +The Composer MUST atomically persist the next stream state, exact immutable +box record, recovery tombstone, request generation, and outbox state before +export. A reader advances only after the complete storage and owning upper +protocol transition commits. + +### STORE-INV-08: Retry has two distinct layers + +Within one courier request generation, every retransmission uses the exact +same fixed `CourierEnvelope` so the courier can deduplicate it. Every network +transmission uses a fresh KEMSphinx packet, route randomness, entry material, +SURB, private reply token, and rendezvous. A later request generation may +reencrypt the same immutable box operation only under explicit bounded rules. + +### STORE-INV-09: Storage ACK is not message delivery + +Courier acceptance means only that the courier accepted bounded work. +Replica commit means only that the stated replicas durably committed a box +record. Neither means that a recipient Composer fetched, authenticated, +committed, displayed, or read the FOG message. + +### STORE-INV-10: Empty read never advances state + +A signed empty result means only that selected replicas observed no record for +that box and epoch when they processed the request. It does not prove permanent +absence and MUST NOT advance the read capability. + +### STORE-INV-11: Fixed traffic classes + +Reads, data writes, tombstone writes, hits, misses, conflicts, retries, +expected errors, and cover operations use the same profile-fixed KEMSphinx and +FOG-WIRE geometry. Timing normalization remains owned by the cover profile. + +### STORE-INV-12: Bounded retention, not archival storage + +Every record, tombstone, key, dedup entry, retry, repair item, and backup copy +has a profile-defined upper lifetime. A peer-supplied timestamp cannot extend +it. FOG does not promise indefinite offline delivery. + +### STORE-INV-13: No capability at the courier + +The courier MUST NOT receive a stream read cap, stream write cap, per-contact +capability root, or a temporary stream capability from which it can enumerate +boxes. The initial profile therefore excludes Pigeonhole `CopyCommand` and +AllOrNothing processing. + +### STORE-INV-14: No downgrade or dynamic shard substitution + +Profiles, storage manifests, shard count, intermediate count, geometry, and +epoch windows are authenticated. Failure MUST NOT select arbitrary replicas, +reuse an old manifest, reduce the replica count, bypass the courier, or parse a +different storage construction. + +## 5. Terminology + +- **stream**: one single-writer sequence of independently addressed boxes; +- **write capability**: secret state that derives future box signing, + encryption, addressing, and tombstone authority; +- **read capability**: secret state that derives future box addressing, + verification, and decryption authority but cannot create valid records; +- **box index state**: evolving counter and KDF state for one stream direction; +- **box ID**: pseudorandom fixed identifier that also participates in record + verification under the selected capability construction; +- **box record**: one fixed authenticated encrypted payload or tombstone; +- **storage epoch**: storage-specific key and retention period, distinct from + the shorter FOG-PKI network epoch; +- **storage manifest**: immutable authority-authenticated replica set, key set, + profile, geometry, and lifetime for one storage epoch; +- **final replicas**: the two manifest replicas selected deterministically for + one box ID; +- **intermediate replicas**: two distinct non-final replicas selected by the + Composer to hide the final pair from the courier; +- **courier request generation**: one immutable envelope and short-lived + courier deduplication lifetime; +- **operation nonce**: random per-generation value visible only after replica + envelope decryption and bound into final-replica receipts; +- **replica receipt**: final-replica-authenticated statement about one box + operation at one epoch and operation nonce; +- **miss**: authenticated observation that no record was present, not proof of + permanent nonexistence; +- **tombstone**: writer-authenticated empty record that prevents later data + resurrection until the namespace expires. + +All integers defined directly by FOG are unsigned network byte order. Fixed +arrays have exactly the profile-defined length. Reserved and padding bytes are +zero after authenticated decryption. Parsers reject truncation, trailing +bytes, non-zero reserved fields, counter wrap, unknown-critical values, and +lengths other than the exact active profile geometry. + +## 6. Stream and Mailbox Model + +### 6.1 Directional pairwise streams + +One pairwise FOG conversation uses two independent streams: + +```text +Alice writer -> Bob reader +Bob writer -> Alice reader +``` + +Each stream root is unique to one relationship, direction, profile, and +generation. Reusing a stream across contacts, applications, groups, or both +directions is forbidden. + +When Bob gives Alice an inbound mailbox grant, Bob first creates the stream, +persists his reader state, and exports the corresponding writer state to +Alice. Alice can then write the stream and, under the BACAP candidate, derive +its read state and tombstone its boxes. This is acceptable only because that +stream contains Alice's own deposits to Bob and no other contact's data. + +The grant MUST NOT provide access to Bob's other inbound streams, outbound +streams, capability registry, message history, local database, storage backup, +or identity keys. + +### 6.2 Capability issuance + +Before exporting a stream grant, the issuing Composer MUST atomically persist: + +- the exact storage profile and stream generation; +- the initial box index and derivation state; +- the local read state and the exported write state; +- the contact and direction binding; +- the authorized storage-manifest range; +- whether the grant is a stream or single-box grant; +- issue, recovery, replacement, and closure status. + +A crash cannot produce two different grants for one persisted stream +generation. An exported write cap is secret bearer material and receives the +same transfer protection as a contact voucher. + +### 6.3 Capability evolution + +After deriving one box, a conforming implementation stages the next index and +evolving KDF state, then securely deletes the retired per-box secret after the +required data record and recovery tombstone are committed locally. + +The active profile sets a maximum box count far below unsigned 64-bit wrap and +a maximum stream lifetime. Stream renewal creates a new independent root and +is authenticated through the existing FOG-MESSAGING ratchet. + +FOG MUST NOT claim capability forward secrecy or backward unlinkability until +the exact construction, state serialization, deletion behavior, and backup +rules have been reviewed. + +### 6.4 Revocation and loss + +A copied read or write capability cannot be remotely revoked. Revocation +means abandoning the stream, distributing a fresh stream grant only to +remaining authorized parties, and allowing old records to expire or be +tombstoned where safe. + +Lost or rolled-back stream state cannot be reconstructed from online replicas. +A stale Composer restore places the affected stream in `RECOVERY_REQUIRED`. +It MUST NOT resume the old index or probe successive boxes in an attempt to +guess live state. + +### 6.5 One-time drops + +`fog-drop` requires a genuinely attenuated single-box writer grant. The grant +must expose only the signing and encryption authority for one exact box and +must not contain a future stream derivation state. + +The BACAP/Pigeonhole candidate is not activated for one-time drops until a +reviewed upstream or separately reviewed attenuation method supplies this +property. Exporting an ordinary unbounded BACAP write cap and merely asking a +sender to use it once is not a one-time capability. + +## 7. Storage Replica Manifest + +### 7.1 Purpose + +Final-shard selection must not change because ordinary PKI epochs add, remove, +or reorder descriptors while stored data remains live. FOG therefore uses one +immutable `StorageReplicaManifest` for each storage epoch. + +The current, previous, and announced next manifests are carried as complete +authority-authenticated objects by FOG-PKI consensus according to their +activation and retention windows. Consumers MUST NOT merge replica lists from +different manifests. + +### 7.2 Canonical object + +The public object uses `FOG-PKI-CBOR-1` and has this ordered body: + +```text +[ + network_id, + storage_manifest_version, + storage_epoch, + storage_profile_id, + geometry_compatibility_id, + valid_from, + fresh_until, + valid_until, + acceptance_window_id, + retention_profile_id, + cover_profile_id, + shard_function_id, + final_replica_count, + intermediate_replica_count, + minimum_replica_count, + ordered_replica_records, + previous_storage_manifest_hash +] +``` + +Each `ordered_replica_records` entry contains: + +```text +[ + replica_node_id, + node_descriptor_hash, + operator_id, + family_ids, + infrastructure_group_ids, + replica_envelope_algorithm_id, + replica_envelope_key_id, + replica_envelope_public_key, + replica_receipt_algorithm_id, + replica_receipt_key_id, + replica_receipt_public_key, + supported_storage_profile_ids +] +``` + +Replica records are sorted by complete `replica_node_id`, duplicate-free, and +byte-identical for every authority signer. One node, operator, prohibited +family, or prohibited infrastructure group MUST NOT occupy more than one of +the four positions used by one claim-bearing operation. + +The body is wrapped as one registered `FOG-PKI-CBOR-1` signed object and one +authenticated envelope containing independent authority signatures over +identical canonical bytes. It is valid only with the active FOG-PKI quorum. + +### 7.3 Sign-once and transition + +Authorities persist the manifest hash for a storage epoch before releasing a +signature. Signing two different manifests for one storage epoch is +equivocation evidence. + +The next manifest is announced early enough that every replica can generate +its purpose-separated next envelope and receipt keys and every offline +Composer can import it. The exact overlap is profile-defined. Failure to +produce a valid manifest stops new storage work for that epoch. + +Ordinary node removal does not rewrite an active storage manifest. Emergency +revocation can stop new operations to a compromised replica, but cannot +silently remap live boxes. Availability loss and a new manifest are handled as +an explicit storage transition. + +## 8. Storage Profile Registry and Epochs + +### 8.1 Immutable profile + +Every active storage profile maps one non-zero unsigned 32-bit identifier to +exact dependencies and behavior. An identifier MUST NOT be reused after any +primitive, field, size, shard rule, epoch rule, limit, response, or failure +behavior changes. + +An exact profile record includes at least: + +```text +[ + storage_profile_id, + capability_specification_and_revision, + capability_primitive_suite_id, + capability_encoding_id, + box_id_length, + box_signature_length, + box_payload_aead_id, + replica_envelope_specification_and_revision, + replica_envelope_suite_id, + replica_receipt_signature_suite_id, + envelope_hash_id, + shard_function_id, + courier_envelope_format_id, + courier_reply_format_id, + replica_request_format_id, + replica_response_format_id, + final_replica_count, + intermediate_replica_count, + minimum_replica_count, + storage_epoch_schedule_id, + key_acceptance_window_id, + retention_profile_id, + geometry_compatibility_id, + box_plaintext_capacity, + courier_dedup_retention, + request_generation_limit, + polling_profile_id, + cover_profile_id, + resource_limit_profile_id, + conformance_vector_set_id +] +``` + +The accepted consensus authorizes exact numeric profile IDs. Operators do not +select primitives, shard counts, geometry, retention, or parser behavior. +Incomplete candidates receive no numeric ID. + +### 8.2 Storage epoch + +The storage epoch is distinct from the FOG-PKI network epoch. Its duration, +origin, key-generation deadline, acceptance overlap, record namespaces, and +garbage-collection boundary are fixed by the authenticated storage profile. + +The candidate acceptance window is structurally equivalent to previous, +current, and next storage envelope keys around a boundary. Exact durations are +not selected in this draft. A request outside the accepted manifest and key +window is rejected without attempting every historical key. + +Replica clocks use the same explicit uncertainty model as FOG-PKI. A request, +courier, replica peer, HTTP header, or operator cannot supply authoritative +time or extend a record lifetime. + +### 8.3 Retained manifests and keys + +A role retains exactly the manifest, public verification material, and private +role keys needed for the profile's bounded previous, current, next, and drain +windows. It rejects older work and deletes expired private envelope and +receipt keys after all accepted request and response lifetimes end. + +The Composer retains public old-manifest material only while a live record may +still be read or a receipt may still be verified. A long-offline Composer must +import an authenticated manifest history, not accept a relay-provided current +replica set on trust. + +## 9. Fixed Formats and Geometry + +### 9.1 Encoding rules + +Online storage operations use fixed binary structures, not CBOR, maps, +compression, generic RPC, or packet autodetection. Every active profile gives +one exact length for each structure and one exact FOG-WIRE record count. + +All read, write, tombstone, miss, hit, conflict, accepted, result, and cover +bodies have equal outer geometry within the active traffic class. Fields that +are absent for one semantic operation are zero before authenticated +encryption. + +### 9.2 Courier envelope prefix + +The fixed `CourierEnvelope` begins with this exact 192-byte prefix: + +```text +offset length field +0 2 storage_format_version +2 1 envelope_kind +3 1 flags +4 4 storage_profile_id +8 8 storage_epoch +16 32 storage_manifest_hash +48 32 intermediate_replica_id_0 +80 32 intermediate_replica_id_1 +112 32 replica_envelope_key_id_0 +144 32 replica_envelope_key_id_1 +176 1 preferred_response_slot +177 1 response_slot_count +178 14 reserved +``` + +`envelope_kind` is `OPERATION`, `flags` is zero, `response_slot_count` is 2, +and `preferred_response_slot` is 0 or 1. The two intermediate IDs are distinct +and in canonical ascending order; preference is an independent CSPRNG choice +and does not reorder them. + +The prefix is followed by profile-fixed areas: + +```text +client_ephemeral_public_key[P] +intermediate_dek_0[D] +intermediate_dek_1[D] +encrypted_inner_request[Q] +zero_padding[Z] +``` + +The complete body, including `Z`, is exactly the compatible KEMSphinx +`user_forward_payload_length`. The multi-recipient envelope authenticates the +prefix and encrypts the complete fixed inner request to either intermediate +replica. The courier cannot decrypt it. + +The courier request hash is: + +```text +envelope_hash = HASH( + storage_profile_domain || complete_fixed_CourierEnvelope +) +``` + +The profile fixes the hash and domain. The hash is a short-lived courier-local +deduplication handle, not a message ID, box ID, account, capability, or +cross-generation identifier. + +### 9.3 Inner replica request + +After envelope decapsulation, an intermediate obtains one fixed +`ReplicaInnerRequest`: + +```text +[ + inner_format_version, + operation_kind, + flags, + storage_profile_id, + storage_epoch, + storage_manifest_hash, + operation_nonce[32], + box_id[B], + box_signature[S], + fixed_box_payload[X], + zero_padding +] +``` + +`operation_kind` is `READ`, `WRITE_DATA`, or `WRITE_TOMBSTONE`. For READ, the +signature and payload areas are zero. For WRITE_DATA, they contain the exact +capability-authenticated record. For WRITE_TOMBSTONE, the signature +authenticates the candidate construction's canonical empty payload and the +fixed payload area is zero. + +The operation nonce contains 256 CSPRNG bits and is unique per courier request +generation. It is not a capability and does not replace box-record +authentication. + +### 9.4 Courier reply + +Every anonymous reply begins with one exact 64-byte prefix: + +```text +offset length field +0 2 storage_reply_format_version +2 1 reply_kind +3 1 actual_response_slot +4 4 storage_profile_id +8 32 envelope_hash +40 1 outer_status +41 23 reserved +``` + +It is followed by one fixed opaque replica-response area `R` and zero padding +to the compatible KEMSphinx reply capacity. `reply_kind` is `ACCEPTED`, +`RESULT`, or `COARSE_REJECT`. ACCEPTED and COARSE_REJECT have a zero response +area. RESULT contains one replica-encrypted aggregate response from an +intermediate. + +The reply size, KEMSphinx route, and external schedule do not reveal the kind. +The courier necessarily knows which coarse reply it constructed but does not +learn the encrypted replica outcome. + +### 9.5 Replica commands + +FOG-WIRE command bodies are fixed by the same compatibility profile: + +- `REPLICA_REQUEST` carries one envelope hash and the fixed public-key, + encapsulation, and ciphertext fields required by one intermediate; +- `REPLICA_RESPONSE` carries the envelope hash, response slot, and one fixed + client-encrypted aggregate reply; +- `REPLICA_SYNC` carries one fixed `SHARD_OPERATION`, `SHARD_RESPONSE`, or + bounded repair body between storage replicas. + +Subtypes are authenticated inside their owning fixed body. FOG-WIRE context +and command selection do not authorize a different storage profile. + +### 9.6 Geometry equations + +For a KEMSphinx forward capacity `U`: + +```text +192 + P + (2 * D) + Q + Z = U +``` + +The encrypted inner capacity must satisfy: + +```text +message_envelope_length + + capability_payload_overhead + + inner_request_overhead + + replica_envelope_overhead + <= Q +``` + +The compatible reply must hold one fixed box record, two final-replica +receipts, aggregate framing, and response encryption. FOG-WIRE must hold the +same objects in its declared record counts. + +No document may treat the calculated 4,096-byte KEMSphinx candidate user +payload as the box or message capacity before a byte-exact geometry calculator +proves every equation in both directions. + +## 10. Replica and Courier Selection + +### 10.1 Eligible set + +The Composer uses only the ordered replica records in the accepted manifest. +Every selected replica must support the exact active storage, envelope, +receipt, and FOG-WIRE profiles and have an unrevoked current endpoint in the +accepted FOG-PKI consensus. + +A claim-bearing manifest has at least four storage replicas controlled by +distinct operators and prohibited-family domains. Co-location or shared +credentials invalidate the independence assumption even if node IDs differ. + +### 10.2 Final shard pair + +The profile's deterministic shard function scores every eligible replica from +the exact manifest using a domain-separated hash over at least the manifest +hash, replica storage identity, and box ID. The two lowest distinct scores are +the final shard pair, with complete node ID as the deterministic tie breaker. + +The candidate deliberately adds FOG network, profile, and manifest domain +binding to the Pigeonhole-derived two-shard selection. This is not byte- +compatible with Katzenpost `Shard2` and requires independent analysis, +byte-identical vectors, and a distinct immutable profile. + +Every Composer and replica computes the pair byte-identically. The courier is +not given the box ID or pair. + +### 10.3 Intermediate pair + +The Composer excludes the final pair, then chooses two distinct intermediate +replicas uniformly with its CSPRNG from the remaining eligible set. It rejects +a choice that repeats a prohibited operator, family, or infrastructure domain +across the courier and four storage positions when the active claim requires +that diversity. + +At exactly four replicas, the two non-final replicas are the only valid +intermediate set. At three or fewer, claim-bearing work stops. A local fixture +may use a named functional-test profile that makes no intermediate/final +unlinkability claim. + +### 10.4 Courier + +The Composer selects an eligible courier independently from the storage set +under the authenticated route and diversity policy. One courier request +generation remains pinned to that courier because its dedup cache holds the +operation state. + +A bounded later generation MAY use another courier after the previous +generation's lifetime and reply material expire. It keeps the same box record +and normally the same intermediate pair to avoid progressively exposing the +box to more storage roles. Exact failover and exposure limits belong to the +retry profile. + +## 11. Write Processing + +### 11.1 Composer write transaction + +FOG-MESSAGING first supplies one already committed immutable +`MessageEnvelope`. The storage writer then MUST: + +1. verify that no earlier box is still unresolved for the stream; +2. clone the current write-capability state; +3. derive one box ID, payload key, signing key, and next stream state; +4. encrypt and authenticate the fixed message envelope as one data box; +5. create the exact signed data record and a canonical signed recovery + tombstone for the same box; +6. select the manifest, final pair, intermediate pair, courier, and envelope + keys under the accepted profiles; +7. build one exact immutable `CourierEnvelope` request generation; +8. atomically persist the next stream state, data record, tombstone, immutable + courier envelope, selection, retry state, and storage outbox status; +9. erase retired per-box private material and discarded staged state; +10. only after commit, make the KEMSphinx operation exportable. + +A crash before step 8 exports nothing. A crash after step 8 recovers the exact +box and courier envelope without re-deriving the box or reusing a per-box key. + +Only one box write per stream is externally outstanding. Later application +messages may queue locally but cannot skip an unresolved storage index. + +### 11.2 Courier dispatch + +After terminal KEMSphinx validation, the courier: + +1. strictly parses the fixed envelope and accepted manifest window; +2. computes the envelope hash; +3. checks the bounded dedup cache; +4. on a cache miss, allocates one bounded state entry and dispatches exactly + one fixed request to each named intermediate replica; +5. uses the current request's SURB for one scheduled ACCEPTED or cached RESULT + reply and does not retain the SURB; +6. caches at most two fixed opaque replica responses until the dedup deadline. + +A cache hit never dispatches the request again. If a cached result is +available, the courier can return it using the fresh SURB carried by the +retransmission. Otherwise it returns ACCEPTED. All replies remain subject to +the external schedule and one-response amplification bound. + +### 11.3 Intermediate processing + +Each intermediate replica: + +1. authenticates the courier through FOG-WIRE; +2. validates exact profile, manifest, key ID, epoch, ciphertext, and limits; +3. decapsulates and authenticates the complete inner request; +4. recomputes and validates the two final replicas; +5. dispatches the exact operation to both final replicas through fixed + `REPLICA_SYNC` shard-operation bodies; +6. collects bounded final receipts without treating a link ACK as a commit; +7. builds one fixed aggregate response encrypted for the Composer ephemeral + key; +8. sends the opaque response to the courier and erases request secrets and + transient box state after the retry window. + +It performs no application parsing and never retains a stream capability. + +### 11.4 Final write transaction + +A final replica verifies the manifest, epoch, operation nonce, box ID, box +signature, record size, and storage quota before mutation. It then applies one +atomic transaction: + +- empty plus valid data: insert the complete record; +- same exact data: idempotent success; +- different data at the same box: conflict, no overwrite; +- any state plus valid tombstone: store the tombstone and delete live data; +- tombstone plus data: tombstone wins, reject resurrection; +- same valid tombstone: idempotent success. + +The replica durably commits database and required local journal state before +signing a commit receipt. A transport ACK, queued write, in-memory update, or +unflushed batch is not a commit receipt. + +### 11.5 Writer completion + +The Composer verifies receipts under the exact two final-replica receipt keys +from the manifest. `REPLICA_QUORUM_COMMITTED` requires matching durable +receipts from both final replicas for the same manifest, epoch, nonce, box ID, +record digest, and result. + +One receipt is degraded evidence, not quorum. A courier ACCEPTED reply is not +replica evidence. The writer retains and retries the immutable box until: + +- two matching commit receipts arrive; +- the receiving Composer's later message-level commit ACK proves successful + retrieval; +- a valid recovery tombstone is committed to both replicas; or +- the operation enters explicit `UNCERTAIN` or `RECOVERY_REQUIRED` state. + +The next storage box is not exported while the current box remains unresolved. + +## 12. Read, Empty, and Tombstone Processing + +### 12.1 Read query + +The reader clones but does not advance its current capability state, derives +the expected box ID, selects the manifest and roles, generates a fresh +operation nonce, and atomically persists one immutable read request generation +before export. + +Retransmission within that generation uses the same courier envelope and fresh +KEMSphinx and reply material. A later poll after a terminal result uses a new +operation nonce and new courier envelope but derives the same box ID until the +stream advances. + +### 12.2 Final read response + +Each final replica atomically reads one box and returns one fixed response: + +- `DATA`: complete record and its digest; +- `TOMBSTONE`: complete signed tombstone and its digest; +- `MISS`: no record observed for the box in that namespace; +- `CONFLICT` or coarse local failure where internal consistency is broken. + +The final replica signs a receipt over the network, storage profile, manifest, +epoch, operation nonce, box ID, result code, and record digest. MISS uses the +canonical empty digest. FOG-WIRE protects the receipt between the final and +intermediate replica. The intermediate then places it in the fixed aggregate +response encrypted toward the Composer, so the courier cannot inspect it. + +A receipt authenticates what one replica reported. It does not make a +malicious replica honest or make absence permanent. + +### 12.3 Aggregate result + +An intermediate aggregate contains two final-replica receipts and at most one +complete fixed box record. The record digest must match both receipts before +the Composer treats the final replicas as converged. + +The Composer handles results as follows: + +- matching DATA receipts: verify box signature, decrypt, validate fixed + payload and pass it to the owning upper protocol transaction; +- matching TOMBSTONE receipts: verify the writer signature, atomically advance + the read capability, and emit no message content; +- matching MISS receipts: report one empty poll locally and do not advance; +- one DATA and one MISS: treat as replication lag and retry under schedule; +- one TOMBSTONE and one older DATA: wait for tombstone convergence; +- different valid data digests: freeze the stream as `STORAGE_CONFLICT`; +- missing, invalid, wrong-manifest, wrong-nonce, or wrong-box receipt: discard + without advancement. + +### 12.4 Cross-layer receive commit + +For a DATA result, storage payload acceptance and FOG-MESSAGING acceptance are +one Composer transaction. The implementation stages the next read-capability +state, storage dedup state, message authentication, ratchet state, reassembly, +inbox, and message ACK state, then commits them together before rendering. + +If the box is authentically written but contains an invalid FOG-MESSAGING +object, the Composer atomically records a bounded poisoned-slot marker and MAY +advance the storage stream without advancing the message ratchet. It sends no +automatic error oracle. This behavior is permitted only for a box whose writer +authentication is valid, because otherwise an online attacker could skip +stream positions. + +The exact malicious-contact UI and quarantine retention belong to +FOG-COMPOSER, but their count and byte limits belong to the storage profile. + +### 12.5 Meaning of empty + +MISS is an expected asynchronous state. It can arise because the writer has +not written, replication is delayed, a replica is malicious, a request reached +the wrong retained epoch, or data expired. + +The reader polls again only under its authenticated retrieval schedule. It +does not immediately retry on MISS, advance to the next box, switch to a +direct replica, or send a message-level acknowledgment. + +## 13. Courier Deduplication, Retry, and Reply State + +### 13.1 Courier cache + +One courier cache entry contains only: + +```text +[ + envelope_hash, + storage_profile_id, + storage_epoch, + storage_manifest_hash, + created_monotonic_time, + intermediate_replica_ids[2], + dispatch_state[2], + opaque_response_slots[2], + terminal_deadline +] +``` + +It contains no SURB, capability, box ID, operation kind, final pair, message +identifier, user identifier, or application state. Cache memory and total +entries are bounded globally and per authenticated replica relationship. + +The cache MAY be volatile because final operations are idempotent. A courier +restart may redispatch an exact request, but cannot change the box record. +Persistent cache, if used, has a dedicated sealing key and the same strict +expiry. + +### 13.2 Request generation + +The Composer persists these minimum states: + +- `READY`: immutable generation committed but not exported; +- `IN_FLIGHT`: exported and eligible for exact retransmission; +- `COURIER_ACCEPTED`: courier accepted bounded work, durability unknown; +- `REPLICA_QUORUM_COMMITTED`: two matching final receipts verified; +- `DEGRADED`: only one matching final receipt or replica unavailable; +- `EMPTY`: one completed read poll returned matching MISS receipts; +- `TOMBSTONED`: matching valid tombstone receipts committed; +- `CONFLICT`: incompatible authenticated final states; +- `EXPIRED`: request-generation retry limit reached; +- `UNCERTAIN`: local durability or remote outcome cannot be resolved; +- `CANCELLED`: local cancellation before a security-critical transition. + +The message outbox remains separate. Storage quorum does not set a message to +ACKED, and message ACK may safely terminate storage retry after proving the +receiver committed the message. + +### 13.3 Retransmission + +Within one request generation, the Composer reuses exact `CourierEnvelope` +bytes and pins the courier and intermediate pair. Every transmission creates a +fresh packet, SURB, private reply token, route, entry material, and transfer +bundle identifier. + +The profile sets maximum attempts, maximum generation age, jitter, backoff, +cache retention, result polling, and maximum later generations. No retry is +immediate or seeded from a box, envelope, contact, or message identifier. + +A later generation builds a new inner request around the same immutable box +record and operation kind, using a fresh operation nonce and ephemeral +envelope key. It starts only after old reply +material and the previous courier cache window end, and it remains subject to +the fixed exposure budget. + +### 13.4 Reply material + +The courier never stores a SURB for later use. It consumes the fresh SURB from +the current KEMSphinx request for at most one fixed reply. If no result is ready +it returns ACCEPTED; the Composer later supplies a new single-use SURB by +retransmitting the same courier envelope. + +No result causes a direct second reply, and no storage ACK causes an ACK of its +own. This keeps amplification at one anonymous reply per accepted request. + +## 14. Replica Receipts and Consistency + +### 14.1 Receipt body + +A final-replica receipt is a profile-fixed canonical object whose signed body +contains: + +```text +[ + receipt_format_version, + network_id, + storage_profile_id, + storage_manifest_hash, + storage_epoch, + final_replica_id, + replica_receipt_key_id, + operation_nonce, + box_id, + operation_kind, + result_code, + record_digest +] +``` + +The signature suite is fixed by the storage profile and key record. A result +cannot name or negotiate its verifier. Receipt verification occurs only after +the enclosing replica response authenticates and parses. + +Receipts are sensitive metadata because they contain box IDs. They are visible +only to the processing storage replicas, then remain inside client-encrypted +replies and Composer state. They MUST NOT be published as generic monitoring +or transparency evidence. + +### 14.2 Durability model + +The initial structural profile has two final replicas and requires two +matching receipts for ordinary write quorum or read convergence. It does not +claim Byzantine consensus, linearizability, permanent durability, or global +read-after-write ordering. + +A malicious final replica can sign false state, withhold work, delete data, or +equivocate. Two independent receipts make the responsible identities locally +detectable but do not restore deleted data. A malicious intermediate can +withhold or reorder receipts but cannot forge a valid final receipt under the +selected signature assumption. + +### 14.3 Conflicts and tombstone precedence + +Two different valid data records for one box indicate writer-state cloning, +malicious writer behavior, or an implementation failure. Replicas preserve +first-write-wins locally, and the Composer freezes on cross-replica divergence. + +A valid tombstone has permanent precedence within its storage-epoch namespace. +Repair always propagates tombstone over data and never propagates data over a +tombstone. + +## 15. Retention, Tombstones, and Garbage Collection + +### 15.1 Record retention + +Each record belongs to one storage-epoch namespace and receives its expiry +only from the authenticated manifest and retention profile. A writer cannot +request longer retention, and a replica cannot selectively extend one user's +record. + +The active profile publishes the minimum and maximum retrieval window in +ordinary time units and storage epochs. Public documentation MUST explain that +an offline recipient who does not retrieve within that window can lose data. + +The candidate evaluates the Pigeonhole previous-and-current epoch retention +shape, but no duration is selected until offline-usage, capacity, abuse, and +cost simulations are complete. + +### 15.2 Tombstone creation + +The writer creates a canonical signed empty record. It is externally the same +size class as read and data operations. The courier cannot distinguish it, but +the replicas necessarily learn that a valid box is being tombstoned. + +A recovery tombstone is generated and stored locally with every data box before +the per-box signing key is erased. It may be exported only under explicit +abandon, recovery, user deletion, or profile policy. + +Routine successful delivery SHOULD normally rely on bounded natural expiry +rather than an immediate tombstone correlated with recipient activity. If +automatic tombstoning is enabled, its minimum hold, random delay, and cover +schedule are authenticated profile parameters. + +### 15.3 Replica tombstone state + +After durable tombstone commit, a replica deletes the live ciphertext and +retains the minimum authenticated tombstone record needed to reject +resurrection. That record remains until the complete namespace expires. + +Garbage collection MUST delete data and tombstones by deterministic epoch +policy, not access recency, contact activity, request frequency, or operator +preference. It runs under bounded IO and cannot block wire processing without +backpressure. + +### 15.4 Deletion limits + +A tombstone is not proof that every historical copy vanished. A recipient may +already hold plaintext; an attacker may have copied ciphertext; storage media, +journals, snapshots, and backups may retain blocks until their documented +destruction window. + +Replica backups use separate encryption keys and bounded retention. Restore +MUST NOT resurrect expired data or replace a tombstone with older live data. +The operator documents the maximum backup-deletion delay and tests restore +against tombstone and expiry state. + +### 15.5 Expired-gap recovery + +Because MISS never advances a reader, a box that expires before retrieval can +block every later box in that stream. If the writer still has an unacknowledged +immutable data record, it MAY reemit that exact record into a currently valid +storage-epoch namespace under a new bounded request generation. If the writer +has abandoned the data, it reemits the precomputed signed tombstone instead. + +The reader advances only after a converged authenticated DATA or TOMBSTONE +result. Cross-epoch reemission may reveal the same box to additional replicas +or link epochs and therefore has a profile exposure limit and simulation gate. +If neither the writer nor a fresh authenticated recovery session is available, +the reader cannot safely skip the gap. + +## 16. Replication, Repair, and Manifest Transition + +### 16.1 Normal replication + +Both intermediate replicas dispatch the authenticated operation to the same +two deterministic final replicas. Final writes are idempotent, so duplicate +paths cannot create duplicate boxes or change first-write-wins behavior. + +The profile fixes the maximum fan-out. One courier request creates at most two +courier-to-intermediate requests, four intermediate-to-final shard operations, +bounded responses, and one anonymous client reply. Implementations MUST NOT +increase fan-out after failure. + +### 16.2 Repair + +`REPLICA_SYNC` supports bounded repair only between the two final replicas for +one manifest and retained epoch. A repair body carries one complete +authenticated box record or tombstone and the evidence required by the +profile. The receiving replica independently verifies the box signature, +manifest, epoch, digest, and tombstone precedence before commit. + +Repair traffic uses fixed FOG-WIRE shapes and the authenticated store-to-store +schedule. It MUST NOT expose an unbounded database listing, stream sequence, +capability root, or arbitrary range query. + +### 16.3 Replica unavailability + +One unavailable final produces degraded durability and no ordinary quorum. +The operation retries under schedule but does not substitute an arbitrary +third final, because that would change deterministic location and reveal more +replicas. + +If a manifest cannot meet its deployment assumptions, new work stops or uses +an already authorized lower-claim functional profile. Existing data remains +bound to the old manifest until expiry. + +### 16.4 Manifest transition + +A new storage epoch creates a new immutable manifest and new replica envelope +and receipt keys. Old and new manifests coexist only for the explicit read, +retry, key, and drain windows. + +FOG does not silently migrate live boxes to a new final set. A writer that +needs a longer-lived logical message re-emits it through a new authenticated +storage operation or renewed stream according to the upper protocol. Cross- +epoch re-emission remains bounded and may expose linkability at replicas, so it +requires explicit simulation and profile rules. + +## 17. Cover, Empty Reads, and Failure Privacy + +### 17.1 Traffic scheduling + +Fixed bytes do not hide record counts or time. Any claim that reads, writes, +misses, or tombstones are externally indistinguishable requires: + +- Composer operations entering the same authenticated client schedule; +- KEMSphinx packets and replies using one geometry; +- courier-to-replica and replica-to-replica links using fixed-throughput or + independently scheduled fixed slots with valid cover; +- retry and result polling independent of application type; +- overload and shutdown behavior included in the simulation. + +Exact rates and distributions remain cover-profile selections. Operators MUST +NOT tune them independently in a claim-bearing deployment. + +### 17.2 Role-local distinguishability + +The courier can distinguish ACCEPTED from a cached opaque RESULT but cannot +decrypt the result. Intermediate and final replicas necessarily distinguish +operation types after envelope decryption. FOG claims only the information +exposure stated in Section 3, not perfect role-local indistinguishability. + +### 17.3 Repeated empty polling + +Repeated reads of one not-yet-written box reveal the same box ID to its final +replicas. Changing KEMSphinx routes, couriers, envelopes, or intermediates does +not remove that final-replica link. + +The retrieval profile therefore bounds polling frequency, uses cover traffic, +avoids immediate reaction to a write or MISS, and includes this leakage in +long-term intersection simulation. FOG makes no unconditional claim that an +honest-but-curious final replica cannot recognize repeat polls. + +### 17.4 Failure classes + +Online remote failures are coarse: + +- malformed or unauthorized envelope: uniform drop or fixed COARSE_REJECT; +- accepted bounded work: fixed ACCEPTED; +- available opaque result: fixed RESULT; +- overload or timeout: fixed coarse outcome under the cover schedule. + +Detailed DATA, TOMBSTONE, MISS, conflict, database, quota, receipt, and +replication results remain inside Composer-encrypted responses where possible. +No error is larger than the request, and one request produces at most one +anonymous reply. + +## 18. Resource Limits and Storage DoS + +Capability-based addressing is not a complete admission system. An attacker +can create unlimited self-owned capabilities, valid signatures, boxes, reads, +or tombstones. Random box IDs prevent guessing another stream but do not +prevent storage exhaustion. + +Every active profile MUST set lower limits within explicit version maxima for: + +- accepted envelope bytes and cryptographic work; +- courier cache entries, bytes, generations, dispatches, and lifetime; +- intermediate in-flight requests, final fan-out, retries, and response bytes; +- final records, bytes, writes, reads, tombstones, receipts, and repair work; +- per-peer FOG-WIRE queues and connections; +- per-manifest and per-storage-epoch total capacity; +- Composer streams, pending boxes, empty polls, poisoned slots, and retained + receipts; +- backup size, journal size, garbage-collection work, and restore input; +- cover backlog and degraded-mode thresholds. + +At minimum, the first structural profile has: + +- exactly two intermediates and two finals per request; +- no more than one anonymous reply per request; +- one unresolved external box per stream; +- no operator-controlled variable replication factor; +- no recursive operation, generic batch, arbitrary range, or server-side + capability traversal; +- no CopyCommand or AllOrNothing operation; +- no requester-selected expiry or replica list outside the manifest. + +Authentication and size validation occur before database mutation. Allocation +is bounded before public-key work where the format permits. Signature-valid +new writes are still subject to global epoch capacity and overload policy. + +When full, a replica preserves already committed records and tombstones until +their policy expiry and rejects new allocation with a coarse fixed response. +It MUST NOT evict according to access recency, user activity, application type, +or payment outside a separately reviewed admission protocol. + +Safe anonymous admission and fair rate control remain open. Until resolved, +FOG does not claim resistance to a determined distributed storage-flooding +adversary. + +## 19. Profile and State Transitions + +### 19.1 Stream states + +The minimum Composer stream states are: + +- `ISSUED`: reader state persisted and writer grant ready for export; +- `ACTIVE`: one writer and one reader state are valid; +- `WRITE_PENDING`: one immutable box and request generation are outstanding; +- `READ_POLLING`: the current box has a persisted read request generation; +- `RENEWING`: a new stream generation is authenticated through messaging; +- `STORAGE_CONFLICT`: replicas returned incompatible authenticated state; +- `FROZEN_CAPABILITY_CHANGE`: an unexpected grant or state change is present; +- `RECOVERY_REQUIRED`: rollback, cloning, loss, or compromise prevents reuse; +- `CLOSED`: no new boxes may be created. + +State files are not an API. A transition occurs only through the atomic +Composer operations defined here and in FOG-MESSAGING. + +### 19.2 Storage profile transition + +A contact switches storage profiles only through an authenticated +FOG-MESSAGING control transition that binds the exact new profile, stream +generation, capabilities, first index, and manifest activation. + +Old and new streams remain separate. There is no profile trial, mixed +capability derivation, dual decryption of one box, or fallback after failure. +The old stream drains only through its previously authorized profile and +retention window. + +### 19.3 Compromise + +Compromise of a write cap permits reading where derivable, arbitrary future +writes, conflicting writes, and tombstones for that stream. Compromise of a +read cap permits future location derivation and decryption but not valid writes +under the selected construction. + +Response requires stopping the stream, replacing the Composer or capability +state, distributing a new stream through an authenticated contact session, +and allowing old data to expire. Rotation to the next box does not remove an +attacker who copied the root capability. + +## 20. Candidate BACAP and Pigeonhole Integration + +### 20.1 Candidate definition + +`FOG-STORAGE-CANDIDATE-BACAP-PIGEONHOLE-1` evaluates: + +- the maintained BACAP implementation and its deterministic blinded Ed25519 + box sequence, evolving KDF state, signatures, and authenticated payload + encryption; +- the maintained Pigeonhole single-box courier and scattered-replica flow; +- two deterministic final replicas, two disjoint random intermediates, and at + least four eligible replicas; +- one fixed multi-recipient envelope for the two intermediates; +- identical courier-envelope retransmission with fresh Sphinx-family packet + and SURB material; +- FOG storage manifests, final-replica receipts, fixed binary formats, atomic + Composer state, and FOG-WIRE commands defined here. + +The candidate does not adopt the Katzenpost client daemon, PKI epochs, +transport wire bytes, service discovery, Sphinx geometry, or operator +configuration as FOG runtime dependencies. + +### 20.2 Deliberate exclusions and deviations + +The initial FOG candidate excludes Pigeonhole CopyCommand and AllOrNothing +streams because they give the courier a serialized write capability and make +it traverse capability-derived boxes. That behavior conflicts with the FOG +courier boundary. + +The candidate adds final-replica receipts so the offline Composer can +distinguish courier acceptance from evidence of final durable commit. This is +a FOG protocol extension requiring its own cryptographic and metadata review. + +FOG-MESSAGING fragmentation remains above storage. Multiple fragments are not +atomically visible at replicas; the receiving Composer releases content only +after authenticated complete reassembly. + +### 20.3 Claims deliberately withheld + +The candidate does not yet establish: + +- direct Katzenpost Pigeonhole interoperability; +- quantum-resistant box authenticity, capability security, or complete + storage confidentiality; +- unlinkability if the courier and relevant replicas collude; +- unlinkability of repeated reads to one empty box at a final replica; +- deletion of every backup or adversarial copy; +- Byzantine consistency, guaranteed delivery, or permanent durability; +- DoS resistance or anonymous fair admission; +- safe one-box attenuation for `fog-drop`; +- compatibility with the calculated 4,096-byte KEMSphinx payload; +- implementation, constant-time, side-channel, or secure-deletion safety. + +### 20.4 Activation gates + +Before promotion to an active numeric profile, FOG MUST freeze and verify: + +1. exact BACAP, Pigeonhole, KDF, signature, AEAD, hash, envelope, and receipt + revisions and source commits; +2. whether the reviewed capability semantics and security analysis cover the + exact exported writer and reader state used by FOG; +3. byte-exact capability, box, courier, replica, receipt, and manifest + serialization; +4. complete forward and reply geometry against one exact KEMSphinx and + FOG-WIRE profile; +5. transcript, network, profile, manifest, epoch, box, and operation-nonce + domain binding without changing upstream primitives silently; +6. deterministic shard and intermediate-selection vectors; +7. crash, rollback, duplicate, conflict, tombstone, repair, expiry, backup, + and manifest-transition behavior; +8. loss, delay, replay, empty polling, courier restart, and replica failure; +9. CPU, memory, disk, bandwidth, amplification, flood, and GC limits; +10. implementation maturity, dependency licensing, side-channel behavior, and + secret deletion; +11. independent cryptographic and implementation review before public claims. + +A required change to BACAP derivation, signing, or encryption creates a +separately identified candidate rather than an undocumented FOG variant. + +## 21. Key and Secret Lifecycle + +| Material | Owner | Persistence | Required destruction or transition | +| --- | --- | --- | --- | +| Stream write root and evolving state | one writer Composer | encrypted mutable state, never online | replace by new stream on compromise, clone, rollback, or profile transition | +| Stream read root and evolving state | one reader Composer; writer MAY derive under candidate | encrypted mutable state, never online | replace by new stream on compromise, clone, rollback, or profile transition | +| Per-box signing and payload keys | Composer transaction | transaction only | erase after data record, recovery tombstone, next state, and outbox commit | +| Immutable signed data record | writer Composer and final replicas | bounded outbox and storage-epoch state | remove after message ACK and recovery margin locally; GC by replica retention | +| Signed recovery tombstone | writer Composer, final replicas only after use | encrypted local outbox until resolution | erase locally after terminal retention; retain at replica to namespace expiry | +| Operation nonce | writer or reader Composer | one courier request generation | erase after generation and receipt retention | +| Composer envelope ephemeral private key | requesting Composer | one request generation | erase after all expected replies and retry retention end | +| Intermediate DEKs and shared secrets | Composer and addressed intermediate transaction | immutable public encapsulation plus transient secret | erase transient decapsulation and response secrets after bounded processing | +| Replica envelope private key | one storage replica | previous, current, next storage-key windows only | erase after accepted request and response drain ends | +| Replica receipt private key | one final replica | one storage-manifest key period | stop signing at retirement; erase after receipt verification drain | +| Courier envelope hash and opaque responses | one courier | bounded dedup cache only | expire at courier dedup deadline; never back up by default | +| Intermediate dispatch and aggregate state | one intermediate replica | bounded transient state | erase after response and retry window | +| Final box record or tombstone | two deterministic final replicas | bounded storage-epoch namespace | deterministic GC; tombstone blocks resurrection until namespace expiry | +| FOG-WIRE Noise key | one online node | role-local profile lifetime | never reuse as envelope, receipt, box, or at-rest key | +| Replica storage-at-rest key | one replica operator | deployment-specific protected storage | separate rotation and backup policy, no protocol authenticity claim | +| Replica backup key | one replica recovery domain | separate from backup ciphertext and live store | rotate and destroy under bounded backup-retention policy | + +Capability roots, per-box secrets, box IDs, operation nonces, courier hashes, +receipts, replica ciphertexts, stream indexes, message envelopes, contacts, and +selection details MUST NOT enter logs, metrics, command arguments, crash +reports, public evidence, or support bundles. + +## 22. Logging and Observability + +Couriers and replicas MAY export delayed, thresholded aggregate counts for +capacity, coarse success, overload, expiry, corruption, and repair only when +the observability profile proves that the aggregation cannot expose a small +activity set. + +They MUST NOT export: + +- box IDs, envelope hashes, operation nonces, receipt digests, or signatures; +- intermediate or final selection per request; +- per-record timestamps, read frequency, miss streaks, or tombstone timing; +- capability bytes, payloads, ciphertext samples, or decryption failures; +- per-contact, per-stream, per-courier-request, or per-source histories; +- fine-grained queue, connection, or storage-access event streams. + +Local debugging that enables any prohibited class places the node outside a +claim-bearing profile and must be disabled by default in release builds. + +## 23. Conformance and Adversarial Tests + +Before the local PoC, FOG-STORAGE requires deterministic positive and negative +tests for: + +- capability issue-before-export, one-writer enforcement, index advancement, + serialization, cloning detection, and stale restore; +- exact 192-byte courier prefix and 64-byte reply prefix; +- every fixed body, padding byte, reserved bit, operation, and result; +- manifest canonical encoding, authority signatures, sign-once state, + chaining, membership, key, diversity, and transition checks; +- shard selection and intermediate exclusion across complete vector sets; +- complete forward and reply geometry calculations; +- Composer crash before and after write-state, record, tombstone, envelope, + and outbox commit; +- reader crash before and after the combined storage and messaging commit; +- exact request retransmission with fresh KEMSphinx and SURB material; +- courier cache hit, miss, expiry, corruption, restart, and overload; +- courier behavior without storing a SURB; +- final empty insert, exact duplicate, conflicting data, tombstone overwrite, + tombstone duplicate, and data-after-tombstone rejection; +- two matching receipts, one missing receipt, invalid receipt, wrong nonce, + wrong epoch, wrong manifest, equivocation, and conflicting digests; +- DATA, TOMBSTONE, MISS, replication lag, poisoned message, and stream freeze; +- replica loss, repair, manifest change, key overlap, expiry, GC, backup + restore, and non-resurrection; +- fixed read, write, tombstone, hit, miss, retry, error, and cover geometry; +- request and response amplification bounds; +- malicious capability-holder CPU, memory, disk, queue, receipt, polling, and + notification exhaustion; +- absence of prohibited values from logs, metrics, crashes, and support data; +- candidate upstream vectors plus FOG-specific integration vectors. + +Testing MUST include parser fuzzing, property tests, transaction fault +injection, simulated power loss, disk-full and partial-write faults, clock +uncertainty, race detection, load and flood testing, cross-implementation +vectors, and restore exercises. + +## 24. Threat and Architecture Traceability + +| Requirement | Primary controls | +| --- | --- | +| `ARC-002` | the networkless Composer owns capabilities, box derivation, final verification, and index advancement | +| `ARC-003` | all client storage work traverses entry, every mix layer, courier, and replicas | +| `ARC-004` | box, envelope, KEMSphinx, and Noise protections use separate protocols and keys | +| `ARC-006` | fixed geometry and scheduling classes do not identify native applications | +| `ARC-007` | stream, box, envelope, receipt, wire, disk, and backup keys have distinct owners and purposes | +| `ARC-008` | exact parser, cache, queue, epoch, record, retry, fan-out, storage, and GC bounds | +| `ARC-009` | replica, manifest, profile, capacity, and receipt failures stop without bypass or downgrade | +| `IF-01` | only committed immutable storage request generations enter Composer export | +| `IF-02` | replies are untrusted until KEMSphinx, storage, receipt, capability, and upper-protocol commit | +| `IF-08` | the courier receives only the fixed opaque storage envelope after every mix layer | +| `IF-09` | exact fixed courier/replica and replica/replica commands, receipts, repair, and bounds | +| `IF-10` | one fresh single-use anonymous reply per transmitted request | +| `TM-NET-01` | fixed shapes, independent schedules, cover requirement, bounded retries, no immediate MISS reaction | +| `TM-NET-02` | rotating box IDs, directional streams, disjoint intermediates, fixed manifests, documented repeat-poll risk | +| `TM-NET-03` | no immediate fallback, bounded fan-out, degraded-state gate, traffic simulation | +| `TM-NET-04` | immutable request generations, courier dedup, idempotent final state, non-resurrection | +| `TM-NET-05` | authenticated envelopes, box records, receipts, strict result binding, uniform remote failure | +| `TM-NET-06` | consensus-authorized immutable profile, manifest, geometry, and application-independent behavior | +| `TM-ROLE-03` | courier blindness, disjoint intermediate/final roles, two final receipts, separate state and keys | +| `TM-ENDPOINT-01` | capability secrets stay in the Composer and commit before export or advancement | +| `TM-ENDPOINT-03` | stale restore freezes streams, separate backups, no live-state copying | +| `TM-APP-01` | one contact-specific writer stream, poisoned-slot handling, bounded payload before upper parsing | +| `TM-OPS-01` | no capability, box, request, receipt, selection, or per-record telemetry | +| `TM-CRYPTO-01` | complete lifecycle table, key purpose separation, bounded overlap and destruction | +| `TM-CRYPTO-02` | fixed profile and manifest, withheld PQ claims, no fallback or self-selected suite | +| `TM-AVAIL-01` | fixed fan-out, quotas, capacity limits, idempotency, GC, coarse overload, explicit admission gap | + +## 25. Open Dependencies + +The structural storage contract is fixed, but these dependencies remain open +before an active profile or daemon: + +- exact BACAP, envelope, receipt, hash, signature, AEAD, and KDF revisions and + reviewed implementations; +- resolution of upstream BACAP implementation maturity and audit gaps; +- exact storage epoch duration, manifest overlap, retention, courier-cache, + polling, retry, cover, and shutdown parameters; +- byte-exact forward, reply, FOG-WIRE, box, and messaging geometry; +- a reviewed receipt construction and metadata analysis; +- reviewed single-box capability attenuation for `fog-drop`; +- safe anonymous admission and fair storage-flood control; +- activation evidence for the structural FOG-COMPOSER vault, database, + external-anchor, and recovery profiles; +- future bulk, atomic multi-box, group, and multi-device protocols. + +No implementation convenience may silently resolve these dependencies. + +## 26. Primary References + +- Katzenpost Pigeonhole protocol specification: + <https://katzenpost.network/docs/specs/pigeonhole/> +- Katzenpost, Understanding Pigeonhole: + <https://katzenpost.network/docs/pigeonhole_explained/> +- Katzenpost HPQC BACAP implementation: + <https://github.com/katzenpost/hpqc/tree/main/bacap> +- Infeld et al., *Echomix: a Strong Anonymity System with Messaging*: + <https://arxiv.org/abs/2501.02933> +- FOG threat model: `FOG-THREAT-MODEL.md` +- FOG architecture: `FOG-ARCHITECTURE.md` +- FOG public key infrastructure: `FOG-PKI.md` +- FOG wire protocol: `FOG-WIRE.md` +- FOG Sphinx profile framework: `FOG-SPHINX-PROFILES.md` +- FOG messaging protocol: `FOG-MESSAGING.md` +- FOG Composer protocol: `FOG-COMPOSER.md` + +These references supply a published construction, implementation target, and +design lessons. They do not make the FOG integration secure by inheritance. +FOG still requires exact profiles, geometry, vectors, implementation review, +resource analysis, simulation, deployment evidence, and independent security +review. |