---
title: Failure and recovery
---

The primary write and every access decision fail closed. Derived data and telemetry fail
open and are repaired. A request that returns success has durably stored what it
promised; anything projected beside it is caught up by a worker. The last section,
[Capacity](#capacity), covers what scales how and where the limits are. Symptom runbooks
are in [troubleshooting](/operations/troubleshooting).

## Errors, retries and idempotency

| Boundary             | Behaviour                                                                                                                                                                                                                                                                                                                                                                                           |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ClickHouse client    | A transient network failure of an idempotent statement is retried once on a fresh connection. A replica-bound server error (codes 10, 242, 999) is retried on up to two new connections. Anything else raises.                                                                                                                                                                                      |
| Statement deadlines  | Past its class deadline (`LOOM_CH_TIMEOUT_READ_S`, `_WRITE_S`, `_DDL_S`) a request fails with 504 `BACKEND_TIMEOUT`, not retried.                                                                                                                                                                                                                                                                   |
| Pool admission       | No free slot within `LOOM_CH_POOL_WAIT_MS`: 503 `OVERLOADED` with `Retry-After: 1`.                                                                                                                                                                                                                                                                                                                 |
| Inserts              | A retried INSERT carries the same `insert_deduplication_token`, so a committed batch is not written twice. Without the token, an insert is not retried. A batched insert shares ClickHouse's flush with concurrent inserts into the same table, a flush that fails as a whole fails each of them, and each retry lands once under its own token ([batched inserts](/architecture/clickhouse#batched-inserts)). |
| Unsupported settings | A replica that rejects the quantized-codes read setting makes the client drop it for the process and retry.                                                                                                                                                                                                                                                                                         |
| Point reads          | Reads after the caller's own write use sequential consistency; a miss right after a write re-reads once (`LOOM_CH_READ_YOUR_WRITES`); a search on a namespace written within 20 s reads sequentially.                                                                                                                                                                                               |
| Schema skew          | A statement naming a table a pending migration would create is refused with 503 `SCHEMA_PENDING` (below).                                                                                                                                                                                                                                                                                           |
| Op idempotency       | Ops declared `idempotent` accept `Idempotency-Key` (HTTP) or `_idempotency_key` (MCP). The answer is kept in `loom.idempotency` for 24 hours, keyed on org, subject, op and key. A duplicate while the first runs gets a 409.                                                                                                                                                                       |
| Primary write        | The fact row and its vector row are awaited committed inserts. A failure fails the request.                                                                                                                                                                                                                                                                                                         |
| Postgres authority   | Under `LOOM_STORAGE_AUTHORITY=postgres` the write is the Postgres commit; its ClickHouse rows are published from it and repaired by the `pg_delivery` worker. Postgres down fails writes and point reads closed (503 `POSTGRES_UNAVAILABLE`) ([storage authority](/architecture/storage-authority#failure)).                                                                                                   |
| Projections          | Graph, chunks, evidence and supersession side tables fail open and are repaired by workers.                                                                                                                                                                                                                                                                                                         |
| Telemetry            | `agent_events` and the events stream are queued in process; an `agent_events` flush is a batched insert, an events-stream flush a committed one. Neither fails a request; a crash can lose the unflushed queue.                                                                                                                                                                                     |
| Workers              | A failing pass is retried with backoff. Work is written as absolute values or new versions, so a repeated pass converges.                                                                                                                                                                                                                                                                           |
| Erasure              | `erasure.request` is a checkpointed saga. `erasure_recovery` resumes it, and a residue census over every table decides success.                                                                                                                                                                                                                                                                     |

**Retryable answers.** `OVERLOADED`, `AUTH_UNAVAILABLE` and `RATE_LIMITED` carry
`Retry-After` (and `retry_after_s`); retry after the hint. `SCHEMA_PENDING` clears when an
operator applies the named migration. `EMBED_TIMEOUT`: a raw HTTP or MCP
caller retries once; an SDK caller has already retried (idempotent ops), so it backs off.
`BACKEND_TIMEOUT` is retryable only for an `idempotent` op, with an idempotency key for a
write. The full list is in the [errors reference](/reference/errors).

**Why the idempotency cache is best effort:** a cluster-consistent cache needs a lock per
request. The ops that use it write the same content on a duplicate run into a
`ReplacingMergeTree`, so a race costs a second handler run, not a second effect. Ops whose
repeat would double an effect (`memory.set_from_messages`) are non-idempotent and bypass
it.

## Consistency boundaries

Cross-table writes are not transactional. The fact and its vector land together; derived
data catches up:

| Derived data                  | Catches up                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BM25 statistics               | Refreshable views every minute; search reads a live tail, so ranking stays exact (`LOOM_MEMORY_SEARCH_BM25_LIVE_TAIL`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Usage counters, reinforcement | One `usage_fold` pass (every 300 s).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Entity-graph projection       | Shortly after the response. A `facts.traverse` of the same namespace that starts after the response waits for it (at most 3 s) on every process of the deployment: each projection in flight is recorded in `loom.graph_projections` (Keeper keys where the server has a Keeper), and its end records the namespace's graph write before clearing it, on the server's clock. A process that dies mid-projection leaves its record for at most 15 s: each process waits it out once (3 s), and until the record goes stale, walks of that namespace there are read fresh, never served from or kept in the memo. Past 32 projections of one namespace at once, the extra ones are not waited on; each counts as a graph write before its `memory.set` answers and again at its end. |

**Why not transactions:** ClickHouse has none across tables, and emulating them adds a
round trip and a failure mode to every write. A durable primary pair plus idempotent repair
keeps the write to one round trip.

## Backpressure

No request waits unboundedly on a shared resource:

- **ClickHouse pool:** a bounded wait, then `OVERLOADED` ([ClickHouse client](#clickhouse-client)).
- **Document bucket:** 8 transfers and 4 streams per process, each a bounded wait
  (`LOOM_CH_POOL_WAIT_MS`), then `OVERLOADED`; listings and deletes (erasure, the sweep)
  wait for one of 4 slots of their own.
- **Rate limits:** 429 `RATE_LIMITED` with `Retry-After`. Over MCP the transport waits once
  for a short hint (`LOOM_MCP_RATELIMIT_RETRY_MAX_WAIT_S`) and re-checks.
- **Query embedding:** `LOOM_MEMORY_SEARCH_EMBED_BOUND_S` (2.5 s) per attempt, retried
  `LOOM_MEMORY_SEARCH_EMBED_BOUND_RETRIES` times (1), then 503 `EMBED_TIMEOUT`.
- **Telemetry queues:** the `agent_events` emitter (10,000 rows) and the events stream
  (20,000) drop and count rows when full (`dropped_queue_full`, `dropped_flush_failed`),
  logging the first drop and every 1000th. Delivery is at most once.
- **Workers:** `usage_fold` skips a pass while ClickHouse has too many pending mutations;
  every pass is bounded by batch size and deadline.
- **Budgets:** an op over its `budget_p99_ms` still answers and leaves a `budget.violated`
  row (at most one per op per `LOOM_BUDGET_VIOLATION_AUDIT_INTERVAL_S`, 60 s).

## `SCHEMA_PENDING`

A server built from a newer commit than its schema (a branch server with
`LOOM_BOOT_MIGRATE=false`, or a colour that booted before its migrations) can name a table
that does not exist yet. `server/core/ch_schema_guard.py` (`guard_pending_schema`) refuses such a statement
with 503 `SCHEMA_PENDING`, naming the table and migration. A retrieval leg that treats a
failed read as optional continues without it, and the ledger row records the table. DDL is
never refused.

The pending set comes from the migration ledger (`server/core/schema/status.py`),
cached 60 s per process. When the ledger reports
nothing applied or more than 25 pending files, the guard goes idle with one warning.
`/v1/health` reports `schema.pending`, and a blue/green switch waits for it to be empty.

## Worker supervision

`server/workers/supervisor.py` runs each enabled worker as its own asyncio task:

- **Stagger.** The first pass waits `crc32(name) % interval`, except that a worker whose
  output table is empty runs at once, and a worker whose last pass is older than its
  interval starts within 300 s of boot.
- **Deadline.** A pass past `min(4 × interval, 900 s)`, floored at 60 s (unless the worker
  declares its own), is cancelled and counted as a failure.
- **Backoff.** A failed pass waits `min(60 s, 2^(n-1) s)`. A pass cut at its deadline, or a
  failed pass of a worker whose interval is longer than its deadline, waits the interval.
- **Degraded.** Three consecutive failures log `DEGRADED`; the streak is
  `loom.worker.consecutive_failures{worker}`, and a pass with a failed leg counts in
  `loom.worker.partial_failures`.
- **Status.** Each pass writes a `loom.worker_status` row (`GET /v1/admin/workers.status`);
  a resume cursor lets a new colour continue a long scan.
- **Shutdown.** `stop_all` cancels the tasks, waits up to 10 s, then flushes status rows.

A worker's exceptions stay in its task. Workers write absolute values or new versions, so
a pass that runs twice converges.

## Repair workers

| Worker                    | Repairs                                                                                                                                               | Guarantee                                                                          |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `projection_repair`       | Replays failed projections (evidence links, chunks, graph side tables, the expiry sweep's search mirror) logged in `loom.memory_projection_failures`. | Unsupported entries stay for inspection.                                           |
| `entity_extractor`        | Extracts entities onto rows that have none; re-emits `memory_embeddings` rows whose `entities` diverge from `memory`.                                 | Every searchable row eventually carries its entities.                              |
| `orphan_vector_reconcile` | Tombstones embedding and entity-vector rows whose memory is tombstoned or gone, after a grace period.                                                 | No vector outlives its fact.                                                       |
| `embedding_worker`        | Embeds `embedding_dim = 0` stubs newest first and memories with no embedding row; re-embeds stubs from an older model.                                | A row that fails 5 times is marked `embed_failed_at` and skipped.                  |
| `erasure_recovery`        | Resumes sagas stuck `in_progress` or `failed` from `steps_completed`.                                                                                 | After `max_attempts` (5): terminal `failed` plus an `erasure.abandoned` audit row. |
| `task_lease_sweep`        | Returns expired leases to `todo`, or `blocked` after their attempts; resumes deferred tasks.                                                          | A crashed agent's task is not lost.                                                |
| `audit_verifier`          | Recomputes audit hashes and verifies the checkpoint chain.                                                                                            | Tampering is reported, not repaired.                                               |

## Fail-closed choices

These refuse instead of degrading, because degrading would widen access or lie about a
deletion:

- **Revocation.** Until a process has loaded the revoked-token set once, every token gets
  503 `AUTH_UNAVAILABLE`. After that, a ClickHouse outage keeps the last snapshot.
- **Authorization.** A failed read of the subject's grant map answers 503, unless the
  process holds a cached entry younger than `LOOM_JWT_REVOCATION_MAX_STALE_SECONDS`
  (600 s) that no grant change has superseded, which it serves
  ([tokens](/architecture/security#tokens)). The single grant lookup
  (`authz.grant_role_for`) counts a failed read as no grant. A restricted namespace whose
  classification cannot be read is denied.
- **Boot migrations.** A failed migration stops the process before any worker starts.
- **Erasure.** A federated backend that cannot confirm deletion fails its step; residue
  anywhere in scope fails the saga.
- **Dev signing secret.** A deployment that requires auth refuses to start with the
  checked-in secret.

These fail open, counted or logged, because refusing would take the service down for an
availability control: traffic classification, the rate limiter, the idempotency cache read,
the audit insert (except the two restores in [data-loss boundaries](#data-loss-boundaries)),
boot calibration, the embedding caches and every telemetry write. See
[security](/architecture/security).

## When a dependency is down

| Down                  | What happens                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ClickHouse            | `/v1/health` and `/v1/health/ready` answer 503; `/v1/health/live` stays 200. Requests fail with `OVERLOADED`, `BACKEND_TIMEOUT` or a 500 carrying the request id. Tokens verify on the last revocation snapshot. Workers fail their passes and back off.                                                                                                                                                                                                                                                                                                                                                                   |
| Embedding provider    | `memory.set` and `memory.bulk_set` fail: no fact is stored without its vector. Paths that can wait (`memory.promote`, chunk embeds) store a stub for `embedding_worker`. A conversation turn is stored with its embed error recorded and stays unembedded (nothing retries it); `conversation.compact` writes no summary and tombstones no turn (`embed-failed`). `memory.search` fails with 503 `EMBED_TIMEOUT` unless the query's vector is in the embedding cache. A hung provider sets no health flag (the log shows `query embed exceeded`); one that answers errors shows `provider_failing:<name>` on `/v1/health`. |
| Embedding key missing | The process serves the fake embedder, with an error log and a `problems` entry; searches match only rows written under the fake model.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| LLM provider          | Each call retries (four attempts). Fail-open stages (HyDE, rerank, the extraction verifier, titles) are skipped; extraction and `memory.ask` synthesis have no substitute. `LOOM_LLM_FALLBACK_MODEL` hands an outage to a second model.                                                                                                                                                                                                                                                                                                                                                                                    |
| Document bucket       | Uploads and downloads of originals kept in the bucket answer 503 `BACKEND_UNAVAILABLE`, and a download of one that is gone 410. An erasure step that reaches the bucket fails closed and `erasure_recovery` retries it. The originals sweep records the pass's failure and its bucket walk resumes from its cursor on the next pass.                                                                                                                                                                                                                                                                                       |
| Langfuse, ClickStack  | Tracing and export drop; requests are unaffected. A configured erasure backend that is down fails its saga step, and recovery retries it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

**Why `memory.set` fails instead of storing a stub:** a stub is invisible to every search
leg until embedded, so a caller told "stored" would not find its own write. Failing lets it
retry with its idempotency key.

## Blue/green rollback

`./redeploy.sh` starts the idle colour and switches only after its `/v1/health` reports
`status: ok`, ClickHouse reachable, `schema.pending` empty, no degraded provider and no
embedding width mismatch, within `LOOM_REDEPLOY_READY_TIMEOUT_S` (a script variable, 180 s).
The old colour is stopped, not removed, and its image tagged `loom:prev`.
`./redeploy.sh --rollback` starts it and switches back. Every migration is additive
(`tests/unit/test_migrations_additive.py`), so the previous build runs against the newer
schema. See [upgrades](/operations/upgrades).

## A replica that dies mid-migration or mid-reconfiguration

Every process is a provider participant: one row in `loom._provider_participants`, written
every 10 s from a thread of its own, and a User-Agent naming its id on every ClickHouse
query it sends. A participant counts as stopped when it retired, or when its last heartbeat
is older than 60 s (server clock) and `system.processes` shows no query of its own on any
node. Nothing else counts: an unreachable process is not a stopped one.

- **Killed mid-migration.** Migrations hold the migration lock only (`ttl_s`, renewed while
  they apply), never the provider lock, which a boot holds just to check that no rebuild is
  unfinished. A second container takes the migration lock over once it expired and applies
  what is left.
- **Killed holding the provider lock** (a rebuild step, a saved configuration). The key's
  TTL is 60 s, renewed every 20 s; a contender takes it over once it expired and the
  holder's participant counts as stopped. `loom provider-lock clear --holder <holder>`
  frees one whose holder has no participant row, which never counts as stopped.
- **Killed mid-reconfiguration.** An expired heartbeat remains a blocker until the
  exact process dies or completes guarded shutdown. A trusted local process can
  verify its PID, namespace, boot and start identity, then retire it after all
  cross-replica queries settle. Use `loom participant retire --participant <id>` on
  that trusted host. Pod deletion or replacement and namespace changes alone do
  not prove death. The workers leader prunes only certified permanent retirements
  older than a day; unknown legacy rows remain visible.
- **A boot that fails after enrolling** remains nonretired unless all its actual
  producers and writes stop. A shutdown deadline does not turn cancellation into
  a retirement certificate.

A process frozen past its heartbeat (a stopped container, a VM pause) holds the
transition until it resumes and drains, or an operator retires it.

## Data-loss boundaries

- **Acknowledged writes.** A successful `memory.set` has committed both its rows.
- **Telemetry and audit rows.** A crash loses the unflushed `agent_events` queue (flushed
  every 100 ms) and events-stream queue (every `LOOM_EVENTS_STREAM_FLUSH_INTERVAL_S`, 1 s).
  The events stream carries budget violations only. Audit, exposure and search-trace rows
  are inserted directly; a failed audit insert is logged and fails only the two ops that
  depend on their record, a `memory.restore` that undoes a retirement and
  `namespace.restore` ([audit limits](/features/observability/audit#limits-and-failure-modes)).
  Retrieval reads none of them.
- **After-response projections** lost in a crash are rebuilt by the repair workers.
- **ClickHouse OSS single node.** No replication and no point-in-time recovery. ClickHouse
  Cloud, or a replicated OSS cluster, is the production path. Back up with ClickHouse
  `BACKUP` or native backups; `dev/backup_loom.py` moves a deployment, an org or a
  namespace, it is not disaster recovery. See [backup and recovery](/operations/backup-and-recovery).
- **Files outside ClickHouse.** Document originals are in ClickHouse
  (`loom.document_blobs`), or, for those written with `LOOM_DOCUMENT_OBJECT_STORE=s3`,
  in an S3-compatible bucket, which is backed up with the bucket's own versioning or
  replication; a lifecycle rule bounds how long those copies keep erased originals. Code
  mirrors live on a volume and re-clone from the remote ClickHouse records on a host
  without them. Provider reconfiguration keeps nothing on the host
  ([provider reconfiguration](/architecture/provider-reconfiguration)).
- **A move to another host or ClickHouse.** Carry ClickHouse, the env config,
  `LOOM_JWT_SECRET` and, while rows name objects in it, the document bucket; everything
  else comes back
  ([moving a deployment](/operations/backup-and-recovery#moving-a-deployment)).

## Capacity

A server process keeps no durable state, so web capacity scales by running more processes.
ClickHouse does the retrieval work (vector scans, text indexes, graph walks), so capacity
past the Python tier is ClickHouse capacity. The workers do not scale by replication: one
supervisor runs them. Procedures are in [deployment](/operations/deployment) and
[performance tuning](/guides/performance-tuning).

| Tier                       | Scales by                                                  | Bounded by                                                                |
| -------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------- |
| Web (HTTP, MCP, dashboard) | More containers behind a load balancer; `LOOM_WEB_WORKERS` | Python CPU per process; the per-process pool; per-replica rate limits     |
| Background workers         | One supervisor per deployment; a larger workers container  | Pass duration against interval; provider quota for LLM and embedding work |
| Retrieval and storage      | ClickHouse compute, memory and storage                     | Rows scanned per tenant read; merge pressure under ingest                 |
| Providers                  | Provider quota and batch sizes                             | Provider rate limits; `loom.embed.pending` shows the embedding backlog    |

### Web tier

| Knob                             | Effect                                                                                                                                                                                                                                       |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| More web replicas                | N identical replicas behind a load balancer, all with `LOOM_BACKGROUND_WORKERS=auto`.                                                                                                                                                        |
| `LOOM_WEB_WORKERS`               | uvicorn processes per container; unset, the image starts one per CPU the container may use, at most one per GiB of memory and at most 8. Each process runs its Python on one core; the local embedder gives each an equal share of the CPUs. |
| `LOOM_REPLICA_COUNT`             | Floor of the replica count that divides each rate-limit bucket (the live serving `loom.instances` rows, below).                                                                                                                              |
| `LOOM_LATENCY_BUDGET_MULTIPLIER` | Scales every op's `budget_p99_ms` before a budget violation fires (default 1.0).                                                                                                                                                             |

A new replica serves once `/v1/health/ready` answers 200: ClickHouse answers, no
migration is pending, the revocation snapshot and signing keyring are loaded and fresh,
the tracker registry and runtime overrides have loaded once (boot waits up to
`LOOM_BOOT_WARMUP_TIMEOUT_S` for both), the similarity calibration is adopted, its
configuration matches the deployment's, and the sandbox runs where it is required. On
SIGTERM it answers 503 there and keeps serving for `LOOM_SHUTDOWN_READY_DELAY_S` before
its port closes. Read-your-writes across processes is handled by ClickHouse (waited
inserts, committed or batched, sequential re-reads), not process affinity.

### ClickHouse client

`server/core/ch.py` gives each process one pool and one admission semaphore of the same size.

| Setting                                         | Default | Effect                                                                                                                 |
| ----------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
| `LOOM_CH_POOL_MAXSIZE`                          | 64      | Connections and concurrent statements per process; a batched insert holds one for its wait and flush.                  |
| `LOOM_CH_POOL_WAIT_MS`                          | 500     | Wait for a slot; past it 503 `OVERLOADED` with `Retry-After: 1`. 0 disables the gate.                                  |
| `LOOM_CH_TIMEOUT_READ_S`                        | 30      | Read deadline.                                                                                                         |
| `LOOM_CH_TIMEOUT_WRITE_S`                       | 60      | `INSERT` / `UPDATE` deadline.                                                                                          |
| `LOOM_CH_TIMEOUT_DDL_S`                         | 300     | DDL, `OPTIMIZE`, `SYSTEM` deadline.                                                                                    |
| `LOOM_CH_BATCHED_INSERTS`                       | true    | Batch the inserts written on every request, each holding a connection for its wait; false commits them, one part each. |
| `LOOM_CH_BATCHED_INSERT_WAIT_MIN_MS`            | 10      | Shortest wait of a batched or waited async insert, what a lone insert waits; the connection is held for the wait.      |
| `LOOM_CH_BATCHED_INSERT_WAIT_MAX_MS`            | 50      | Longest wait of a batched or waited async insert a request waits on; the connection is held for it.                    |
| `LOOM_CH_BATCHED_INSERT_BACKGROUND_WAIT_MAX_MS` | 200     | The same for the usage-event flusher's ticks and drain and the write-mark flush.                                       |

**Why a bounded wait:** when ClickHouse slows, an unbounded queue turns every request into
a client timeout after the work was queued anyway. A short wait fails fast with a retryable 503. The gauges `loom.ch.pool.in_use`, `waiters`, `overloaded_total` and `timeouts_total`
show which side is saturated.

### Traffic classes

Every call has a class: `production`, `interactive` (default), `bench` or `background`
(set by the supervisor, never by a request). It comes from the token claim, else the
namespace registry, else the request hint. It changes scheduling, never results:

- ClickHouse read settings per class (`LOOM_CH_SETTINGS_BY_TRAFFIC_CLASS`): `production`
  priority 1, `interactive` 2, `background` 5 with 3 threads, `bench` 10 with 4 threads.
- Rate-limit buckets (`LOOM_RATELIMIT_CLASS_FRACTIONS`; `bench` gets half).
- LLM policy per class (`LOOM_LLM_POLICY_BY_TRAFFIC_CLASS`).

A fresh `bench-…` namespace is classed `bench`, so a load test does not starve production.

### Rate limits

`server/core/ratelimit.py` keeps token buckets per (namespace, op class) and per (org, op
class); the org tier is three times the namespace tier. An optional per-actor tier takes a
fraction of the namespace bucket (`LOOM_RATELIMIT_ACTOR_FRACTION`, 1.0 = off). A call must
pass every bucket, and a refusal decrements none.

| Op class           | Covers                                                                                                        | Capacity | Refill / s |
| ------------------ | ------------------------------------------------------------------------------------------------------------- | -------- | ---------- |
| `memory`           | `memory.*`, `conversation.*` (except the exact-key reads and `memory.import`)                                 | 100      | 17         |
| `memory.read`      | Exact-key reads: `memory.get`, `bulk_get`, `history`, `working_memory.get`, …                                 | 300      | 50         |
| `memory.import`    | `memory.import`                                                                                               | 10       | 2          |
| `knowledge.search` | `knowledge.search`                                                                                            | 20       | 3          |
| `knowledge.query`  | `knowledge.query`                                                                                             | 20       | 3          |
| `knowledge.read`   | The other `read_only` `knowledge.*` ops, `knowledge.inspect` aside: `knowledge.get`, `knowledge.list`, …      | 50       | 10         |
| `knowledge.add`    | Every other `knowledge.*` op, `knowledge.inspect`, `knowledge.synthesize` and `knowledge.dedup_list` included | 10       | 2          |
| `code`             | `code.*` (except `code.outcome`, which draws from `memory`)                                                   | 60       | 10         |
| `insights.query`   | `insights.query`                                                                                              | 20       | 3          |
| `assistant.ask`    | `assistant.ask` and the assistant's chat route                                                                | 10       | 1          |
| `default`          | Everything else                                                                                               | 50       | 10         |

These are the namespace-tier defaults (`DEFAULT_LIMITS`, `op_class_for`):

- **Per replica.** Buckets live in the process or host. Each replica holds the limit
  divided by the hosts with a live serving `loom.instances` row, so the replicas' sum
  approximates the limit. `LOOM_REPLICA_COUNT` is the floor of that count, and the count
  while the rows cannot be read.
- **Within a host.** With `LOOM_WEB_WORKERS > 1`, processes share one bucket table in a
  memory-mapped file (`LOOM_RATELIMIT_SHARED=auto`, `LOOM_RATELIMIT_SHARED_PATH`,
  `LOOM_RATELIMIT_SHARED_SLOTS`). `/v1/health` reports `ratelimit.backend`.
- **Fail open.** A limiter error admits the call and is counted.

**Why local buckets:** a ClickHouse round trip per admission would put the database on
every request's path to protect the database.

### Background workers

One supervisor runs per deployment. In `auto` mode every process of the deployment
(`LOOM_DEPLOYMENT=1`) contends for one lease in ClickHouse: key `workers_supervisor` in
the Keeper lock table `loom._migration_lock` (strict KeeperMap). Outside the deployment
(an iso or branch server on the same ClickHouse) `auto` runs no workers and the boot log
says so; `on` runs them there, but only the deployment's holder does the leader's work
(retention ALTERs, tracker namespace rows, the published configuration, the provider
participant sweep).
Expiry is judged on the ClickHouse server's clock. With T = `LOOM_SUPERVISOR_LEASE_TTL_S`
(default 30 s, at least 24):

| Step   | When                                                                         | Why                                                                                         |
| ------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Renew  | every T/6; a failed renewal is retried every T/12                            | Several failures fit before the holder gives up                                             |
| Lose   | T/2 after the last renewal that landed was sent                              | Measured from the send, so the server's stamp is never earlier                              |
| Stop   | at that deadline, even while a renewal call is still blocked                 | A hung call cannot keep the workers running; each call is sent once, timeout max(1, T/24) s |
| Drain  | up to T/6, so the workers are stopped by 2T/3                                | A workers child still running then is killed                                                |
| Expiry | T after the server stamped the last renewal; then another process takes over | T/3 of margin after the drain                                                               |

A holder that missed its renewals starts no new pass until it renews (a paused process
resuming does not run one beside the new holder), and one whose Keeper key still names it
takes the lease back at once instead of waiting out the TTL. A holder that finds the lease
taken stops its workers; when the holder dies, another process takes over after T. A
process that becomes the holder after boot does the leader's boot work too (tracker
namespace rows, published settings, `LOOM_RETENTION_*_DAYS`).

Stopping the workers cancels each pass at its next `await`. Work the cancel cannot reach
(a thread such as the PageRank or community computation, a pass that resumes after a
pause before its lease loop runs) stops at the pass's next `pass_context.check_stop()`,
which the long loops call between batches: the ingest legs, the SCIP delta rounds,
`code_pr_history`, `code_graph_stats` and `entity_communities`. It raises once the pass
was stopped or the lease is no longer fresh, so the pass sends no further write. There
are no fencing tokens: a statement already sent still lands.

A process stands aside (the holder stops its workers and gives the key up; a contender
does not ask) while:

- it drains after SIGTERM, so another process takes over during its ready delay;
- its configuration differs from the deployment's published one (it is not ready, and
  could write vectors in the wrong space or sign with the wrong root);
- a live `loom.instances` row of a newer build contends for the same lease.

The last is the rolling-upgrade hand-over. Each row carries its build's schema version
(its last migration number) and the lease it contends for right now: none while that
process drains, has drifted or gave the lease up, so such a row never makes an older
build yield. The hand-over fires only when the schema versions differ, so a code-only
release never hands over: the old holder keeps the workers until it stops. Once no
newer build is live (a rollback), the older one contends again. A process whose workers
child keeps failing to start gives the lease up for five minutes (below).

So:

- Several hosts can run in `auto`; one of them runs the workers. `on` skips the election.
- A second copy of the maintenance workers wastes ClickHouse work; a second copy of the LLM
  workers doubles their scans and spend.
- The holder runs the workers in a child process (`LOOM_WORKERS_PROCESS=child`), so a
  long pass does not share the event loop that serves its requests (passes on the
  serving loop roughly doubled request p90: `memory.get` 148 ms against 74–79 ms during
  a `code_ingest` pass). The child runs the
  supervisor and what its workers use (the ClickHouse client, the provider configuration,
  the event emitter, the tracker and repository registries), not the serving lifespan.
  It reads the lease deadline from its parent every second; the deadline stops moving
  once the lease is not fresh. The child is gone by that deadline plus T/6: it exits by
  itself then, its parent kills it then, and the kernel kills it when the parent dies
  (`PR_SET_PDEATHSIG`). The child's event loop beats to its parent every second, and
  `/v1/health` shows a child with no beat for 15 s as not alive. A draining process
  (SIGTERM) stops its workers, inline or in a child, and gives the lease up while it
  still serves. A child that exits within 60 s of starting five times in a row makes its
  process give the lease up and report `workers_lease_given_up`; another process takes
  the workers, and this one asks again five minutes later.

Worker work is not partitioned across processes. A worker keeps up through bounded passes:
a batch or time budget per pass (for example `embedding_worker` embeds up to 100 stubs;
`usage_fold` folds up to 1000 memories within 120 s and backs off at 20 pending mutations)
and the deadline above. A pass that routinely hits its deadline means the workers
container needs more CPU or the provider more quota.

### Batch sizes and queues

| Path                              | Bound                                                                                                      |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Request body                      | `LOOM_MAX_REQUEST_BYTES` (32 MiB); 413 above it                                                            |
| `memory.bulk_set`                 | Up to 1000 entries in one `embed_many`; conflict checks up to `LOOM_BULK_CONFLICT_CHECK_MAX_ENTRIES` (200) |
| `memory.bulk_get`                 | Up to 1000 keys                                                                                            |
| `memory.set_from_messages`        | Up to 400 messages                                                                                         |
| Embedding requests                | Up to 100 inputs per provider request (OpenAI, Gemini)                                                     |
| `agent_events` emitter            | Queue of 10,000 rows, flushed every 0.1 s or 500 rows                                                      |
| Events stream (budget violations) | Queue of 20,000, flushed every `LOOM_EVENTS_STREAM_FLUSH_INTERVAL_S` or `_FLUSH_BATCH` rows                |
| Worker status rows                | Flushed every 15 s or 200 rows                                                                             |

### Vector cost

A memory stores up to two vectors (value and key) of `Array(BFloat16)`: 6 KiB at 1536
dimensions, 12 KiB at 3072. Storage and scan bytes grow linearly with `LOOM_EMBEDDING_DIM`.
There is no HNSW index, so a read scans every vector of the tenant's namespace and latency
grows with namespace size. Quantized codes shrink the scan where the codec can be on
([gated settings](/architecture/clickhouse#gated-clickhouse-settings)).
Details are in [ClickHouse](/architecture/clickhouse#vector-storage).

### Measuring

This page states no latency or throughput numbers; they depend on the ClickHouse
deployment, network distance and namespace size. Measure your own, against an isolated
server:

| Command                                                     | Measures                                                                                               | Output              |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------- |
| `make bench OP=memory.set C=10 D=30`                        | p50 / p95 / p99 and RPS of one op at concurrency C for D seconds (`scripts/ops/benchmark.py`)          | stdout, or `--json` |
| `python bench/search_latency.py --namespaces … --queries …` | `memory.search` p50 / p95 per namespace size band, with per-stage times from `retrieval_work.stage_ms` | `--out` CSV         |
| `make bench-quick`                                          | Retrieval quality on a 30-question sample (needs real providers)                                       | CSV under `/tmp`    |

`scripts/ops/benchmark.py` writes into a fresh `bench-…` namespace, so it runs in the
`bench` class; a `memory.search` run needs a namespace a `memory.set` run filled. See
[running benchmarks](/benchmarks/running).

### What to scale

| Signal                                                          | Scale                                                     |
| --------------------------------------------------------------- | --------------------------------------------------------- |
| Web CPU high, request p99 rising, ClickHouse answering fast     | Web replicas or `LOOM_WEB_WORKERS`                        |
| `loom.ch.pool.waiters` and `OVERLOADED` rising, ClickHouse fast | `LOOM_CH_POOL_MAXSIZE`, or more replicas                  |
| `BACKEND_TIMEOUT` and slow `system.query_log` entries           | ClickHouse compute, or smaller namespaces per read        |
| `loom.embed.pending` rising                                     | Embedder throughput (provider quota)                      |
| `loom.worker.consecutive_failures` or pass durations growing    | The workers container's CPU and memory                    |
| Merge pressure, memory errors under ingest                      | ClickHouse compute and memory, or lower write concurrency |

The metrics are described in [monitoring](/operations/monitoring).

### Known limits

- **One worker leader per deployment.** Worker passes are not spread across hosts.
- **Per-replica rate limits** are approximate: bucket fill is per host. The sign-in and
  sign-up attempt limits are counted across every process in `loom.auth_attempts`.
- **Check-then-insert across replicas** (a sign-up's name and email, a personal
  namespace, an OAuth code or refresh token, an idempotency key in flight) runs under a
  deployment-wide claim, a strict KeeperMap key in `loom._migration_lock`, with the
  check read sequentially. A second
  contender gets a 409; a claim store that cannot answer is a 503 (an idempotency key
  then runs as a plain miss).
- **Events-stream telemetry** (budget violations) is drained on `SIGTERM`; a `SIGKILL` or
  OOM loses at most one flush interval of it, never audit or memory rows.
- **Namespace size drives read latency**, because vector reads scan the namespace's rows
  with no vector index.
- **Single-node ClickHouse OSS** has no replication and no failover.

## Knowledge restore

`knowledge.restore` ([restoring a deleted document](/features/knowledge#restoring-a-deleted-document))
must never bring back what an erasure reached. An erasure leaves its own marks: it stamps
every chunk of its scope, a deleted document's too, and supersedes a live row in place.
Restore also reads the erasure ledger for an erasure that may reach the document and had
not finished when the document was written: of its namespace, of a namespace list naming
it, or of the user who wrote the restored revision (of any user, for the live leftovers of
a cut-off restore, which no signature ties to a writer).

It reads the ledger before writing the row and again after: when the second read finds an
erasure, it takes back what it wrote, and it does the same when anything fails once it
starts writing. Under a purge it deletes its own row, so no copy of the content is left.
When a merge has folded the delete's row into it, that leaves the source with no row: a
purge that covers the document would leave none either, and one that only may cover it
(an unreadable list, or any user's purge for a cut-off restore's leftovers) costs the
document's retired row, which is the fail-closed choice.
Otherwise, and when the ledger cannot be read, it writes the row back retired, keeping any
mark an erasure put on it meanwhile: deleting it could leave the source with no row, since
a merge may fold the delete's row into it at any moment. When no row of the source is left
(a purge took them all), it writes nothing back.

Under the Postgres authority restore writes through the document heads ([storage
authority](/architecture/storage-authority#knowledge-documents)). It refuses, retryably, while a
publication of the source may still land in ClickHouse (one owed, or one the delivery
worker took over and settled within the insert bound of its lease), since a delete's
retirement of the chunks could otherwise land after it brought them back. Its write-back
commits over the version it read, so a text write that took the source meanwhile refuses
it (409). One that commits and cannot publish is undone and answers 503. A purge undo
also takes back the restored head and its delivery, and waits out that delivery's lease
before it deletes the ClickHouse row.

A restore whose caller goes away (the client disconnects, the request is cancelled)
before it answers still runs to its end under the claim and takes itself back, since no
audit row would record it; its undo finishes before the claim is given up. A caller that
goes away after the restore answered, while the transport writes its audit row, can leave
the restore standing with no audit row, as any audited write can.

What it cannot take back:

- A restore cut off before its undo leaves the chunks searchable under a retired row;
  calling it again finishes it. A delete that runs first retires those chunks stamped at
  the document's own delete, so a restore still finds them.
- An erasure that registers and completes between its two ledger reads, followed by a
  failure of the undo (or of the second read and the undo, or the process ending between
  them): the row stays live, and a later call answers `live`.
- A purge the undo cannot read the ledger for leaves the row retired: the purge's residue
  census, which counts every row, reports it only if it runs after the undo.
- The original sweep deletes an original no live row names once it is old enough, and
  checks only its age at the delete. A sweep pass that read the namespace while the
  document was deleted can delete the original after restore found it, so the restored
  row names an original that is gone (and answers `kept`); the download then fails. Any
  write that names an existing original again shares this window.

Concurrent writes: a text write (`knowledge.add`, `knowledge.upsert`) and the delete of a
live document take no claim. Under the ClickHouse authority, a text write that lands on
the same source during a restore can leave both versions' chunks searchable, as it can
during a delete, until the next write of that source; one that started before the
restore stores its row under a lower version, so the restored row hides it, as a
delete's row hides one that started before the delete: write it again. Under the
Postgres authority, whichever of the two commits second answers 409 and retires its own
chunks. When a restore takes itself back after a later write's row won, it leaves that
row as it is and retires only its own chunks. A delete of an already deleted source
retires its leftover chunks under the source's claim, reading the document again under
it with sequential consistency: one a restore brought back meanwhile is live then and
keeps its chunks. It retires and counts only the chunks up to the tombstone; a newer
write's chunks, still to commit, stay, and a delete that finds only those answers that
nothing was deleted.

## Knowledge move

`knowledge.move` ([moving documents](/features/knowledge#moving-documents)) writes each
document into the namespace it goes to, then deletes it where it was, so a cut-off move
leaves the document in both places and never in neither. Per document, it reads the place it
goes and answers `exists` for another live document there before it writes anything; under
the claims it reads both again. It then stores the original and the extraction artifact in
the new namespace, outside any claim (an original no row names yet is the sweep's, as any
upload whose row did not land). An original held as a ClickHouse row is copied inside
ClickHouse; one in the bucket, or one moved there meanwhile, is read and written again under
the new namespace's place there. An original an erasure took (`blob_erased_at` on the
document) is not copied, even when the bytes of its sha are still kept for another upload.

Then, under the source's claim in both namespaces, taken in sorted order and not waited for
(a held source answers `busy` at once), it refuses (`busy`) when either place changed since
it read them, when a publication of either place may still land, or while an ingest job of
either is open (its publish would write the document back where it was, or over the copy).
It copies the chunks of the document's revision, with their vectors, inside ClickHouse
(`INSERT … SELECT`, every version the row serves: a metadata edit leaves the chunks at their
own version), then the tables, then the row, and deletes the source over the version it
read. A copy left by an earlier move that stopped before its delete (the same text, title,
description, tags, metadata, author and ingest time where it goes) is not written again when
it serves as many live chunks as the source and the same tables: only the source is deleted.
One with fewer chunks, or with no tables where the source has some (a cancelled row insert
can land after its take-back, which leaves the row alone), is written over, as a deleted
document is, before the source is deleted. One with more is whole: a delete cut off between
retiring the source's chunks and writing its tombstone (it retires them first) leaves the
source row live with none, and only the delete is left; a copy is never written over one
that serves more chunks than the source: one that also lacks its tables answers `exists`.
Tables there that differ otherwise were changed on one side since (an approval, a
clarification), which cannot be told from a change on the other: that copy, like one whose
text or metadata was edited, is another document and answers `exists`.

Anything that does not land whole, or a write that took either place before the delete,
takes back what the move wrote where it goes, and that source answers `busy`. The take-back
runs to its end before the claims are given up, however often the request is cancelled
meanwhile (the cancel is raised once it is done). Its first statement retires the copy's
chunks under every embedding model at once; a take-back starts by its claims' cut-off, or
right after a delete begun before it that the Postgres authority refuses, whose commit
answers within `pg_commit_lock_timeout_ms` and `pg_statement_timeout_ms` (2 s and 5 s by
default). With the statement's own bound (15% of the TTL, 9 s), that retirement ends by
about 40 s, inside the claim's 60 s TTL. The rest can outlast it, but a later move that then
takes the claims reads the copy short: under the claims it checks again that an earlier
move's copy is whole, answers `busy` when it is not, and on a later call writes the copy
again; the other sources of the call still move. After the retirement the take-back drops
its commit (under the Postgres authority), deletes its row, then its chunks and its tables:
a take-back that stops partway leaves no chunk serving without a row, and a cancelled
`INSERT … SELECT` that lands after the chunks' delete is retired too. The work under the
claims is cut off at 40% of the claim's 60 s TTL (24 s); a document whose copy runs past
that is taken back and answers `busy`. The delete runs to its end however the request or the
claims end meanwhile, and the document is counted as moved the moment its tombstone is in,
so a cut-off or a cancel while the tombstone is in flight still answers moved and audits it,
and a tombstone the Postgres authority refuses then still takes the copy back. A failure of
the delete itself (a lost answer, which fails the call) may follow the tombstone, so the
copy stays: the document is in both places, or only where it goes, and a retry finds the
copy and only deletes the source, or answers `not_found`. Once the tombstone is in, the move
is made: the source's tables are deleted after the claims are given up, as a delete of a
live document takes none, so no cut-off reaches it; a failure there is logged, and a later
delete of the source settles them. A document counts as moved the moment its tombstone is
in, under the claims: its audit row and its table cleanup follow whatever ends the call
after that (a cancel as the claims are given up, or during the cleanup). A failure that is
not one source's (a store that does not answer) is raised once every source of the call has
answered, after the audit row of those that moved.

The erasure ledger is read for each document twice, as `knowledge.restore` reads it
(`unfinished_reaching` for the document's namespace and writer, filtered with `may_cover`):
before its original is stored, for an erasure that may reach it where it is (unfinished when
its text was written there: its first chunk's `created_at`, which a move stamps with when it
copied the chunks, while the row keeps its ingest time) or the place it goes (unfinished
when the move started), and again under the claims, just before its delete, for one
registered since: one may have passed the copy, this move's or an earlier move's. That one
takes back a copy this move wrote, leaves an earlier move's copy to the erasure, and answers
`erased` without deleting the source, so an `erased` document always stays where it was.
Under the Postgres authority a delete over a document an erasure removed meanwhile answers
`erased` too. The first read comes just before the original is stored in the new place, so
an erasure whose originals step could miss it must register in the moment between that read
and the store; the re-read before the delete then answers `erased`, and that original, named
by no row, is the sweep's after `document_blob_min_age_seconds`, as any original whose row
did not land. The move deletes no original itself: one is keyed by its bytes, and a
concurrent write of the same bytes whose row has not landed yet may need it.

Concurrent writes: a text write takes no claim. Under the Postgres authority the copy's row
and the delete each commit over the version read, so a write that took the source or the
place first refuses them (409), and the copy is taken back. Under the ClickHouse authority
the move reads the target's head under the claims before it copies and again after its row:
a write with a later version wins and the copy is taken back, while one that landed between
that first read and the row (the whole chunk and table copy), with an earlier version, is
replaced, as with two text writes of one source.

## Invariants

- A successful write has committed its primary rows; a failure of either fails the request.
- A retried INSERT carries its deduplication token, or is not retried.
- `OVERLOADED` and `AUTH_UNAVAILABLE` carry a retry hint; `SCHEMA_PENDING` waits on an
  operator.
- A worker pass is idempotent: running it twice converges.
- Access checks and erasure fail closed; availability controls and telemetry fail open,
  counted.
- No request waits unboundedly on the pool, a provider or a queue. Every statement waits at
  most `LOOM_CH_POOL_WAIT_MS` for a slot and runs at most its class deadline.
- A traffic class changes scheduling and buckets, never the rows returned.
- Every migration is additive, so the previous build runs against the current schema.

## Source entry points

| What                     | Where                                                                                                                                                                                                                                       |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Client retries and gates | `server/core/ch.py` (`_with_retry`, `_acquire_slot`, `_bounded`, `statement_timeout`), `server/core/ch_schema_guard.py`                                                                                                                     |
| Error codes              | `server/core/errors.py`                                                                                                                                                                                                                     |
| Idempotency              | `server/core/idempotency.py`                                                                                                                                                                                                                |
| Supervisor and lease     | `server/workers/supervisor.py`, `server/workers/leader.py`, `server/workers/registry.py` (`default_max_pass_seconds`)                                                                                                                       |
| Repair workers           | `server/memory/workers/projection_repair.py`, `orphan_vector_reconcile.py`, `embedding.py`, `entity_extractor.py`; `server/erasure/workers/recovery.py`; `server/tasks/workers/lease_sweep.py`; `server/insights/workers/audit_verifier.py` |
| Revocation snapshot      | `server/auth/revocation.py` (`load_at_boot`)                                                                                                                                                                                                |
| Query-embed bound        | `server/memory/app/search/embed.py`                                                                                                                                                                                                         |
| After-write projections  | `server/memory/app/after_write.py`, `server/memory/app/projection_failures.py`                                                                                                                                                              |
| Telemetry queues         | `server/observability/events.py`, `server/observability/events_stream.py`, `server/workers/status_store.py`                                                                                                                                 |
| Traffic classes          | `server/observability/traffic_class.py`, `server/identity/app/traffic_class_registry.py`                                                                                                                                                    |
| Rate limiter             | `server/core/ratelimit.py` (`DEFAULT_LIMITS`, `op_class_for`), `server/core/ratelimit_shared.py`, `server/auth/attempts.py`                                                                                                                 |
| Schema status            | `server/core/schema/status.py`                                                                                                                                                                                                              |
| Blue/green               | `redeploy.sh`, `dev/docker-compose.colour.yml`                                                                                                                                                                                              |
| Load bench               | `scripts/ops/benchmark.py`, `bench/search_latency.py`                                                                                                                                                                                       |

## Related

- [Troubleshooting](/operations/troubleshooting): symptom-first runbooks.
- [Monitoring](/operations/monitoring): the metrics named here.
- [Security](/architecture/security): the fail-closed access checks.
- [ClickHouse](/architecture/clickhouse): tables, vector columns, query attribution.
