Skip to content

Proxy: a conversation reserves a document's bytes before it reads it - #918

Merged
schmitch merged 1 commit into
mainfrom
claude/601-proxy-metadata-budget
Nov 17, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/601-proxy-metadata-budget

Conversation

@schmitch

@schmitch schmitch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Fixes #601

The problem

One upstream conversation keeps at most 4 × proxy.max_metadata_bytes of documents (#230), but converse counted a document only after fetch returned it. I traced one fetched document of size S on main, from the upstream read to the kept value:

step alive
download on disk past proxy.spool_memory_bytes
SpooledBody::into_bytes Vec S (grown by doubling, so up to 2 S of capacity)
CachedFetch::found + base64 String, 4/3 S
the cache's serde_json::to_vec + JSON Vec, 4/3 S (up to 8/3 S of capacity)
the cache's from_slice cache entry 4/3 S + a second String 4/3 S
base64 decode + Vec S, the value kept

So the issue's "3–4 copies" holds: about 11/3 S at the decode, more while the JSON grows. Four fetches run at once, each doing this beside what the conversation already kept, and a document past the budget was still read, stored and cached before it was dropped.

The change

  • Reserve before reading. ConversationBudget (crates/core/src/repo/proxy.rs) replaces the counter in converse. A fetch reserves its document's bytes as soon as it knows how many they are and before it has them in memory. There are three places: the upstream's answer (spooled to disk, its length known), the stored copy (proxy_document.size), and the local cache's copy (its size is read from the cached JSON without decoding it). A document that does not fit is not read into memory.
  • The read is limited to what is left, as the issue asks: the download's size limit is min(max_metadata_bytes, what the budget has left), so a response with a Content-Length over it is not downloaded at all.
  • One copy instead of three. The body is encoded straight into the cache's JSON and decoded straight from it (no intermediate String either way), and the cache allocates that JSON at its exact size. The only copy beside a document being read is the cache's encoding, 4/3 S.
  • At most one max_metadata_bytes of documents is encoded at a time per conversation. A second large document waits, with its bytes reserved and its body still on disk.
  • Docs: docs/deployment.md (Memory), docs/format-plugins.md, forge.toml.example and the config's doc comment state the bound below.

Deadlock and starvation

  • Nothing waits for bytes of the conversation's budget: a reservation succeeds at once or the fetch fails as over budget. A conversation never holds part of it while waiting for the rest.
  • The one wait is for the encoding permit. It is taken only inside a fetch that reads from the upstream or the stored copy, after its bytes are reserved. Whoever holds it waits for nothing else of the conversation (no budget, no other fetch, no cache key), and releases it when its document is cached and decoded, or when the fetch fails or is dropped.

The bound, before and after

In multiples of proxy.max_metadata_bytes (32 MiB by default):

before after
documents kept or being read 4 kept + 4 being read 4 in all
copies made to cache them about 8/3 per document being read, 4 at once 4/3, for at most one document's worth
one conversation, worst case about 19 × (600 MiB) 5 1/3 × (171 MiB)

Measured with crates/core/tests/proxy_memory.rs: a test binary with a counting allocator, an in-process upstream serving 8 documents of just under max_metadata_bytes (4 MiB in the test), and a format asking for all of them in one round.

peak during the conversation main this branch
local cache too small to keep a document 18.5–19.2 × 6.7–6.8 ×
local cache keeping the documents (the committed test) 28.5 × 9.4–9.5 ×
  • The committed test asserts 4 + 4/3 (the conversation) + 3 × 4/3 (the encodings the cache keeps of the three documents read before) + 2 MiB. It fails on main at 28.5 ×.
  • With a cache too small to keep a document, the branch measures one encoding (4/3) over 5 1/3: moka frees an entry it does not keep at its next housekeeping, not at once. That is the cache's memory, not the conversation's, and why the committed test uses a cache that keeps them.
  • The 600 MiB and 171 MiB figures are the measured and derived factors scaled to 32 MiB; I did not run the test at 32 MiB.

What remains outside the bound

  • Downloads in progress: up to proxy.spool_memory_bytes (2 MiB) of each stays in memory before it spills to disk, 4 at a time. The docs say so.
  • The local cache's copies are its own memory (cache.l1_max_capacity × 16 KiB), as before.
  • The instance's budget (proxy.max_metadata_memory_bytes, Proxies: one memory budget for the upstream documents conversations hold #673) is still taken after a document is read: fetch_held is unchanged apart from carrying the reservation. Across conversations, each can be one document and its encoding over that budget while it finds out the budget is spent.
  • The format's own parsing of the documents, as docs/deployment.md already says.
  • The docker proxy's tag-list rounds (crates/server/images/proxy.rs) are not touched.

Behaviour changes to know about

  • A document past the conversation's budget is no longer stored or cached on the way to being dropped. The next operation that wants it downloads it.
  • When two conversations read the same document at once and the one doing the read runs out of its budget, the other asks again instead of taking that failure (up to 3 times).
  • forge-cache gains get_or_try_insert_local_encoded_with_tags_expiring and serializes every value twice (once to count its bytes). BlobStore::read_all and SpooledBody::into_bytes allocate the known size at once.

#770

#770 is the same code and is not fixed here. This change does not add to it: the conversation's reservation is given back together with the document while fetch_held waits for the instance's budget, and nothing here waits on another conversation. If #770 is fixed by giving back the earlier rounds' holds and reading those documents again, the re-read goes through fetch and reserves again.

No admin API changes, so no admin screen.

Verification

Run locally, with Postgres 18 and versitygw from testcontainers (pulled through mirror.gcr.io):

Not run locally: the forge-server and forge-e2e suites (the session's disk did not hold the server's test build; CI's test and e2e jobs run them) and the web checks (nothing under web/ changes). The retry between two conversations reading one document has no test of its own.

🤖 Generated with Claude Code

https://claude.ai/code/session_012e6DJae3KEQWi1VkmDNZVw

One upstream conversation keeps at most 4 x proxy.max_metadata_bytes of
documents (#230), but a document was counted once it had been downloaded,
read into memory, base64-encoded for the local cache, serialized, parsed
back and decoded. The four fetches of a round each held their document and
those copies beside what the conversation kept already: 19 x
max_metadata_bytes measured (28 x with the local cache's copies), where
the docs said 4 x.

A fetch now reserves its document's bytes in the conversation's budget as
soon as it knows how many they are and before it has them in memory: the
upstream's answer is on disk then, the stored copy is a row's size, the
cached copy is the cache's own. A document that does not fit is not read,
and a download stops at what the budget has left. Nothing waits for bytes
of that budget, so a conversation never holds a part of it while waiting
for the rest.

The copies made on the way are down to one, the cache's encoding (4/3 of
the document): the body is encoded straight into the cache's JSON, which is
allocated at its size, and decoded straight from it. A conversation encodes
at most one max_metadata_bytes of documents at a time, so it holds at most
5 1/3 x max_metadata_bytes, which the docs now say.

Fixes #601

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJae3KEQWi1VkmDNZVw
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026

Copy link
Copy Markdown
Member Author

The merge queue removed this PR because test failed in run 1507 on one test, a_stalled_pull_of_a_downloading_layer_releases_its_staging_space_once_stored (crates/server/tests/registry.rs): its read_some waited more than 10 s for the first MiB of a docker layer.

I do not think this failure is this PR's:

Nothing is changed on the branch. It needs to be queued again, which I cannot do.


Generated by Claude Code

@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit e3dd7fe Nov 17, 2026
19 checks passed
@schmitch
schmitch deleted the claude/601-proxy-metadata-budget branch November 17, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants