Skip to content

Add LLM gateway caching guardrails skill and prompt - #94

Merged
rmkr-dev merged 2 commits into
mainfrom
add-llm-gateway-caching
Nov 11, 2026
Merged

rmkr-dev merged 2 commits into
mainfrom
add-llm-gateway-caching

Conversation

@rmkr-dev

Copy link
Copy Markdown
Owner

Adds the llm-gateway-caching-guardrails skill and matching add-llm-gateway-caching prompt.

The cost/latency entry prices cached tokens and the routing entry tracks capabilities, but nothing says when a gateway may reuse an answer or how to keep a cache from leaking across tenants or serving the answer to a different question. This pair covers that.

  • Three tiers rolled out in order of risk: provider prompt caching (no answer reuse), an opt-in exact response cache, and a semantic cache only on routes that qualify.
  • Prompt caching: static content first and byte-stable, per-provider markers and limits kept in config with a checked-on date, write premiums only where the prefix repeats, cached read/write tokens recorded for the cost ledger.
  • Exact cache key covers tenant, auth scope, resolved model, prompt and tool versions, normalized redacted messages, generation parameters, and corpus version. Only complete, valid, non-refused, non-tool responses are stored, with a TTL.
  • Semantic tier searches only inside exact-match partitions, skips requests with IDs, amounts, dates, or code, and takes its threshold from a labeled near-miss pair set per route, with the similarity-vs-distance direction written down.
  • Invalidation by version bump, source tag, and erasure request, plus a kill switch and purge per route and tier.
  • X-Cache status header, poisoning and leakage controls, metrics without payloads, and CI tests for keys, eligibility, near misses, invalidation, and cache outages.

Cross-links to the cost/latency, routing, PII redaction, structured output, RAG retrieval, eval gate, and Spring Redis cache entries rather than repeating them.

Only skillbook/skills.json and promptbook/prompts.json change. Catalogs are 56 skills and 56 prompts. Local site check and script tests passed.

@rmkr-dev
rmkr-dev merged commit 7913e85 into main Nov 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant