diff --git a/promptbook/prompts.json b/promptbook/prompts.json index c4b1bbf..4897688 100644 --- a/promptbook/prompts.json +++ b/promptbook/prompts.json @@ -1042,6 +1042,26 @@ ], "formatNotes": "All three. Point the tool at the gateway module, router and adapter classes (`**/*Router*.java`, `**/*Gateway*.java`, `**/*Adapter*.java`, `**/llm/gateway/**`), the routing config, and the existing HTTP client and breaker setup. Copilot: llm-gateway-routing path instructions plus those files in agent mode (or save the prompt as `.github/prompts/add-llm-gateway-routing-fallback.prompt.md`). Claude Code: the llm-gateway-routing-fallback-guardrails skill plus the router, adapter, and config files. Codex: the AGENTS.md LLM gateway routing section, then the diff. Leave cost tables, inbound rate limits, idempotency storage, and prompt-injection filters to their own skills.", "body": "Task: Add multi-provider routing and fallback to the LLM gateway I specify. Callers should name an alias. The gateway walks that alias's ordered targets, falls through only on retryable failures before any response bytes are committed, keeps per-target fallbackOnBadRequest off unless a target opts in. When the final target returns 400, 413, or 422, return that status. When the final target fails another way, return 502 and list every attempt, including an earlier 400 then a last 503, auth failures, and timeouts. Do not use 503 for that mixed walk or for auth exhaustion, and do not use 504 for timeouts. Finish with an inventory, a config and code diff, two-provider tests, and rollout notes.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Gateway name and module: {{GATEWAY_NAME}} at {{APP_PATH}}\n- Stack: {{STACK}} (Spring Boot version; Spring AI | LangChain4j | plain HTTP adapters | mixed)\n- Providers: {{PROVIDERS}} (e.g. Azure OpenAI deployments, Amazon Bedrock, OpenRouter, Together, Modal, other OpenAI-compatible base URLs)\n- Aliases: {{ALIASES}} (alias → current model or deployment, and the intended primary then fallbacks)\n- Strategy and stickiness: {{STRATEGY}} (priority | weighted) and {{STICKY}} (none | session, plus the session key and window)\n- Streaming: {{STREAMING}} (yes | no | mixed per alias)\n- Latency SLO and hard deadline: {{SLO}} (e.g. p95 8s, deadline 10s per alias)\n- Residency allowlist: {{RESIDENCY}} (regions each alias may call, or \"unrestricted\")\n- Observability stack: {{OBSERVABILITY}} (Micrometer/Prometheus | OpenTelemetry | both; correlation-id header name)\n- Bad-request fallback: {{BAD_REQUEST_FALLBACK}} (which target ids may set fallbackOnBadRequest, or \"none\")\n- Pain: {{PAIN}} (single-region outage | 400s returned as 502 | mid-stream model switch | no metrics | none)\n\nRequirements:\n\n1. Inventory\n - Read {{APP_PATH}}: routing or model config, gateway filter or controller, each provider client, retry and breaker config, and call sites that pass a raw model id\n - Table: alias or call site | provider | model or deployment | region | streaming | retry today | breaker today | secret source\n - Flag raw model ids outside the table, retries on the HTTP client and inside the gateway, 400s counted as breaker failures, and logs that store prompts or completions\n\n2. Routing table and selection\n - One alias per caller-facing model name. Each target has id, provider, model or deployment, region, secretRef, priority or weight, capabilities, minAttemptBudget, and fallbackOnBadRequest (default false)\n - {{STRATEGY}} for primary selection. Fallback order is fixed at the start of the request. Weighted choice is a pure function of the eligible set and a stable key\n - {{STICKY}}: per request, or pin the target that last succeeded for the session key. A failed target is not pinned. Sticky does not override health, {{RESIDENCY}}, or capabilities\n - Skip ineligible targets before HTTP: disabled, open breaker (except the half-open probe), missing capability, region outside {{RESIDENCY}}, remaining deadline below minAttemptBudget\n - One pass. Attempt cap equals the eligible list. Turn off HTTP-client retries for this POST so only the walk retries\n\n3. Failure classes\n - Adapters map vendor errors to timeout, rate_limit, upstream_5xx, connection, client_rejection (400/413/422), auth_config (401/403/deployment 404), or cancelled\n - Fall through on timeout, 429 (honor Retry-After only when it fits the remaining deadline; wait at most once on the last target), 5xx, connection, and auth_config\n - Client rejection falls through only when that target's fallbackOnBadRequest is true. {{BAD_REQUEST_FALLBACK}} is the only place the flag may be true\n - Client rejection does not increment breaker failures. Auth_config and upstream 5xx do. A 429 applies a short cooldown, not a long open\n - Capability gaps skip the target with reason capability_skip and never become a synthetic upstream 400\n\n4. Status when every target fails\n - fallbackOnBadRequest defaults off and covers only 400, 413, and 422\n - No attempt: 422 no_capable_target, or 503 no_eligible_target when the cause is health or deadline. Empty attempt list. These are not a failed upstream call\n - Final target returned 400, 413, or 422: surface that status. A 503 followed by a final 422 stays 422. Do not wrap it in 502\n - Final target failed another way (timeout, 429, 5xx, connection, or auth/config): return 502. The body lists every attempt in order, including an earlier 400 and the last 503, each auth status (401, 403, deployment 404), and each timeout. Do not return 503 for that mixed walk, 503 for auth exhaustion, or a blanket 504 for timeouts\n - Problem+json for the 502 carries code upstream_failed, alias, and attempts (target id, failure class, HTTP status when one exists). A surfaced 400, 413, or 422 carries the client-rejection code and that HTTP status. Do not copy the upstream error document\n\n5. Streaming ({{STREAMING}})\n - Flush no model bytes to the caller until the first token, so a pre-commit failure can still fall through\n - After the first flushed token, cancel the upstream, emit a terminal error (SSE event: error with code stream_interrupted, alias, and attempt), and do not call the next provider\n - On caller disconnect, cancel the upstream and stop the walk\n - Non-streaming calls stay buffered until the attempt settles so the status rules in section 4 still apply\n\n6. Adapters, secrets, deadlines\n - One LlmProviderAdapter per provider family. OpenAI-compatible backends (Azure OpenAI, OpenRouter, Together, Modal, and similar) may share one HTTP adapter parameterized by base URL, header style, API version, and deployment-versus-model path. Bedrock is its own adapter\n - The router has no vendor SDK imports and no provider-name branches\n - secretRef only in config. Load values from env or Key Vault. Fail startup when a base URL contains a key\n - Per-attempt read timeout is the remaining {{SLO}} deadline. Skip a target that cannot fit minAttemptBudget\n\n7. Budgets and observability ({{OBSERVABILITY}})\n - Child span per attempt under the inbound trace. One correlation id on every attempt log\n - Metrics: attempt latency histogram, attempt counter, fallback counter, response status-class counter. Labels stay within alias, provider, configured model, outcome, failure_class, reason, status_class\n - Pass every attempt, including failures, to the existing cost and budget hook before the next target starts. Do not add a price table or a quota ledger in this change\n - Alert notes (as code or as rollout notes): fallback-rate climb, any auth_config, no_eligible_target\n\n8. Safety\n - Attempt logs and spans omit raw prompts and completions. Leave a debug switch off in production config\n - Do not add a redaction engine or a prompt-injection filter here. Note the upstream component that should own redaction\n - Targets stay an allowlist. Ignore provider, region, and credential fields on the inbound body\n\n9. Tests\n - Pure walk tests on a fake adapter, plus two HTTP stubs (WireMock, MockWebServer, or Testcontainers) and no live cloud account\n - Cover: fallback order on 503 then 200; 400 with the switch off leaves the second stub at zero requests; 400 with the switch on reaches the second target; a final 422 after a 503 stays 422; 429 then 400 returns 400; 400 with the switch on then 503 returns 502 listing both attempts; both timeouts return 502 listing both attempts; 401 then 403 returns 502 listing both auth statuses; both 503s return 502 listing both attempts; tools request skips a target that lacks tools; stream reset before the first token uses the second provider; two flushed tokens then a reset emit stream_interrupted and do not call the second provider; one pass means one request per stub; correlation id matches across attempts\n - Assert the problem body does not contain the prompt fixture\n\n10. Rollout\n - Ship the table with the current single target first if {{GATEWAY_NAME}} is already in production, and log the target the walk would pick\n - Enable the second target on one alias. Watch fallback rate, 4xx versus 5xx, and auth_config for a bounded window before widening\n - Rollback is removing the spare target from the alias, or disabling it, without a code revert\n - Call out any alias where {{BAD_REQUEST_FALLBACK}} is on, and why a 400 from that model is model-specific\n\nConstraints:\n- Minimal diff. Match the module's config style and test style. No new resilience library when one is already on the classpath.\n- No secrets, live endpoints, or key-bearing URLs in git or in test fixtures.\n- Do not weaken deadlines to make tests pass. Drive timeouts with the stub.\n- Leave price tables, inbound rate limits, Idempotency-Key storage, HTTP pool sizing, and prompt-injection filters to their existing skills. Wire hooks where this walk depends on them.\n- Copilot, Claude Code, and Codex: concrete diffs, the inventory table, and a checklist. Do not paste this prompt into AGENTS.md.\n\nOutput format:\n1. Inventory table and flagged call sites\n2. Config and code diff (routing table, router walk, adapter interface, status mapping, streaming commit, metrics)\n3. Tests added and what each asserts\n4. Rollout notes: enable order, dashboards, rollback, and any fallbackOnBadRequest exceptions\n\nStart from {{APP_PATH}}. Classify failures, surface a final 400, 413, or 422, and return 502 listing every attempt when the final target fails another way, before adding a second provider or a weighted policy." + }, + { + "id": "add-llm-pii-redaction", + "title": "Add PII and Secret Redaction to an LLM Gateway", + "whenToUse": "Add or repair redaction on a Java/Spring LLM gateway so personal data and secrets are removed before Azure OpenAI, Bedrock, OpenRouter, or any OpenAI-compatible call, and before logs, traces, caches, and eval fixtures — with a per-route mode, a fail-closed default, streaming hold-back, and a test that the stub provider and the log appender never see raw values.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "pii", + "redaction", + "security", + "spring-boot", + "gateway" + ], + "formatNotes": "All three. Point the tool at the gateway module, the provider client, logging config, cache keys, and eval export. Copilot: llm-pii-redaction path instructions plus those files in agent mode (or save the prompt as `.github/prompts/add-llm-pii-redaction.prompt.md`). Claude Code: the llm-pii-redaction-guardrails skill plus the filter, vault, and test files. Codex: the AGENTS.md LLM redaction section, then the diff. Leave prompt-injection trust maps, log JSON format, routing fallback, eval gate wiring, and price tables to their own skills.", + "body": "Task: Add PII and secret redaction to the LLM gateway I specify. Build one redacted view of each prompt and use it for the provider call, for logs, metrics, and traces, and for prompt caches and eval fixtures. Choose mask, typed placeholder, or reversible tokens per route. Default to fail-closed when a detector errors. Hold back stream tails so a value split across chunks is still caught. Finish with a minimal diff, fixtures, and a test that a stub provider and the log appender never see the raw values.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Gateway name and module: {{GATEWAY_NAME}} at {{APP_PATH}}\n- Stack: {{STACK}} (Spring Boot version; Spring AI | LangChain4j | plain HTTP client | mixed)\n- Providers: {{PROVIDERS}} (Azure OpenAI, Amazon Bedrock, OpenRouter, other OpenAI-compatible base URLs)\n- Routes: {{ROUTES}} (route or alias → data class, and whether the response may restore values to the caller)\n- Mode per route: {{MODE}} (mask | typed-placeholder | reversible-token)\n- Policy per route: {{POLICY}} (fail-closed is the default; name any fail-open route and why)\n- Recognizer: {{NER}} (none | azure-language | in-process analyzer) and its timeout\n- National ID schemes in scope: {{NATIONAL_IDS}} (list, or \"none\")\n- Streaming: {{STREAMING}} (yes | no | mixed)\n- Cache and eval export: {{SINKS}} (prompt cache path, eval fixture path, or \"none\")\n- Latency budget: {{LATENCY_BUDGET}} (p95 for the deterministic stage, recognizer timeout, alias deadline)\n- Observability: {{OBSERVABILITY}} (Micrometer | OpenTelemetry | both; log appender used in tests)\n- Pain: {{PAIN}} (raw prompts in logs | PII at the provider | split tokens on the stream | no tests | none)\n\nRequirements:\n\n1. Inventory\n - Read {{APP_PATH}}: the provider client, filters that log bodies, HTTP wiretaps, prompt-cache keys, eval exporters, and any existing regex or PII call\n - Table: call site | sees raw prompt? | sees raw completion? | sink (provider, log, trace, cache, fixture)\n - Flag a system-prompt sentence used as the only protection, a shared map of raw values, and a detector that treats every 16-digit run as a card\n\n2. One redacted view, in this order\n - After authn, before the provider HTTP call, redact user messages, tool results, and retrieved text. Leave role names, tool names, JSON keys, and document delimiters in place. Scan system text for interpolated secrets\n - Pass that same view to every provider and every fallback. Do not redact differently per provider\n - Logs, span attributes, the prompt cache, and eval exports under {{SINKS}} record this view. A debug flag does not open a second copy of the raw body\n - On {{MODE}} `reversible-token`, restore placeholders only on the response bytes for this caller. Logs and the cache keep placeholders. Then mask any card, email, or secret the model introduced that this request did not mint\n\n3. Detectors\n - Deterministic, in process, always on for the entities you enable: email (including plus-tags), phone (leading + or a known trunk pattern, with a digit-length check), payment cards (13–19 digits, spaces and dashes ignored, Luhn required), API-key assignment shapes, and JWTs (three base64url segments, header JSON with `alg`)\n - {{NATIONAL_IDS}}: implement only named schemes, with that scheme's published length and checksum. If the list is \"none\", leave national-id detection off\n - A checksum failure is not a finding. Secrets and JWTs are never reversible, even when the route mode restores emails or phones, and even when a longer personal-data span contains or overlaps them\n - {{NER}}: optional. Confidence floor per entity type, exclude `person_type` and any other category that marks `user` / `assistant` or dates the product needs. Map spans back onto the original string. Do not concatenate the service's own redacted chunks\n - Allowlist: reviewed literals only (support address, product names, the placeholder syntax). A span that matches the allowlist is kept and counted as `allowed`, and that match removes only that span. No test card numbers in production config\n - Overlap, on the original string, start inclusive and end exclusive. Redact the union of the surviving spans, then split that union at each span boundary. A slice that intersects an API key, a JWT, or another secret stays non-restorable: mask it, or use a placeholder that is not written to the vault and is not restored. That holds when a longer reversible NER or other personal-data span contains the secret, and when the spans only partly overlap, so the shared bytes and each exclusive tail are covered and the secret is not restored with the personal-data slices. Any other slice follows {{MODE}}. If at least one deterministic finding covers a restorable slice, label it with the deterministic finding that starts earlier. If those starts are equal, use the deterministic finding that ends later. An NER finding does not supply the label for that slice. If every finding on the slice is from the recognizer, use the one that starts earlier, and if those starts are equal, the one that ends later. If a slice intersects more than one secret, label it with the secret finding that starts earlier. If those starts are equal, use the secret finding that ends later. The slice stays non-restorable. Merge adjacent slices only when type, action, and restorability match. Length and confidence are not inputs to coverage or restorability. Replace the slices from the end of the string. Re-parse JSON and fail the request if the body no longer parses. This combines spans the enabled detectors returned. It does not mean detection is complete\n\n4. Modes and the vault\n - `mask`: irreversible character mask (last-4 only where the scheme says that remainder is safe)\n - `typed-placeholder`: `[TYPE_n]`, stable for the same value within the request, discarded after the response with no restore\n - `reversible-token`: the same placeholders plus a request-scoped in-memory vault, cleared in `finally`, not stored in Redis, logs, or the cache. Restore only tokens this request minted. An API key, JWT, or other secret is never a vault entry, and a personal-data token does not contain those bytes when the spans overlap\n - A prompt cache hit restores with this request's vault only. A placeholder this request did not mint stays a placeholder\n - Apply {{MODE}} and {{POLICY}} per route under {{ROUTES}}. `default-policy` is fail-closed\n\n5. Streaming ({{STREAMING}})\n - Redact the full request before the provider call\n - On the response, buffer and flush only the prefix that cannot still grow into an entity or a placeholder. Resolve overlaps on that buffer before flushing, and leave any byte still covered by a span in the buffer unflushed. Hold the suffix. Cap the secret scan (document the cap) and do not split a UTF-8 sequence\n - At stream end, flush the tail through the same detector. On error or cancel, drop the buffer and the vault without logging it\n - The log appender sees redacted text. The caller sees restored text only in reversible mode\n\n6. Fail-closed and fail-open\n - Recognizer throw, timeout, or unparseable result on a fail-closed route: do not call the provider, do not cache the raw body, return problem+json 503 `redaction_unavailable`. Zero findings is success\n - On a fail-open route named in {{POLICY}}: send the deterministically redacted view, increment `detector_error`, alert. If the in-process scanner throws, fail closed anyway\n - Subtract the recognizer timeout from the alias deadline before routing starts, inside {{LATENCY_BUDGET}}\n\n7. Counters and logs ({{OBSERVABILITY}})\n - `llm_redaction_entities_total` labels: `route`, `entity_type`, `action` (`masked`, `placeholder`, `tokenized`, `allowed`, `detector_error`)\n - `llm_redaction_seconds` labels: `route`, `outcome` (`ok`, `failed_closed`, `failed_open`)\n - No label, span attribute, log field, or usage-store column carries a value or a hash of a value\n - Remove or gate any wiretap or `log.debug` of the raw provider body\n\n8. Tests\n - Golden fixtures: spans plus the expected string for each mode. Synthetic values only\n - False positives that stay: UUID, git SHA, semver, epoch millis, order number, the words `user` and `assistant`, the token `[EMAIL_1]`, a 16-digit run that fails Luhn\n - False negatives that redact: plus-tag email, dashed PAN that passes Luhn, country-code phone, JWT without a `Bearer` prefix\n - Overlap, reversible mode. Stub the recognizer with a personal-data span and let the deterministic layer find the API key or JWT. Nested: the personal-data span strictly contains the secret. In the stub request and the log appender, the secret is absent and the personal-data text outside the secret is absent. Partial: the spans overlap and each extends past the other. In the stub request and the log appender, the secret, the shared bytes, and the personal-data-only tail are absent. In both, the caller response and the vault do not contain the secret, and restoring personal-data bytes outside the secret does not restore it. These fixtures do not show that every secret was detected\n - Leak test: stub provider plus the log appender from {{OBSERVABILITY}}. The recorded request and every log event omit the fixture email, the PAN (spaced and compact), and the JWT. Counters move. On a reversible route the stub echoes `[EMAIL_1]`, the caller response contains the email, the appender contains the placeholder and not the email, and the vault is empty afterward\n - Streaming: PAN split across two stub deltas never appears in the client bytes or the appender. Placeholder split across two deltas restores once on a reversible route\n - Fail-closed recognizer failure: zero stub requests, HTTP 503 `redaction_unavailable`, raw fixture absent from logs. Fail-open: deterministic hits still redacted, one stub request, `detector_error` incremented\n - Two requests that share a placeholder shape do not restore each other's values from the cache\n\n9. Verification\n - Inventory table from step 1, updated to show each sink now sees the redacted view\n - Route table: route, mode, policy, recognizer on or off, entities enabled\n - Latency note: deterministic p95 target, recognizer timeout, what was subtracted from the alias deadline\n - Tests added and what each asserts\n\nConstraints:\n- Minimal diff. Match the module's config style and test style. No new HTTP client and no new resilience library.\n- No real customer text, live endpoints, or secrets in git or in fixtures. Use synthetic values and published test PANs.\n- Do not weaken the provider call timeout to make tests pass. Drive recognizer failures with a fake client.\n- Leave prompt-injection trust maps, JSON log encoder setup, routing and fallback policy, eval CI wiring, and price tables to their existing skills. Call those hooks where this stage sits next to them; do not reimplement them.\n- Copilot, Claude Code, and Codex: concrete diffs, the inventory table, and a checklist. Do not paste this prompt into AGENTS.md.\n\nOutput format:\n1. Inventory table and the sinks that still see raw text\n2. Config and code diff (detectors, modes, vault, stream hold-back, policy, counters)\n3. Tests added and what each asserts, including the leak test\n4. Route table and the latency budget\n\nStart from {{APP_PATH}}. Wire the redacted view through the provider and the log appender, and land the leak test, before enabling a recognizer or a fail-open route.\n" } ] } diff --git a/skillbook/skills.json b/skillbook/skills.json index 8c6aad8..9224311 100644 --- a/skillbook/skills.json +++ b/skillbook/skills.json @@ -1040,6 +1040,26 @@ ], "formatNotes": "All three. Copilot: path-scoped `.github/instructions/llm-gateway-routing.instructions.md` with applyTo for `**/llm/gateway/**`, `**/*Router*.java`, `**/*Gateway*.java`, `**/*Routing*.java`, `**/*Adapter*.java`, `**/*Provider*.java`, and `**/application*.{yml,yaml,properties}`. Optional on-demand procedure at `.github/skills/llm-gateway-routing-fallback-guardrails/SKILL.md`. Claude Code: `.claude/skills/llm-gateway-routing-fallback-guardrails/SKILL.md` plus a CLAUDE.md fact that aliases walk ordered targets, a caller 4xx stops the walk unless that target sets fallbackOnBadRequest, and a committed stream never changes provider. Codex: a short \"LLM gateway routing\" brief in root `AGENTS.md` (the rule, the table path, the two-stub test command) while the procedure stays in SKILL.md. Pair with llm-cost-latency-budget-guardrails for pricing each attempt, llm-app-prompt-injection-guardrails for untrusted content and upstream redaction, llm-eval-ci-regression-gate, llm-eval-agent-workflow, and rag-retrieval-eval-guardrails for eval, spring-boot-http-client-guardrails for client timeouts and pools, spring-boot-resilience4j for the breaker primitive, api-rate-limiting-guardrails for inbound quotas, and api-idempotency-key-guardrails for the public Idempotency-Key. This skill owns outbound routing, fallback classes, the final status (surface a final 400, 413, or 422; otherwise 502 listing every attempt), streaming commit, and adapter boundaries.", "body": "# Skill: LLM Gateway Routing and Fallback Guardrails\n\n## Goal\n\nMake multi-provider chat, completions, and similar calls fail over on purpose. Callers name an alias such as `chat-default`. The gateway resolves it to ordered targets (provider, model or deployment, region), walks them under the caller deadline, and reports an honest status when the walk fails. Timeouts, 429s, 5xx, and connection errors move to the next healthy target. A 400, 413, or 422 stops the walk unless that target sets fallbackOnBadRequest. After the first streamed token, the gateway stays on that target. When the final target returns 400, 413, or 422, the caller sees that status. When the final target fails another way, the caller sees 502 and the body lists every attempt.\n\nThis skill owns the routing table, primary versus fallback selection, which failure classes fall through, the per-target `fallbackOnBadRequest` switch, streaming commit rules, the adapter boundary, and the tests that prove multi-target behavior. Pricing, token caps, and reserve-then-reconcile budgets stay in **llm-cost-latency-budget-guardrails**. Prompt-injection boundaries stay in **llm-app-prompt-injection-guardrails**. Eval gates stay in **llm-eval-ci-regression-gate**, **llm-eval-agent-workflow**, and **rag-retrieval-eval-guardrails**. Breaker primitives stay in **spring-boot-resilience4j**. Client timeouts, pools, and TLS stay in **spring-boot-http-client-guardrails**. Inbound quotas stay in **api-rate-limiting-guardrails**. An `Idempotency-Key` on the public POST stays in **api-idempotency-key-guardrails**.\n\n## When to apply\n\n- A second provider or region for chat, completions, embeddings, or a similar method (Azure OpenAI, Amazon Bedrock, OpenRouter, Together, Modal, or another OpenAI-compatible backend)\n- Replacing a hard-coded model name with an alias the gateway owns\n- An outage where one region's 500s took the product down, or a rejected request came back as 502 and clients retried for hours\n- A content-filter or schema 400 that a product owner wants to replay on another model\n- A stream that mixes two models' tokens, or hangs after the first provider dies mid-response\n- Review of router, gateway, or adapter code before a new alias is enabled\n\n## The rule\n\nThe caller names an alias. The gateway walks that alias's eligible targets once, and starts the next attempt only when the failure is retryable and no response bytes have been committed. A 400, 413, or 422 stops the walk unless the target that produced it has `fallbackOnBadRequest: true`. The switch defaults to false. When the final target returns 400, 413, or 422, surface that status. When the final target fails another way — a 5xx after an earlier 400, an auth failure, a timeout, a 429, or a connection error — return 502 and list every attempt. Do not answer that mixed walk with 503, timeouts with a blanket 504, or auth exhaustion with 503.\n\n## Routing model\n\nAn alias is the only model name callers and config are allowed to use. Provider model ids and Azure deployment names live in the routing table.\n\n| Field | Meaning |\n|---|---|\n| `id` | Stable name in metrics and logs (`azure-east-chat`, `bedrock-backup`) |\n| `provider` | Adapter key: `azure-openai`, `bedrock`, `openrouter`, `together`, `modal`, or `openai-compatible` |\n| `model` | Provider model id, or an Azure deployment name. The adapter knows which the wire protocol wants |\n| `region` | Region or endpoint name. A URL with a credential in it is a startup error |\n| `secretRef` | Env var or Key Vault secret. The table stores the reference, never the value |\n| `priority` | Lower runs first. Also the fallback order after a weighted primary is chosen |\n| `weight` | Used only when the alias strategy is `weighted` |\n| `fallbackOnBadRequest` | Default `false`. Applies to this target only |\n| `capabilities` | Subset of `stream`, `tools`, `json_schema`, `vision`, `embeddings`. Missing capability skips the target before HTTP |\n| `minAttemptBudget` | Shortest remaining deadline still worth starting (connect plus a useful read) |\n\n**Priority.** Sort eligible targets by `priority`. First is primary; the rest fall back in that order.\n\n**Weighted.** Pick the primary by weight, then fall back in priority order. Selection is a pure function of the alias, the eligible set, and a stable key (request id or the sticky key), so a test can pin it and a caller retry prefers the same primary. The walk does not re-roll after the first failure.\n\n**Sticky.** `sticky: none` (the default) resolves health on every call — the right mode for stateless completions and embeddings. `sticky: session` pins a conversation id to the target that last succeeded for that key, for a documented window (the conversation lifetime, or about 30 minutes idle). Failures are not pinned. Sticky does not override health, residency, or capabilities; an ineligible pin is skipped and a later success updates it.\n\nEligible means enabled, breaker closed (or this call is the half-open probe), capabilities cover the request, region is on the alias residency allowlist, and remaining deadline is at least `minAttemptBudget`. Attempt cap equals that list. Do not revisit a target already tried for this request.\n\nThe adapter's HTTP client does not retry this POST. The walk is the retry; stacking both multiplies paid calls and burns the deadline. Apply an `Idempotency-Key`, when the public API has one, before the walk so a caller retry that already streamed tokens replays the stored outcome.\n\n```yaml\nllm:\n gateway:\n aliases:\n chat-default:\n strategy: priority # priority | weighted\n sticky: none # none | session\n deadline: 10s\n residency: [eastus, us-east-1]\n targets:\n - id: azure-east-chat\n provider: azure-openai\n deployment: chat-primary\n region: eastus\n secretRef: kv:llm-azure-east\n priority: 1\n fallbackOnBadRequest: false\n minAttemptBudget: 800ms\n capabilities: [stream, tools, json_schema]\n - id: bedrock-backup\n provider: bedrock\n model: chat-backup\n region: us-east-1\n secretRef: kv:llm-bedrock\n priority: 2\n fallbackOnBadRequest: false\n minAttemptBudget: 800ms\n capabilities: [stream, tools]\n```\n\nEmbeddings and similar methods share this table, these failure classes, and the final-status function. They usually use `sticky: none` and have no stream commit point. Give each method a small adapter method (`complete`, `stream`, `embed`) and one shared policy. A second copy of the status rules will drift.\n\n## Failure classes\n\nAdapters map vendor payloads into these classes. The router switches on the class, not on a provider error string.\n\n| Class | Typical signals | Next target? | Affects health? |\n|---|---|---|---|\n| Timeout | Connect timeout, read timeout, deadline hit during the attempt | Yes | Yes |\n| Rate limit | HTTP 429, or a slow-down code mapped to 429 | Yes, when `Retry-After` does not fit the remaining deadline. On the last target, wait once if the delay fits, then stop | Short cooldown only. A 429 does not open a long breaker |\n| Upstream 5xx | 500, 502, 503, 504 | Yes | Yes |\n| Connection | Reset, DNS, TLS handshake, empty response | Yes | Yes |\n| Client rejection | 400, 413, 422, including a content-filter or schema failure mapped onto those statuses | Only when this target's `fallbackOnBadRequest` is true | No. A rejected request is not an unhealthy model |\n| Auth or config | 401, 403, or a 404 that means the deployment or model id is missing | Yes. Another target may be configured. This is not the bad-request switch | Yes. Alert, and keep the target out until the credential or deployment name is fixed |\n| Caller cancelled | Client closed the connection | No. Cancel the upstream and stop | No |\n\nTurn `fallbackOnBadRequest` on only when a 400 from that model is model-specific (unsupported parameter, stricter content filter) and the next target may see the same canonical request. Leave it false for an oversized body, a bad schema, or a malformed message list: those fail on every provider, and replaying them spends budget while hiding a client bug. The flag does not retry the same target and does not apply to 401 or 403.\n\nSkip a target that lacks tools, JSON schema, or vision with reason `capability_skip` before any HTTP call. If that empties the list, return 422 `no_capable_target` with an empty attempt list — the gateway's status, because no upstream saw the call.\n\nCount only the health column toward the breaker the service already runs. A burst of 400s leaves it closed. An open breaker skips the target with reason `unhealthy`, except the half-open probe.\n\n## When every target fails\n\n`fallbackOnBadRequest` defaults to false and covers only 400, 413, and 422. The caller status depends on what the final target did, not on passing through whichever status arrived last.\n\n1. No attempt was made. Return 422 `no_capable_target` when capabilities emptied the list. Return 503 `no_eligible_target` when every target was unhealthy, disabled, or out of deadline. Both responses carry an empty attempt list. Use them only when nothing was called: a 422 the final target actually returned is that status, not `no_capable_target`, and a target that was called and then failed another way is 502, not `no_eligible_target`.\n2. The final target returned 400, 413, or 422. Surface that status. Do not wrap it in 502. An earlier 503 followed by a final 422 is still 422.\n3. The final target failed another way: timeout, 429, upstream 5xx (500, 502, 503, or 504), connection error, or auth or config (401, 403, or a deployment 404). Return 502. The body lists every attempt in order, including an earlier 400 that fell through and the last 503, each auth status, and each timeout. Record `Retry-After` on a 429 attempt when the upstream sent one; the caller status is still 502. Do not return the last 503, a blanket 504, or 503 for auth exhaustion. A caller-facing 401 would blame the caller for the gateway's provider key or deployment name, so those statuses stay inside the attempt list.\n\nThe 502 body is problem+json: stable code `upstream_failed`, the alias, and `attempts` (target id, failure class, and HTTP status when the target returned one). A surfaced 400, 413, or 422 uses the client-rejection code and that HTTP status. Do not copy the upstream error document. Those documents often echo the prompt.\n\n## Streaming\n\nThe commit point is the first model-output byte flushed to the caller. Flush nothing before that token, so a connect or wait failure can still fall through. Skip early heartbeats; a flushed heartbeat is a commit.\n\nBefore the commit point, a retryable failure starts the next target, and the caller sees one stream. Do not buffer a full generation to preserve a later fallback.\n\nAfter the commit point the status is already 200 and the caller holds one model's tokens. Cancel the upstream. Emit one terminal error a client can tell from a normal stop — for SSE, `event: error` with `{ \"code\": \"stream_interrupted\", \"alias\": \"\", \"attempt\": }` — then close. Do not open the next provider or append another model's tokens. The error frame is how a closed 200 says the answer stopped early.\n\nBuffer non-streaming calls until the attempt settles, so fallback and final-status rules still apply. If the caller disconnects, cancel the upstream and stop the walk.\n\n## Adapter boundary\n\nThe router depends on one interface. Each provider is an implementation. The router does not import a vendor SDK and does not branch on provider name.\n\n```text\ninterface LlmProviderAdapter {\n String provider();\n Capabilities capabilities(Target target);\n AttemptResult complete(CanonicalRequest request, Target target, Deadline deadline);\n void stream(CanonicalRequest request, Target target, Deadline deadline, StreamSink sink);\n}\n```\n\n`AttemptResult` carries the failure class, HTTP status when one exists, an allowlisted provider code, token counts when usage was reported, and latency. It carries neither the raw prompt nor the raw error document.\n\nThe canonical request is messages, tools, response format, sampling parameters, max output tokens, and the stream flag. Azure OpenAI, OpenRouter, Together, Modal, and other OpenAI-compatible backends share one HTTP adapter. Config supplies base URL, header style (`api-key` or `Authorization: Bearer`), API version, and deployment-versus-model path. Amazon Bedrock is its own adapter: signing and the Converse or invoke body are a different schema. A new provider is a new class plus a table row.\n\nResolve `secretRef` from the environment or Key Vault (**azure-key-vault-app-integration**). Send credentials in headers. Reject a base URL that already contains a key.\n\nConnect and read timeouts fit the deadline passed into the adapter, using the client builder, pool, and TLS rules from **spring-boot-http-client-guardrails**. Skip a target with reason `deadline` when remaining budget is below `minAttemptBudget`. The read timeout is that remainder, not the happy-path SLO.\n\n## Budgets and latency\n\nThe alias deadline is the hard stop behind the latency SLO (for example a 10s deadline behind an 8s p95). Subtract time already spent before each attempt. Record attempt latency, and time to first token on streams, as histograms with the labels below. Hand every attempt — success, transport failure, and a 400 taken because the switch is on — to the cost and budget hook before the next target starts, so a walk cannot outrun a budget checked only on the first call. This module does not own the price table, max-token caps, or the ledger.\n\n## Observability\n\nEach attempt is a child span of the gateway request. Every attempt log carries the inbound correlation id.\n\nRecord alias, target id, provider, model or deployment, region, attempt index, failure class, HTTP status, reason, latency, whether the stream had committed, and token counts when the provider sent them. Reasons are a fixed set: `timeout`, `rate_limit`, `upstream_5xx`, `connection`, `bad_request`, `auth_config`, `capability_skip`, `unhealthy`, `deadline`, `cancelled`, `stream_committed`.\n\n- `llm_gateway_attempt_seconds` — histogram. Labels: `alias`, `provider`, `outcome`.\n- `llm_gateway_attempts_total` — counter. Labels: `alias`, `provider`, `model`, `outcome`, `failure_class`.\n- `llm_gateway_fallback_total` — counter. Labels: `alias`, `reason`.\n- `llm_gateway_response_total` — counter. Labels: `alias`, `status_class` (`2xx`, `4xx`, `5xx`).\n\n`model` is the configured id. Request ids, user ids, prompt hashes, and provider messages are not labels. Alert on a rising fallback rate, any `auth_config`, and `no_eligible_target`.\n\n## Safety\n\nDefault logs and traces hold routing metadata only. A debug switch that prints prompts or completions stays off in shared environments. Redaction of secrets and personal data in the prompt is upstream of this router: **llm-app-prompt-injection-guardrails** for untrusted content and output sinks, **spring-boot-structured-logging-guardrails** for the log pipeline. Body logging is not a fallback debugging tool.\n\nTargets are an allowlist. The inbound body cannot name a provider URL, a region, or a credential. Residency is a hard filter. `fallbackOnBadRequest` can move a request one provider's content filter rejected onto a provider with a different filter. Write that decision next to the flag. The default stays off.\n\n## Tests\n\nUse a pure walk test with a fake adapter, plus two HTTP stubs (WireMock, MockWebServer, or a Testcontainers mock). No cloud account.\n\n| Case | Setup | What passes |\n|---|---|---|\n| Happy primary | Primary 200 | Secondary unused. Caller 200. One attempt |\n| Fallback on 503 | Primary 503, secondary 200 | Caller 200. Order primary then secondary. Reason `upstream_5xx` |\n| No fallback on 400 | Primary 400, switch false | Caller 400. Secondary unused. Breaker failure count unchanged |\n| Switch on | Primary 400, switch true, secondary 200 | Caller 200. Both called. Reason `bad_request` |\n| Last 4xx kept | Primary 503, secondary 422, switch false | Caller 422, client-rejection code |\n| 429 then 400 | Primary 429, secondary 400, switch false | Caller 400 |\n| Switch then 5xx | Primary 400 switch on, secondary 503 | Caller 502. Body lists the 400 and the 503 |\n| All timeouts | Both time out | Caller 502. Body lists both timeouts. Both called |\n| Auth exhausted | Primary 401, secondary 403 | Caller 502. Body lists both auth statuses |\n| Capability skip | Needs tools; primary has none; secondary 200 | Primary stub unused. Reason `capability_skip` |\n| None capable | No target supports tools | Caller 422 `no_capable_target`. Both stubs unused |\n| Stream before commit | Primary resets before any token | Caller sees only the secondary stream |\n| Stream after commit | Primary flushes two tokens, then resets | Those tokens plus `stream_interrupted`. Secondary unused |\n| Correlation | Two attempts | Same correlation id on both spans and logs |\n| Sticky | Two calls, same session, primary healthy | Same target id |\n| Deadline skip | Budget below `minAttemptBudget` | Target not called. Reason `deadline` |\n| Single pass | Both return 503 | Caller 502 listing both 503s. Each stub received one request |\n\nAssert the problem body does not contain the prompt fixture.\n\n## Workflow\n\n1. Inventory aliases, call sites that still pass a raw model name, providers, and any retry already on the HTTP client or a breaker.\n2. Add the canonical request, `AttemptResult`, and `LlmProviderAdapter`. Wrap today's client as a one-target adapter. A final 400, 413, or 422 stays that status; any other final failure is 502 listing the attempt.\n3. Move model choice into the table. If the service is live, ship one target per alias and log the id the walk would have picked.\n4. Add adapter classification and the walk. Leave every `fallbackOnBadRequest` false.\n5. Add the commit rule and `stream_interrupted` before a second streaming target is enabled.\n6. Add per-attempt metrics and the shared correlation id. Confirm a 400 does not move the breaker.\n7. Add the decision tests and the two-stub tests.\n8. Add the spare target on one alias. Watch fallback rate, status class, and auth failures for a bounded window.\n9. Turn the bad-request switch on only for a named target, with the reason in the config change.\n\n## Tool-specific notes\n\n- **GitHub Copilot:** path-scoped `.github/instructions/llm-gateway-routing.instructions.md` with `applyTo` for `**/llm/gateway/**`, `**/*Router*.java`, `**/*Gateway*.java`, `**/*Routing*.java`, `**/*Adapter*.java`, `**/*Provider*.java`, and `**/application*.{yml,yaml,properties}`. Put The rule, the failure-class table, and the final-status steps at the top. Keep the longer procedure in an optional `.github/skills/llm-gateway-routing-fallback-guardrails/SKILL.md`. Ask for the alias inventory before editing the walk.\n- **Claude Code:** `.claude/skills/llm-gateway-routing-fallback-guardrails/SKILL.md` holds the procedure. Add one `CLAUDE.md` fact: \"The LLM gateway resolves aliases to ordered targets, falls through only on retryable failures before any streamed token, leaves fallbackOnBadRequest off unless a target opts in, surfaces a final 400, 413, or 422, and returns 502 listing every attempt when the final target fails another way.\"\n- **Codex:** root `AGENTS.md` stays a short brief — The rule, the routing-table path, and the two-stub test command. The procedure lives in `SKILL.md`, or in a nested `AGENTS.md` beside the gateway module if the repo already uses nested briefs. Do not paste this whole skill into `AGENTS.md`.\n\n## Anti-patterns\n\n- Callers pass provider model ids and the gateway forwards them\n- A final 400, 413, or 422 is rewritten as 502, so the client retries a rejected request\n- An earlier 400 followed by a last 503 is returned as 503, or as that last upstream status, instead of 502 listing both attempts\n- Timeouts are answered with a blanket 504, or auth exhaustion with 503, instead of 502 listing every attempt\n- `fallbackOnBadRequest: true` on every target, so every malformed body is replayed everywhere\n- HTTP 400 and 422 increment the breaker, and one bad client opens the model for everyone\n- The HTTP client retries the POST and the gateway also walks fallbacks\n- The next provider opens after tokens were already flushed\n- A truncated stream is stored as success because the status was 200\n- One class full of `if (provider.equals(\"bedrock\"))` owns both routing and vendor payloads\n- API keys, or URLs that contain keys, sit in the routing YAML\n- Attempt logs or span attributes contain prompt or completion text\n- A health check sends a real user prompt to keep a target warm\n- Fallback picks a region outside the alias's residency allowlist\n- The target list cycles until the deadline expires\n\n## Checklist\n\n- [ ] Every caller-facing model name is an alias; each target names provider, model or deployment, region, and a secret reference\n- [ ] Strategy and sticky mode are set per alias, and the primary is not re-rolled during the walk\n- [ ] The walk is one pass, and the HTTP client does not also retry that attempt\n- [ ] Timeouts, 429, 5xx, and connection errors fall through. 400, 413, and 422 fall through only when that target's switch is true\n- [ ] 401, 403, and deployment-not-found continue the walk and mark the target unhealthy. When one of them is the final failure, the caller gets 502 listing every attempt, not 503\n- [ ] Client 4xx does not open the breaker\n- [ ] A final 400, 413, or 422 is surfaced as that status. Any other final failure is 502 whose body lists every attempt, including an earlier 400 then a last 503, auth failures, and timeouts\n- [ ] Streams fall through only before the first flushed token. A mid-stream failure cancels the upstream and emits `stream_interrupted`\n- [ ] Each provider implements `LlmProviderAdapter`. Bedrock is not forced through the OpenAI-compatible adapter\n- [ ] Attempt spans share a correlation id and record provider, model, status, latency, tokens when known, and a reason from the fixed set\n- [ ] Default logs and traces omit raw prompts and completions\n- [ ] Two-stub tests cover fallback order, a 400 with the switch off, a preserved final 422, a 400 then 503 returned as 502 listing both attempts, timeouts and auth exhaustion returned as 502, and no mid-stream switch\n- [ ] Each attempt is visible to the cost and budget hook. This module does not own prices or inbound quotas\n\n## Copy-ready SKILL.md stub\n\n```markdown\n---\nname: llm-gateway-routing-fallback-guardrails\ndescription: Use when a Java or Spring LLM gateway routes chat or completions across Azure OpenAI, Bedrock, OpenRouter, Together, Modal, or other backends — alias to ordered targets, health-aware priority or weighted selection, fallback on timeout 429 5xx and connection errors, per-target fallbackOnBadRequest defaulting off for 400/413/422, a final 400/413/422 surfaced as that status and 502 listing every attempt when the final target fails another way, no mid-stream provider switch, one adapter per provider, per-attempt metrics, and two-stub tests.\n---\n\nFollow LLM Gateway Routing and Fallback Guardrails: callers name an alias, and the gateway walks that alias's eligible targets once. Fall through on timeout, 429, 5xx, connection errors, and auth or deployment-config failures. A 400, 413, or 422 stops the walk unless that target sets fallbackOnBadRequest, which defaults to false. Do not switch providers after the first streamed token; cancel and emit stream_interrupted. When the final target returns 400, 413, or 422, surface that status. When it fails another way, including an earlier 400 then a last 503, an auth failure, or a timeout, return 502 and list every attempt. Do not use 503 for that mixed walk or for auth exhaustion, and do not use a blanket 504 for timeouts. Keep provider SDKs behind adapters, keep secrets out of the routing table, log attempt metadata under one correlation id, and prove the walk with two provider stubs.\n```" + }, + { + "id": "llm-pii-redaction-guardrails", + "title": "LLM PII and Secret Redaction Guardrails", + "whenToUse": "Adding or reviewing redaction in a Java/Spring Boot LLM gateway, or in a service that calls Azure OpenAI, Amazon Bedrock, OpenRouter, or any OpenAI-compatible provider — so personal data and secrets are removed before the provider call, before logs, metrics, and traces, and before prompt caches and eval fixtures, with an explicit per-route fail-closed or fail-open policy.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "pii", + "redaction", + "security", + "spring-boot", + "gateway" + ], + "formatNotes": "All three. Copilot: path-scoped `.github/instructions/llm-pii-redaction.instructions.md` with applyTo for `**/llm/**`, `**/*Redact*.java`, `**/*Pii*.java`, `**/*Vault*.java`, `**/gateway/**`, and `**/application*.{yml,yaml,properties}`. Optional on-demand procedure at `.github/skills/llm-pii-redaction-guardrails/SKILL.md`. Claude Code: `.claude/skills/llm-pii-redaction-guardrails/SKILL.md` plus a CLAUDE.md fact that provider calls, logs, traces, caches, and eval fixtures see only the redacted view, and the default policy is fail-closed. Codex: a short \"LLM redaction\" brief in root `AGENTS.md` (the pipeline order, the route policy, the leak-test command) while the procedure stays in SKILL.md. Pair with llm-app-prompt-injection-guardrails for trust boundaries and output sinks, spring-boot-structured-logging-guardrails for log shape, llm-gateway-routing-fallback-guardrails for the call that happens after redaction, llm-eval-ci-regression-gate for golden-set hygiene, and llm-cost-latency-budget-guardrails for the latency stage. This skill owns detection, redaction modes, the per-request vault, streaming hold-back, and the leak tests.", + "body": "# Skill: LLM PII and Secret Redaction Guardrails\n\n## Goal\n\nKeep personal data and secrets out of places they do not belong when a Java or Spring Boot service calls a model. The gateway (or the app in front of Azure OpenAI, Amazon Bedrock, OpenRouter, or any OpenAI-compatible provider) produces one redacted view of the prompt, and that view is what the provider, the logs, the traces, the prompt cache, and the eval fixtures are allowed to see.\n\nThis skill owns where that view is built, the detector stack, the three redaction modes, streaming chunk boundaries, the per-request vault, the per-route fail-closed or fail-open policy, audit counters that carry entity types and not values, the latency budget for the stage, and the test that proves a raw value never reaches the stub provider or the log appender.\n\nTrust boundaries, tool agency, and output encoding stay in **llm-app-prompt-injection-guardrails** (that skill's output screen and \"do not log the payload\" rule are the reason this pipeline exists; they are not the pipeline). Log shape — JSON, MDC, levels — stays in **spring-boot-structured-logging-guardrails**. Alias routing, fallback, and attempt metadata stay in **llm-gateway-routing-fallback-guardrails**, which runs after this stage and must not see a raw body. Golden-set versioning and the ban on real customer rows stay in **llm-eval-ci-regression-gate**. Recording this stage's time inside gateway overhead stays in **llm-cost-latency-budget-guardrails**. Column masking inside a warehouse job stays in **data-platform-pipeline-review**.\n\nA line in the system prompt that says \"do not reveal personal data\" is not a control. Public guidance on sensitive-information disclosure (OWASP LLM Top 10, sensitive information disclosure) treats that wording as bypassable. The control is this code path.\n\n## When to apply\n\n- A new LLM gateway, or a Spring AI, LangChain4j, or plain HTTP client that forwards chat or completions\n- A route whose prompt includes free text, retrieved chunks, tool results, or file extracts\n- A review that finds prompt or completion text in logs, traces, a prompt cache, or an eval JSONL file\n- Adding a named-entity recognizer beside local patterns (a hosted PII API such as Azure AI Language, or a self-hosted analyzer in the detect-then-anonymize style)\n- A streaming endpoint where a card number, an email, or a placeholder can split across chunks\n\n## Where redaction runs\n\nBuild the redacted view once, after the caller is authenticated and before any of the following:\n\n1. **Before the provider call.** Redact user and tool messages, and any retrieved text that will be copied into the prompt. Scan system instructions and tool schemas for secrets the app interpolated; do not strip role names or tool names. Every fallback attempt from **llm-gateway-routing-fallback-guardrails** sends this same view. Do not redact a different way per provider.\n2. **Before logs, metrics, and traces.** Application logs, client wire logs, span attributes, and exception messages use this view. A debug flag does not open a second path for the raw body.\n3. **Before prompt caches and eval fixtures.** The cache key and the stored completion use the redacted text. An export that builds a golden file writes the redacted view or a synthetic case, never a customer transcript. **llm-eval-ci-regression-gate** already forbids real personal data in `evals/golden`; this stage is how that file stays clean.\n\nOn the way out, only a reversible route writes original values back, and only onto the response bytes for that caller. Logs, traces, the cache, and the next provider call keep placeholders. After restore, scan the model text again for values the model introduced (a card or a secret that was not in the request). Mask those. Do not add them to the vault.\n\n## Detection layers\n\nRun the layers below and collect spans on the original string. Combine them in Overlap resolution before replacing anything. A finding record stores type, start, end, and action. It does not store the matched text. The vault, when the mode needs one, is a separate request-scoped object.\n\n### 1. Deterministic patterns and checksums (always on)\n\nIn-process, no network. Cover the types the route turns on:\n\n- **Email.** A practical `local@domain` pattern with a dot in the domain. Plus-tags (`ada+news@example.com`) match. Do not try to encode the full address RFC in one expression.\n- **Phone.** A leading `+` or a national trunk pattern the product actually receives, plus a digit length for that pattern. A bare run of digits inside an order id or a timestamp is not a phone.\n- **Payment card.** Take a 13–19 digit candidate, ignore spaces and dashes, and accept it only when it passes the Luhn check. From the right, double every second digit, subtract 9 when a product is over 9, and accept when the sum is divisible by 10. A 16-digit run that fails Luhn is not a card.\n- **National IDs.** Only schemes this service is in scope to handle. Name each scheme in config and check the published length and checksum for that scheme. Leave the detector off until at least one scheme is named. There is no single \"world ID\" pattern.\n- **API keys and other secrets.** Known assignment shapes (`api_key=`, `Bearer ` followed by a token) and published key prefixes the team has listed. A match is a credential, not a personal-data field.\n- **JWTs.** Three base64url segments whose header decodes as JSON containing `alg`. The token is a secret even when you do not check the signature. Signature checks belong to **jwt-oauth-security-review**, not here.\n\nWhen a scheme has a checksum, a pattern hit that fails the checksum is not a finding. Secrets and JWTs are never reversible, including on a route whose mode restores other entities. A longer personal-data span that contains or overlaps one does not make it reversible. Overlap resolution keeps that classification.\n\n### 2. Optional named-entity recognizer\n\nUse this for person names, street addresses, and other free text the patterns miss. Two shapes are enough:\n\n- A hosted PII API (Azure AI Language is the usual one on Azure). It returns entity category, confidence, and a redacted string. Call it with workload identity on a private endpoint. Do not enable the service's own payload logging.\n- A self-hosted analyzer in the detect-then-anonymize style: recognizers (patterns, checksums, a model) produce spans, and a second step applies an operator (mask, replace, hash). Keep that second step inside this gateway so the operator and the vault stay in one place.\n\nRules for either shape:\n\n- Off unless the route sets `ner: true`. A confidence floor is set per entity type; start from the false-positive fixtures and record the floor you picked. A low model score does not cancel a deterministic hit, and it does not remove that hit's bytes from the union in Overlap resolution.\n- Drop categories that wreck prompts. Role labels (`user`, `assistant`, `system`) and, when the product needs them, dates. Exclude them in config, not in a prompt.\n- Honor the service's per-document size cap. Chunk with overlap, detect on the chunks, and map spans back onto the original string. Do not glue the service's redacted chunks together; that shifts offsets and can break JSON.\n- The recognizer sees raw text, so its timeout is part of the route deadline. A timeout is a detector error, handled by the route policy below.\n\n### 3. Allowlist\n\nPer route, a reviewed list of literals and patterns that must survive: the public support address, product names, JSON keys, the placeholder syntax itself.\n\n- A span that matches a detector and the allowlist is kept and counted as `allowed`. That match removes only the span that matched. A different finding that overlaps it stays, including an API key or JWT.\n- Do not allowlist \"any 16 digits\", a customer-id shape with no namespace, or a test card number in a production config. Test values live in fixtures.\n- The allowlist is data, reviewed in the same pull request as a behavior change.\n\n## Overlap resolution\n\nRun this after the three layers above and before any replacement, on the request and again on the response scan. Spans use the original string, start inclusive and end exclusive. Take the findings that remain after checksum failures are dropped and allowlist matches are removed.\n\nThe sensitive set is the union of those spans. Redact every byte in the union. Split the union at each span boundary, then label the slices:\n\n1. A slice that intersects an API key, a JWT, or another secret is non-restorable. Mask it, or replace it with a typed placeholder that is absent from the vault and is never restored. The secret keeps that classification when a longer reversible NER or other personal-data span contains it, and when the two spans only partly overlap. Those shared bytes stay out of the vault, so restoring the personal-data slices does not restore the secret. If the slice intersects more than one secret, the label is the secret finding that starts earlier. If those starts are equal, the label is the secret finding that ends later. The slice stays non-restorable either way.\n2. A slice with no secret follows the route mode. If at least one deterministic finding covers it, the label is the deterministic finding that starts earlier. If those starts are equal, the label is the deterministic finding that ends later. An NER finding does not supply the label for that slice. Bytes that only an NER span covers stay in the union and follow the route mode. If every finding on the slice is from the recognizer, the label is the one that starts earlier. If those starts are equal, the label is the one that ends later.\n3. Merge two adjacent slices only when entity type, action, and restorability match. A non-restorable slice stays its own replacement, so it is not folded into a reversible token.\n\nLength and confidence are not inputs to coverage or to restorability. The replacement units are these slices. Apply them from the end of the string so earlier offsets stay valid. On a stream, resolve the buffer before flushing a prefix, and leave any byte still covered by a span in the buffer unflushed.\n\nTwo offset examples, on the original string:\n\n- Nested. NER personal-data is `[0, 100)` and an API key is `[40, 60)`. The redacted view covers `[0, 100)`. `[40, 60)` is non-restorable. `[0, 40)` and `[60, 100)` follow the route mode.\n- Partial. An email is `[0, 20)` and an API key is `[10, 40)`. The redacted view covers `[0, 40)`. `[10, 40)` is non-restorable, so the shared email bytes are not restored. `[0, 10)` follows the route mode.\n\nThis combines spans the enabled detectors returned. A value no detector marked is unchanged. A passing overlap test does not mean the recognizer or the patterns found every personal-data value or secret.\n\n## Redaction modes\n\nOne mode per route, applied to the whole request. Pick it in config.\n\n| Mode | What the model sees | Vault | What the caller sees |\n|---|---|---|---|\n| `mask` | A character mask, or last-4 only where the scheme says that remainder is safe to show | None | The same mask |\n| `typed-placeholder` | `[EMAIL_1]`, `[PHONE_2]`. The same value in this request always gets the same token, so the model can refer to it | Built and discarded. Not used to restore | The placeholders |\n| `reversible-token` | The same placeholders | In-memory map for this request only | Originals, and only on the response path |\n\nPlaceholder shape is `[TYPE_n]`: `TYPE` from a fixed enum, `n` starting at 1 per type per request. Keep the type names short and the counter small so the longest placeholder is a known bound (well under 32 characters). The model sees the type, which is what keeps the prompt usable: \"reply to [EMAIL_1]\" still has a referent. A solid mask does not.\n\nVault rules for `reversible-token`:\n\n- Request-scoped memory inside the process that calls the provider and writes the response. Clear it in a `finally` block. It is not written to logs, traces, the prompt cache, Redis, or a second replica.\n- Keyed by this request only. Two callers who both produce `[EMAIL_1]` do not share a map.\n- Restore only placeholders this request minted, and only onto the bytes written to that caller.\n- Secrets and JWTs stay on `mask` or `typed-placeholder` even when the route's mode is `reversible-token`. A personal-data token does not contain their bytes when the spans overlap. Overlap resolution is the split that enforces this.\n- A later support tool that needs the mapping is a different store: encrypted, access-logged, short retention. It is not Log Analytics and it is not the vault.\n\nCache interaction: store the redacted completion. On a hit, restore with this request's vault only. A placeholder this request did not mint stays a placeholder (or becomes a mask). Applying another request's map would attach the wrong person to the token.\n\nIf redaction leaves a message that is only placeholders, send it. The model can answer or ask for a non-sensitive restatement. Do not paste the raw text back to \"help\" the model. After replacement, parse the message list again. If the body is no longer valid JSON, fail the request with the route policy. Do not send a broken document and hope the provider accepts it.\n\nLeave JSON keys, role names, tool names, and the document delimiters from **llm-app-prompt-injection-guardrails** intact. Redact what is inside them.\n\n## Streaming and chunk boundaries\n\nThe request is fully buffered and redacted before the provider call. Chunking matters on the response, and on any detector you run over model output.\n\nA detector that looks at one chunk in isolation misses a card split as `\"4242 42\"` + `\"42 4242 4242\"` and misses a placeholder split as `\"[EMA\"` + `\"IL_1]\"`. Hold a tail:\n\n1. Append the new payload to a buffer. Redact the text of an SSE `data:` field. Leave the event framing alone, and do not put prompt text in a heartbeat comment.\n2. Resolve overlaps on the buffer, then flush only the prefix that cannot still grow into an entity you scan or into a placeholder. Hold the suffix. Leave any byte still covered by a span in the buffer unflushed.\n3. Size the hold to the longest entity you scan. Cap scanned secrets (a JWT is the usual long one; pick a cap, for example 512 characters, and document it) and add a small overlap when the response recognizer needs context. Placeholder restore holds only the longest placeholder you issue, which is much shorter.\n4. Do not split a UTF-8 sequence.\n5. When the stream ends, run the same detector and the same overlap resolution on the tail and flush it.\n6. On error or client cancel, drop the buffer and the vault. The error log gets the error class, not the buffer.\n\nBytes you flush to the log and the trace are the redacted form. In reversible mode the caller socket gets restored text, and the log still shows `[EMAIL_1]`. Restored text is not an MDC value and not a span attribute.\n\nThis hold-back is independent of the routing skill's commit rule. That rule decides whether a fallback may start. This one decides which characters of a chunk are safe to emit. Apply redaction before the gateway treats a byte as committed model output.\n\n## Fail-closed and fail-open\n\nThe policy is explicit on every route. The default is fail-closed. Fail-open is a written exception for a route whose data class is not sensitive.\n\n- **Fail-closed.** The recognizer throws, times out, or returns a result you cannot parse: do not call the provider, do not write the raw body to a cache or a log, and return problem+json `503` with code `redaction_unavailable`. Zero findings is a success, not an error.\n- **Fail-open.** The same recognizer failure: call the provider with whatever the deterministic layer already redacted, increment `detector_error`, and alert. Fail-open does not mean \"skip redaction\". If the in-process deterministic scanner throws, fail closed anyway. That throw is a bug, not a dependency outage.\n- Deterministic findings already in hand are kept under both policies.\n\n```yaml\nllm:\n redaction:\n default-policy: fail-closed\n secret-scan-cap-chars: 512\n detectors:\n deterministic:\n entities: [email, phone, payment_card, api_key, jwt]\n # national_id stays off until schemes are listed\n ner:\n enabled: false\n timeout: 200ms\n min-confidence: 0.75\n exclude: [person_type, date]\n routes:\n support-chat:\n policy: fail-closed\n mode: reversible-token\n ner: true\n public-faq:\n policy: fail-open\n mode: typed-placeholder\n ner: false\n allowlist:\n literals:\n - support@example.com\n```\n\n## Never log the raw prompt\n\nOne filter is the only code allowed to record an LLM body, and it records the redacted view.\n\n- No `log.debug` of the serialized request or response on the provider client. No HTTP wiretap that dumps bodies unless the tap runs the redactor first.\n- Problem details and provider-error handling include entity counts and types. Provider error documents often echo the prompt; do not copy them into logs (**llm-gateway-routing-fallback-guardrails** already refuses that copy).\n- Metric labels and trace attributes are route, entity type, action, and counts. Not the value, and not a hash of the value. Emails, phones, and national IDs are low-entropy; a hash is a lookup table waiting for a wordlist.\n- MDC, baggage, and the usage store from **llm-cost-latency-budget-guardrails** do not carry message text.\n\n## Audit counters\n\n- `llm_redaction_entities_total` — counter. Labels: `route`, `entity_type`, `action` (`masked`, `placeholder`, `tokenized`, `allowed`, `detector_error`).\n- `llm_redaction_seconds` — histogram. Labels: `route`, `outcome` (`ok`, `failed_closed`, `failed_open`).\n- Span attributes: `redaction.entity_count` (a number) and `redaction.mode`. No span event contains the text.\n\nAlert when `detector_error` is non-zero on a fail-closed route, and when entity counts fall to zero while traffic does not (a detector turned off by accident).\n\n## Latency budget\n\nSet a budget and measure it. A deterministic pass over a typical prompt (about 8 KB) should stay within a few milliseconds on a warm JVM; treat a p95 above about 10 ms without a recognizer as a bug (a catastrophic expression, or scanning a binary attachment that should have been skipped). Record the stage on the gateway-overhead histogram in **llm-cost-latency-budget-guardrails** (auth, redaction, routing).\n\nA recognizer adds a network call. Put its timeout in the 150–300 ms band unless the route's deadline forces a tighter cap, and subtract that time from the alias deadline before the routing walk starts, the same way `minAttemptBudget` is applied in **llm-gateway-routing-fallback-guardrails**. Call the recognizer once per request. The log path reuses the view; it does not call the recognizer again.\n\nIf recognizer p95 does not fit the route SLO, turn it off on that route. Do not move the provider call ahead of the deterministic scan to save the time.\n\n## Tests\n\nSynthetic fixtures only. No customer dumps, no production allowlist of test cards.\n\n**Golden file.** For each case: input, expected spans (type, start, end), and the expected string under `mask`, `typed-placeholder`, and `reversible-token`. Include an email with a plus-tag, a phone with a country code, a published test PAN written with spaces that passes Luhn, a JWT-shaped token, and an `api_key=` assignment. Add a national-id case only for a scheme the config names.\n\n**False positives that must not fire.** UUIDs, git SHAs, semver, epoch milliseconds, order numbers, the role words `user` and `assistant`, the placeholder `[EMAIL_1]` itself, and a 16-digit run that fails Luhn.\n\n**False negatives that must fire.** The plus-tag email, the dashed PAN, the country-code phone, and a JWT with no `Bearer` prefix.\n\n**Overlap (reversible mode).** These cases check combination. Stub the recognizer so it returns the personal-data span at the fixture offsets, and let the deterministic layer find the API key or JWT. A live recognizer is not what this test measures.\n\n- **Nested.** The stubbed personal-data span strictly contains the secret. In the stub request and the log appender, the secret is absent and the personal-data text outside the secret is absent. The caller response and the vault do not contain the secret. Restoring the personal-data slices does not put the secret back.\n- **Partial overlap.** The stubbed personal-data span and the secret span overlap, and each extends past the other. In the stub request and the log appender, the secret, the shared bytes, and the personal-data-only tail are absent. The caller response and the vault do not contain the secret, the shared bytes, or the secret-only tail.\n- The message list still parses as JSON.\n- Passing these cases does not mean every secret in real traffic was detected. A secret no enabled detector returned is outside this test.\n\n**Leak test (required).** A stub provider (WireMock, MockWebServer, or the suite's existing stub) and a log appender (`ListAppender` or `OutputCaptureExtension`). Post a chat body that contains the fixture email, the spaced PAN, and the JWT.\n\n- The stub's recorded body contains none of those strings (nor the PAN with spaces removed).\n- No logged event contains them.\n- Counters increment for `email`, `payment_card`, and `jwt`.\n- On a reversible route, the stub returns text that echoes `[EMAIL_1]`. The caller response contains the original email. The appender contains `[EMAIL_1]` and does not contain the email.\n- After the call returns, including on the exception path, the vault is empty.\n\n**Streaming.** The stub splits the PAN across two deltas. The bytes written to the client and the bytes written to the appender do not contain the PAN. A second case splits `[EMAIL_1]` across two deltas and, on a reversible route, the caller receives the restored email once.\n\n**Policy.** Recognizer throws on a fail-closed route: the stub receives zero requests, the caller gets `503` `redaction_unavailable`, and the raw fixture is absent from the appender. The same throw on a fail-open route: the deterministic email is still redacted, the stub receives one request, `detector_error` increments.\n\n**Cache.** Two requests redact to the same placeholder shape. The second response does not restore the first caller's value.\n\n## Workflow\n\n1. Inventory every call site, logger, wiretap, cache, and eval export that can see a prompt or a completion.\n2. Add the deterministic layer, the three modes, overlap resolution, and `default-policy: fail-closed`. Secrets stay non-reversible on every slice they touch, and the union of overlapping spans is what gets redacted.\n3. Point the provider client, logs, traces, cache, and fixture export at the redacted view.\n4. Add response hold-back. Add restore-on-the-way-out only for routes that set `reversible-token`.\n5. Add the counters and the latency histogram. Check the label set.\n6. Land the golden, false-positive, false-negative, streaming, policy, and leak tests before a route is enabled.\n7. Enable the recognizer only on routes whose deadline still fits the timeout, with the excluded categories written down.\n\n## Tool-specific notes\n\n- **GitHub Copilot:** path-scoped `.github/instructions/llm-pii-redaction.instructions.md` with `applyTo` for `**/llm/**`, `**/*Redact*.java`, `**/*Pii*.java`, `**/*Vault*.java`, `**/gateway/**`, and `**/application*.{yml,yaml,properties}`. Put the pipeline order, the mode table, and the fail-closed default at the top. Keep the procedure in an optional `.github/skills/llm-pii-redaction-guardrails/SKILL.md`. Ask for the call-site inventory before editing a filter.\n- **Claude Code:** `.claude/skills/llm-pii-redaction-guardrails/SKILL.md` holds the procedure. Add one `CLAUDE.md` fact: \"LLM provider calls, logs, traces, caches, and eval fixtures see only the redacted view. The default redaction policy is fail-closed. Secrets are never restored.\"\n- **Codex:** root `AGENTS.md` stays a short brief — pipeline order, the config key for the route policy, and the leak-test command. The procedure lives in `SKILL.md`, or in a nested `AGENTS.md` beside the gateway module when the repo already uses nested briefs. Do not paste this whole skill into `AGENTS.md`.\n\n## Anti-patterns\n\n- Redacting in the log encoder and still sending the raw body to the provider\n- A debug flag, wiretap, or `toString()` on the request object that prints the raw prompt in production\n- Treating a system-prompt sentence as the redaction control\n- A vault in a shared cache, keyed by a guessable id, or reused across callers\n- Restoring placeholders into the log line, the trace, or the cached completion\n- A recognizer as the only detector, so a timeout on a fail-open route sends raw cards\n- Matching every 16-digit run and skipping Luhn\n- A production allowlist of test PANs\n- Scanning each stream chunk on its own, with no tail hold\n- Fail-open as the unspoken default\n- Entity values or their hashes as metric labels\n- A customer transcript committed under `evals/golden`\n- Calling the provider after the recognizer timed out on a fail-closed route\n- One surviving span when findings overlap, chosen by length or by a deterministic tie-break only when lengths are equal, so a reversible personal-data span restores a contained API key or JWT, or a partial overlap leaves the other span's tail in clear text\n- Writing an original outer span back over slices that already separated a secret from the surrounding personal data\n\n## Done when\n\n- [ ] One redacted view is built before the provider call, and logs, traces, caches, and eval exports use that view\n- [ ] Deterministic detectors cover email, phone, Luhn-checked cards, configured national IDs, API keys, and JWTs, with checksum failures dropped\n- [ ] The recognizer is optional, confidence-gated, category-filtered, and span-mapped back onto the original text\n- [ ] Allowlist hits are explicit and counted as `allowed`\n- [ ] Each route sets `mask`, `typed-placeholder`, or `reversible-token`, and secrets are never reversible\n- [ ] Overlapping spans redact the full union. Any slice that intersects an API key, JWT, or other secret stays non-restorable. Reversible-mode tests cover a nested secret and a partial overlap: the secret is not restored, and the stub request and the log appender do not contain the secret, the shared bytes, or a personal-data tail from those spans\n- [ ] The vault is request-scoped, cleared in `finally`, and used to restore only on the caller response\n- [ ] Stream output holds a tail so entities and placeholders split across chunks are still handled\n- [ ] `default-policy` is fail-closed, and every fail-open route is named in config\n- [ ] Raw prompts are absent from logs, metric labels, and trace attributes; counters are by entity type and action\n- [ ] Redaction time is inside the gateway latency budget, and the recognizer timeout is subtracted before routing\n- [ ] Golden, false-positive, false-negative, streaming, policy, cache, and leak tests pass, including a test where the stub provider and the log appender never see the fixture values\n\n## Copy-ready SKILL.md stub\n\n```markdown\n---\nname: llm-pii-redaction-guardrails\ndescription: Use when a Java or Spring LLM gateway, or a client of Azure OpenAI, Bedrock, OpenRouter, or another OpenAI-compatible provider, must redact personal data and secrets before the provider call and before logs, traces, caches, and eval fixtures. Covers deterministic detectors (email, phone, Luhn cards, national IDs, API keys, JWTs), an optional recognizer, allowlists, mask / typed placeholder / per-request reversible tokens, streaming hold-back, fail-closed by default, counters by entity type, a latency budget, and a leak test against a stub provider and a log appender.\n---\n\nFollow LLM PII and Secret Redaction Guardrails: build one redacted view before the provider call, and send that view to logs, traces, the prompt cache, and eval fixtures. Run deterministic detectors first (checksums included; cards must pass Luhn), then an optional recognizer, then an allowlist. Choose mask, typed placeholder, or reversible tokens per route. Keep the vault in the request, clear it in finally, and restore values only on the response to that caller. Secrets and JWTs are never restored, including when a longer personal-data span contains or overlaps them. Redact the union of overlapping spans, and keep each secret slice non-restorable. Hold back the tail of a stream so a value split across chunks is still caught. Default to fail-closed when a detector errors. Count entities by type and action, never by value. Prove with a test that the stub provider and the log appender do not receive the raw fixtures.\n```\n" } ] }