Skip to content

perf(llm): preserve prompt prefixes and apply provider-specific caching - #1421

Open
appergb wants to merge 2 commits into
fix/overview-segment-radiusfrom
perf/provider-prompt-cache
Open

appergb wants to merge 2 commits into
fix/overview-segment-radiusfrom
perf/provider-prompt-cache

Conversation

@appergb

@appergb appergb commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Changing the foreground app previously rewrote the start of the entire system prompt, preventing reuse of otherwise identical style rules. Stable rules now precede per-request app/cursor context across polish, translation, QA and multimodal composition; text envelopes and injection defenses are preserved.

Cache policy uses verified HTTPS service origins, API formats and known model capabilities: OpenAI routing and supported Responses breakpoints, Anthropic stable blocks, OpenRouter sticky routing/history caching, selected DashScope models, and Ark Seed routing hints. DeepSeek and other implicit-cache services keep stable prefixes without incompatible vendor parameters. Existing caller controls, store/TTL choices and all prompt text are preserved. Multi-turn requests retain history reuse.

Usage records now snapshot service/protocol and canonical inclusive/additive cache accounting. The UI shows token-weighted cache share and comparable-report counts per service/channel/model/protocol. Unknown legacy reports stay unknown. No rate guarantee or live-provider improvement is claimed; full native Gemini/Codex/attempt accounting remains tracked separately.

Validation: 1,469 Core library tests passed before the final focused history-policy additions; all 17 cache-policy tests and the request-conversion/header integration test passed, as did all-targets strict Clippy. Frontend helper tests, production build, Chromium/WebKit Usage checks and a populated WebKit fixture passed. Isolated iOS Core preparation passed. The macOS arm64 app/DMG build, codesign checks and disk-image verification passed on final commit 7caeb87. Remote macOS CI 42579132960 and iOS CI 42579095750 both passed on that same commit, including the iPhone device build.

Stacked on #1420. See docs/prompt-cache-policy.md for the maintained contract and official sources.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T07:25:23.311755Z 7caeb87 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb8e36e818

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +357 to +360
"dashscope"
if format == LlmRequestFormat::ChatCompletions && dashscope_explicit(config) =>
{
split_first_message(body, format, false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve history caching for multi-turn DashScope requests

For multi-turn requests to supported Qwen models, including selection Q&A through chat_completion_history_streaming, this branch enables explicit caching but marks only the static system prefix. DashScope documents that explicit and implicit caching are mutually exclusive, so the repeated selection and conversation history after that marker lose the implicit-cache eligibility they had before this change. Long conversations consequently reprocess that history on every turn, increasing input costs and latency. Preserve implicit caching when conversation history is present, or add an explicit breakpoint covering the completed history as the Anthropic policy does.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant