Repository navigation
Conversation
Decoder-based embedding IRs exported before optimum-intel 1.26, such as
Echo9Zulu/Qwen3-Embedding-0.6B-int8_asym-ov (optimum-intel 1.25.2), have
position_ids as a graph input. OVModelForFeatureExtraction.forward() only
passes input_ids and attention_mask (it drops extra kwargs), so:
- GPU: every request fails with "Got unexpected inputs: expecting the
following inputs {'position_ids', 'attention_mask', 'input_ids'}",
even a single string.
- CPU: inference runs on an unset tensor and returns NaN, which then
fails JSON encoding ("Out of range float values are not JSON
compliant: nan").
When the compiled model lists position_ids among its inputs, build them
from the attention mask (cumsum - 1, padding positions set to 1, the
same rule optimum-intel's OVModelForCausalLM uses, so left and right
padding both work) and call the compiled model with all three inputs.
IRs without position_ids (encoder models, current optimum-intel
exports) keep the forward() path unchanged.
Fixes SearchSavior#106.
Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
Hardening for the position_ids path added in the previous commit:
- Also fill token_type_ids (zeros) and decoder_input_ids when the IR
takes them, as OVModelForFeatureExtraction.forward() does, so an IR
with position_ids plus one of these still gets every input.
- At load, log when an IR takes position_ids, and warn about inputs the
engine cannot fill, with a pointer to re-export.
- Integration test: a padded batch of different lengths instead of one
text, finite unit-length vectors, and a text alone vs in the batch.
The old test (one vector, len > 0) passed on the bug; this one fails
on main with the old Qwen3-Embedding export ("Got unexpected inputs:
expecting ... position_ids").
Checked on both IR layouts of Qwen3-Embedding-0.6B int8: the Echo9Zulu
export (optimum-intel 1.25.2, takes position_ids) and a re-export with
optimum-intel 2.3.0.dev0 (input_ids + attention_mask only, forward()
path).
Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
Plain OpenAI clients (openai SDK, llama-index OpenAILikeEmbedding) send only model and input, never OpenArc's tokenizer config. The tokenizer defaults were padding=False and truncation=False, so any batch of texts with different lengths failed with "Unable to create tensor, you should probably activate truncation and/or padding". /v1/embeddings now fills server defaults (padding=True, truncation=True, max_length=8192, the values of Qwen3-Embedding's reference usage) for every tokenizer field the client did not set. A client config still wins field by field. Truncation is preferred over rejecting long inputs because an engine-side error unloads the model. `dimensions` was copied into the tokenizer's max_length, which cut the input to N tokens and still returned full-size vectors. In the OpenAI API it sets the output size: the route now keeps the first N values and re-normalizes (Matryoshka-trained models such as Qwen3-Embedding). dimensions < 1, or larger than the model's vector, is a 400. Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
Align /v1/embeddings with the OpenAI API and with how TEI, vLLM and Ollama serve it: - Truncation length: the fixed max_length=8192 default would still overflow models with a smaller limit (BERT-style, 512). With no client max_length the engine now truncates to the model's own limit (config max_position_embeddings / tokenizer model_max_length), capped at 8192 tokens to bound memory per request. TEI (auto_truncate, default on) and Ollama (truncate, default true) also truncate to the model limit by default. - encoding_format: "base64" returns little-endian float32 bytes, the OpenAI wire format, which TEI, vLLM and Ollama also implement. The openai SDK requests base64 by default and decodes it; a client that asked for base64 explicitly used to get floats. Values other than "float" and "base64" are rejected. - usage: prompt_tokens and total_tokens count the real (non-padding) input tokens instead of always 0. Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
meatposes
marked this pull request as ready for review
November 22, 2026 07:49
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: Lets standard OpenAI apps get embeddings from OpenArc without errors or extra settings.
stacked on #204 (the position_ids fix): the first 2 commits here are that PR. review that one first; only the last 2 commits are new here.
OpenAI clients (openai sdk, llama-index
OpenAILikeEmbedding, Paperless-ngx's AI index, anything) sendmodel+inputand noconfig. on main:PreTrainedTokenizerConfigdefaults topadding=False, truncation=False, so the tokenizer raises "Unable to create tensor, you should probably activate truncation and/or padding". it happens in the worker, so the model is unloaded too. this hits every embedding model, not only Qwen3dimensionsis copied into the tokenizer'smax_length, sodimensions=256cuts the input to 256 tokens and still returns full-size vectors. in the OpenAI API it's the output sizeencoding_formatis ignored,"base64"returns floatsusage.prompt_tokensis always 0this makes the route behave the way the OpenAI API does and the way TEI, vLLM and Ollama serve it:
max_position_embeddings/ tokenizermodel_max_length), capped at 8192auto_truncatedefault on, Ollamatruncatedefault true; vLLM and OpenAI rejectdimensions=Nmax_length, full-size outputencoding_format="base64"usage.prompt_tokensdefaults only fill tokenizer fields the client didn't set in
config, so a client config still wins field by field. truncating instead of rejecting long inputs also matters here because an error inside the engine unloads the model. unknownencoding_formatvalues are rejected by the schema.not done: vLLM only allows
dimensionsfor models flagged matryoshka; there's no such flag in an OpenVINO export, so here it's allowed for any model and is the caller's call. the openai sdk asks for base64 by default and decodes it; it also accepted the float lists before this PR.tested on a B50, Qwen3-Embedding-0.6B int8 (the Echo9Zulu export and an optimum-intel 2.3 re-export), 16 checks each, all pass: a plain batch of 3 different lengths gives three 1024-d unit vectors, the same text alone and in a padded batch gives cos 1.000000, "water heater repair invoice" is closer to "plumbing bill for the hot water tank" (0.78) than to "daycare tuition statement" (0.53),
dimensions=256gives 256-d unit vectors, a ~15k token input is truncated and answered, base64 decodes to the float vectors, baddimensionsandencoding_formatare 4xx and the model stays loaded.also ran in one process next to kokoro, whisper, qwen3-asr, qwen3-tts and diarization for 2 minutes of mixed traffic: 636 embedding batches, 209 kokoro requests, 9 qwen3-asr transcriptions, 0 errors, every model stayed loaded.
latency on the B50: 1 short string 22 ms, 8 texts ~0.2 s, 32 x ~260 tokens 3.7 s.
unit: route defaults, client config overriding them,
dimensions, base64 format, usage, the model-limit truncation rule (Qwen3 32k -> 8192, bert 512 -> 512). unit suite: only main's 2 known kokoro failures.https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR