Skip to content

embeddings: make /v1/embeddings work for plain OpenAI clients - #205

Open
meatposes wants to merge 4 commits into
SearchSavior:mainfrom
meatposes:emb-openai-compat
Open

meatposes wants to merge 4 commits into
SearchSavior:mainfrom
meatposes:emb-openai-compat

Conversation

@meatposes

Copy link
Copy Markdown
Contributor

TL;DR: Lets standard OpenAI apps get embeddings from OpenArc without errors or extra settings.


stacked on #204 (the position_ids fix): the first 2 commits here are that PR. review that one first; only the last 2 commits are new here.

OpenAI clients (openai sdk, llama-index OpenAILikeEmbedding, Paperless-ngx's AI index, anything) send model + input and no config. on main:

  1. any batch with texts of different lengths fails: PreTrainedTokenizerConfig defaults to padding=False, truncation=False, so the tokenizer raises "Unable to create tensor, you should probably activate truncation and/or padding". it happens in the worker, so the model is unloaded too. this hits every embedding model, not only Qwen3
  2. dimensions is copied into the tokenizer's max_length, so dimensions=256 cuts the input to 256 tokens and still returns full-size vectors. in the OpenAI API it's the output size
  3. encoding_format is ignored, "base64" returns floats
  4. usage.prompt_tokens is always 0

this makes the route behave the way the OpenAI API does and the way TEI, vLLM and Ollama serve it:

main this PR others
batch of different lengths, no config "Unable to create tensor" (and the model unloads) padded TEI, vLLM, Ollama pad server-side
input longer than the model tokenizer/model error truncated to the model's limit (max_position_embeddings / tokenizer model_max_length), capped at 8192 TEI auto_truncate default on, Ollama truncate default true; vLLM and OpenAI reject
dimensions=N becomes tokenizer max_length, full-size output first N values, re-normalized; < 1 or > vector size is a 400 same in TEI, vLLM, Ollama
encoding_format="base64" ignored, floats little-endian float32, base64 (OpenAI wire format) same in TEI, vLLM, Ollama
usage.prompt_tokens 0 real input tokens, no padding all report it

defaults only fill tokenizer fields the client didn't set in config, so a client config still wins field by field. truncating instead of rejecting long inputs also matters here because an error inside the engine unloads the model. unknown encoding_format values are rejected by the schema.

not done: vLLM only allows dimensions for models flagged matryoshka; there's no such flag in an OpenVINO export, so here it's allowed for any model and is the caller's call. the openai sdk asks for base64 by default and decodes it; it also accepted the float lists before this PR.

tested on a B50, Qwen3-Embedding-0.6B int8 (the Echo9Zulu export and an optimum-intel 2.3 re-export), 16 checks each, all pass: a plain batch of 3 different lengths gives three 1024-d unit vectors, the same text alone and in a padded batch gives cos 1.000000, "water heater repair invoice" is closer to "plumbing bill for the hot water tank" (0.78) than to "daycare tuition statement" (0.53), dimensions=256 gives 256-d unit vectors, a ~15k token input is truncated and answered, base64 decodes to the float vectors, bad dimensions and encoding_format are 4xx and the model stays loaded.

also ran in one process next to kokoro, whisper, qwen3-asr, qwen3-tts and diarization for 2 minutes of mixed traffic: 636 embedding batches, 209 kokoro requests, 9 qwen3-asr transcriptions, 0 errors, every model stayed loaded.

latency on the B50: 1 short string 22 ms, 8 texts ~0.2 s, 32 x ~260 tokens 3.7 s.

unit: route defaults, client config overriding them, dimensions, base64 format, usage, the model-limit truncation rule (Qwen3 32k -> 8192, bert 512 -> 512). unit suite: only main's 2 known kokoro failures.

https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR

Decoder-based embedding IRs exported before optimum-intel 1.26, such as
Echo9Zulu/Qwen3-Embedding-0.6B-int8_asym-ov (optimum-intel 1.25.2), have
position_ids as a graph input. OVModelForFeatureExtraction.forward() only
passes input_ids and attention_mask (it drops extra kwargs), so:

- GPU: every request fails with "Got unexpected inputs: expecting the
  following inputs {'position_ids', 'attention_mask', 'input_ids'}",
  even a single string.
- CPU: inference runs on an unset tensor and returns NaN, which then
  fails JSON encoding ("Out of range float values are not JSON
  compliant: nan").

When the compiled model lists position_ids among its inputs, build them
from the attention mask (cumsum - 1, padding positions set to 1, the
same rule optimum-intel's OVModelForCausalLM uses, so left and right
padding both work) and call the compiled model with all three inputs.
IRs without position_ids (encoder models, current optimum-intel
exports) keep the forward() path unchanged.

Fixes SearchSavior#106.

Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
Hardening for the position_ids path added in the previous commit:

- Also fill token_type_ids (zeros) and decoder_input_ids when the IR
  takes them, as OVModelForFeatureExtraction.forward() does, so an IR
  with position_ids plus one of these still gets every input.
- At load, log when an IR takes position_ids, and warn about inputs the
  engine cannot fill, with a pointer to re-export.
- Integration test: a padded batch of different lengths instead of one
  text, finite unit-length vectors, and a text alone vs in the batch.
  The old test (one vector, len > 0) passed on the bug; this one fails
  on main with the old Qwen3-Embedding export ("Got unexpected inputs:
  expecting ... position_ids").

Checked on both IR layouts of Qwen3-Embedding-0.6B int8: the Echo9Zulu
export (optimum-intel 1.25.2, takes position_ids) and a re-export with
optimum-intel 2.3.0.dev0 (input_ids + attention_mask only, forward()
path).

Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
Plain OpenAI clients (openai SDK, llama-index OpenAILikeEmbedding) send
only model and input, never OpenArc's tokenizer config. The tokenizer
defaults were padding=False and truncation=False, so any batch of texts
with different lengths failed with "Unable to create tensor, you should
probably activate truncation and/or padding".

/v1/embeddings now fills server defaults (padding=True, truncation=True,
max_length=8192, the values of Qwen3-Embedding's reference usage) for
every tokenizer field the client did not set. A client config still
wins field by field. Truncation is preferred over rejecting long inputs
because an engine-side error unloads the model.

`dimensions` was copied into the tokenizer's max_length, which cut the
input to N tokens and still returned full-size vectors. In the OpenAI
API it sets the output size: the route now keeps the first N values and
re-normalizes (Matryoshka-trained models such as Qwen3-Embedding).
dimensions < 1, or larger than the model's vector, is a 400.

Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
Align /v1/embeddings with the OpenAI API and with how TEI, vLLM and
Ollama serve it:

- Truncation length: the fixed max_length=8192 default would still
  overflow models with a smaller limit (BERT-style, 512). With no
  client max_length the engine now truncates to the model's own limit
  (config max_position_embeddings / tokenizer model_max_length), capped
  at 8192 tokens to bound memory per request. TEI (auto_truncate,
  default on) and Ollama (truncate, default true) also truncate to the
  model limit by default.
- encoding_format: "base64" returns little-endian float32 bytes, the
  OpenAI wire format, which TEI, vLLM and Ollama also implement. The
  openai SDK requests base64 by default and decodes it; a client that
  asked for base64 explicitly used to get floats. Values other than
  "float" and "base64" are rejected.
- usage: prompt_tokens and total_tokens count the real (non-padding)
  input tokens instead of always 0.

Claude-Session: https://claude.ai/code/session_01H45h8vWtUhMNST1Fk179WR
@meatposes
meatposes marked this pull request as ready for review November 22, 2026 07:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant