{"intro":"Upstream work on the LLM serving stack: routing and guardrails, KV-cache reuse, attention kernels, tokenizers and quantized-weight loading. Most of it is one kind of bug. The system does not crash or log; it returns a confident answer that is wrong. Verified against the GitHub API on 21 Oct 2026.","stats":[{"number":"114","label":"Pull requests"},{"number":"86","label":"Merged upstream"},{"number":"71","label":"Issues filed"},{"number":"10","label":"Projects"}],"groups":[{"id":"semantic-router","name":"vLLM Semantic Router","repo":"vllm-project/semantic-router","blurb":"The routing layer of the vLLM project (not the vLLM engine itself). Committer and lead of the Router Models & Inference Runtime working group.","prs":98,"merged":83},{"id":"transformers","name":"Hugging Face transformers","repo":"huggingface/transformers","blurb":"Quantized-weight loading (merged) and tokenizer routing.","prs":5,"merged":2},{"id":"mistral-rs","name":"mistral.rs","repo":"EricLBuehler/mistral.rs","blurb":"Vision-model support and engine fixes.","prs":5,"merged":1},{"id":"rbln","name":"RBLN · work at Rebellions","repo":"RBLN-SW","blurb":"NPU serving stack: issue reports and prototypes."},{"id":"own-repos","name":"My own repos","repo":"subin9","blurb":"Public NPU and Rust inference work."},{"id":"vllm","name":"vLLM","repo":"vllm-project/vllm","blurb":"LoRA classification heads, a Rust frontend and OpenTelemetry conformance.","prs":3,"merged":0},{"id":"ollama","name":"Ollama","repo":"ollama/ollama","blurb":"Batch size for noncausal decision-model GGUFs.","prs":1,"merged":0},{"id":"exaone","name":"LG AI EXAONE","repo":"LGAI-EXAONE/EXAONE-4.0-32B","blurb":"Chat-template fix on the model repo.","prs":1,"merged":0}],"items":[{"id":"hf-quantized-weights","date":"2026-07-08","dateLabel":"Jul 8 – Aug 20, 2026","kind":"pr","project":"huggingface/transformers","group":"transformers","ref":"PR #47152 · PR #48072","status":"2 merged upstream","title":"Silent corruption in FP8 weight loading: a fix, its revert, and 42 relabelled repos","body":["FP8 checkpoints saved with status \"frozen\" loaded raw fp8 weights and never applied their scales: `q_proj` max was 448 instead of ~0.68. There was no error; the model just generated noise. I co-authored the fix (#47152, merged 22 Jul), which loads them with their `weight_scale`. It then broke checkpoints whose `quantization_status` was mislabeled, so I reproduced the break within a day, sampled 536 Hub configs to size it, and wrote the revert (#48072, merged 20 Aug), which both the transformers and Red Hat maintainers approved. I then listed 42 nm-testing repos with the wrong `quantization_status`, and Red Hat relabelled them (\"should be fixed now\")."],"tags":["Quantization","FP8","Merged upstream"],"links":[{"label":"Issue #47145","url":"https://github.com/huggingface/transformers/issues/47145"},{"label":"PR #47152","url":"https://github.com/huggingface/transformers/pull/47152"},{"label":"Revert #48072","url":"https://github.com/huggingface/transformers/pull/48072"}],"headline":"FP8 checkpoints loaded with scales never applied, so models generated noise with no error. Co-authored the fix, wrote the revert when it broke mislabeled checkpoints, and listed 42 Hub repos that Red Hat relabelled.","featured":true,"breakdown":{"problem":"FP8 checkpoints saved with status \"frozen\" loaded without their scales. There was no error; the model just generated noise.","cause":"Raw fp8 weights were loaded and their scales never applied: `q_proj` max was 448 instead of ~0.68.","fix":["Co-authored the fix, which loads these checkpoints with their `weight_scale` (#47152, merged 22 Jul).","When that fix broke checkpoints with a mislabeled `quantization_status`, reproduced the break within a day and sampled 536 Hub configs to size it.","Wrote the revert (#48072, merged 20 Aug), approved by both the transformers and Red Hat maintainers.","Listed 42 nm-testing repos with the wrong `quantization_status`."],"result":["2 PRs merged upstream: the fix (#47152) and its revert (#48072).","Red Hat relabelled the 42 repos (\"should be fixed now\")."]}},{"id":"sr-truncation-class","date":"2026-10-06","dateLabel":"Aug 31 – Oct 6, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"11 PRs · #3206 · #3643 · #3402 · #3460 · #4567","status":"11 merged upstream","title":"Guardrails that read only the first 512 tokens: fixed at every call site, then at the kernel","body":["The classifiers truncated input at 512 tokens without saying so. A guardrail that scored only the opening of a prompt and found nothing looked exactly like one that had read the whole prompt. I found this first in the security API, then in the response jailbreak filter, the PII API, the hallucination detector and RAG retrieval. These six call sites were fixed across #3206, #3334, #3369, #3643, #3650 and #3841, all through one overlapping-window walk in which a clean verdict requires every chunk to have been scored. I then removed the reason for the limit. The classifiers built a full seq × seq attention matrix, and moving them onto a chunked SDPA kernel cut peak memory at 8,192 tokens from 15,789 MiB to 1,003 MiB for one global ModernBERT attention block on CPU in F32 (15.4 GiB to 1.0 GiB, about 16× lower; a local layer drops 62×). Qwen3 at 8K, previously not runnable, now fits in 3.4 GiB, and every site is pinned to its dense implementation within 1e-4. A follow-up (#4567) fixed where chunked hallucination spans landed: each was placed by searching the answer for its chunk's text, so in an answer that repeats itself, spans from later chunks were reported on earlier copies."],"tags":["Guardrails","Attention kernels","Memory"],"links":[{"label":"PR #3206","url":"https://github.com/vllm-project/semantic-router/pull/3206"},{"label":"PR #3334","url":"https://github.com/vllm-project/semantic-router/pull/3334"},{"label":"PR #3369","url":"https://github.com/vllm-project/semantic-router/pull/3369"},{"label":"PR #3643","url":"https://github.com/vllm-project/semantic-router/pull/3643"},{"label":"PR #3650","url":"https://github.com/vllm-project/semantic-router/pull/3650"},{"label":"PR #3841","url":"https://github.com/vllm-project/semantic-router/pull/3841"},{"label":"PR #3402","url":"https://github.com/vllm-project/semantic-router/pull/3402"},{"label":"PR #3460","url":"https://github.com/vllm-project/semantic-router/pull/3460"},{"label":"PR #4567","url":"https://github.com/vllm-project/semantic-router/pull/4567"}],"headline":"Guardrails silently read only the first 512 tokens. Fixed at 6 call sites over six PRs, then removed the cause: 15.4 → 1.0 GiB peak memory at 8K tokens (global layer).","featured":true,"breakdown":{"problem":"Guardrail classifiers truncated input at 512 tokens without saying so. A guardrail that scored only the opening of a prompt looked the same as one that read all of it.","cause":"The classifiers built a full seq × seq attention matrix, which is why the limit existed.","fix":["Found the truncation in the security API, the response jailbreak filter, the PII API, the hallucination detector and RAG retrieval.","Fixed all six call sites with one overlapping-window walk, where a clean verdict requires every chunk to be scored (#3206, #3334, #3369, #3643, #3650, #3841).","Moved the classifiers onto a chunked SDPA kernel to remove the reason for the limit.","Fixed where chunked hallucination spans landed: in an answer that repeats itself, spans from later chunks had been reported on earlier copies (#4567)."],"result":["Peak memory at 8,192 tokens fell from 15,789 MiB to 1,003 MiB for one global ModernBERT attention block on CPU in F32 (15.4 GiB to 1.0 GiB, about 16× lower; a local layer drops 62×).","Qwen3 at 8K, previously not runnable, now fits in 3.4 GiB, and every site is pinned to its dense implementation within 1e-4.","11 PRs merged upstream."]}},{"id":"sr-runtime-reliability","date":"2026-10-10","dateLabel":"Sep 1 – Oct 10, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #3303 · #3454 · #4425 · #4508 · #4629 · #4630","status":"6 merged upstream","title":"Serving-path failures that never showed up in the logs","body":["A signal pinned to CPU loaded the FP16 ONNX graph and paid for a cast around every operator, running about 10× slower with nothing logged, because model ranking never saw the execution provider (#3303). Five published \"fp16\" files held FP32 weights; #3454 traced them to the export tool, which CI had never tested. In the model runtime, an unpaired surrogate in a request returned 500 instead of 400 (#4508). A device the host lacked failed only after the model had been downloaded, and again on every retry; the device is now checked before the download, and the supervisor's forced kills are logged instead of silent (#4629). The two images that install the runtime resolved its dependencies from lower bounds, so two builds of one release could differ; both now install from one lock (#4630). Half of the CLI unit tests had also never run in CI, and #4425 now runs all 3,016."],"tags":["Reliability","ONNX Runtime","CI"],"links":[{"label":"PR #3303","url":"https://github.com/vllm-project/semantic-router/pull/3303"},{"label":"PR #3454","url":"https://github.com/vllm-project/semantic-router/pull/3454"},{"label":"PR #4425","url":"https://github.com/vllm-project/semantic-router/pull/4425"},{"label":"PR #4629","url":"https://github.com/vllm-project/semantic-router/pull/4629"},{"label":"PR #4630","url":"https://github.com/vllm-project/semantic-router/pull/4630"}],"headline":"10× CPU slowdown with nothing logged, 500s on malformed input, a download before a doomed device check, and half the CLI tests never running in CI.","featured":true,"breakdown":{"problem":"Several serving-path failures left nothing in the logs: a CPU signal ran about 10× slower, malformed input returned 500, a missing device failed only after the model download, and half of the CLI unit tests never ran in CI.","cause":"The slowdown came from model ranking never seeing the execution provider, so a signal pinned to CPU loaded the FP16 ONNX graph and paid for a cast around every operator.","fix":["Fixed the CPU slowdown (#3303) and traced five published \"fp16\" files that held FP32 weights to the export tool, which CI had never tested (#3454).","Made an unpaired surrogate in a request return 400 instead of 500 (#4508).","Checked the device before the download and logged the supervisor's forced kills instead of leaving them silent (#4629).","Installed the runtime in both images from one lock instead of from lower bounds (#4630), and ran all CLI unit tests in CI (#4425)."],"result":["All 3,016 CLI unit tests now run in CI.","A device the host lacks now fails before the download, not after it and again on every retry.","6 PRs merged upstream."]}},{"id":"sr-gpu-fast-path","date":"2026-10-13","dateLabel":"Oct 9 – 13, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #4779 · #4811 · #4813","status":"3 merged upstream","title":"GPU fast paths for the router's own models, admitted only where answers hold","body":["The runtime's fused element-wise kernels were written for MI300X. On NVIDIA they do not reproduce the eager ops bit for bit (`add_rmsnorm` differs by one BF16 ulp), so #4779 registers them on CUDA as approximate kernels, used only for a model family that opts in under a 99%-per-question-type agreement rule. `max_speed` selected them at the time; since the runtime replaced its profiles with engine arguments, the default `--dtype auto` does. Decision 2.0 opts in: 11% to 48% faster on an RTX 4070 Ti SUPER, with none of 102 decisions changed. Vela 2.0 does not, because its 0.8B changed 23 of 336 span answers. #4811 measures each fast path switched off on an MI300X and shows that Vela 2.0 never replayed a CUDA graph. #4813 makes its decoder forest replay per shape on CUDA: on an RTX 4070 Ti SUPER a one-question request goes from 67.7 to 17.7 ms on the 0.8B, still bit for bit. A reviewer's run showed that each forest graph held tens of MiB that no limit counted, and that a kept result could be overwritten by the next request; the forest path now returns a copy, and the graph budget charges each capture the device memory it takes, up to 4 GiB or half of the free memory (#5009). ROCm stays eager, since a live forest graph faults inside the release image's ROCm 7.2.3. All three PRs are merged."],"tags":["GPU kernels","CUDA graphs","Serving"],"links":[{"label":"PR #4779","url":"https://github.com/vllm-project/semantic-router/pull/4779"},{"label":"PR #4811","url":"https://github.com/vllm-project/semantic-router/pull/4811"},{"label":"PR #4813","url":"https://github.com/vllm-project/semantic-router/pull/4813"}],"headline":"Fused kernels on CUDA only where answers hold: Decision 2.0 11–48% faster with 0 of 102 decisions changed; Vela forest graphs 67.7 → 17.7 ms on the 0.8B on an RTX 4070 Ti SUPER (4B: 1.9×).","featured":true,"breakdown":{"problem":"The runtime's fused element-wise kernels were written for MI300X. On NVIDIA they do not reproduce the eager ops bit for bit (`add_rmsnorm` differs by one BF16 ulp).","fix":["Registered the fused kernels on CUDA as approximate kernels, used only for a model family that opts in under a 99%-per-question-type agreement rule (#4779). `max_speed` selected them at first; the default `--dtype auto` does now.","Measured each fast path switched off on an MI300X, which showed that Vela 2.0 never replayed a CUDA graph (#4811).","Made Vela 2.0's decoder forest replay per shape on CUDA (#4813).","After a reviewer's run found uncounted graph memory and kept results being overwritten, made the forest path return a copy and the graph budget charge each capture its device memory, up to 4 GiB or half of the free memory (#5009)."],"result":["Decision 2.0 opts in: 11% to 48% faster on an RTX 4070 Ti SUPER, with none of 102 decisions changed. Vela 2.0 does not, because its 0.8B changed 23 of 336 span answers.","Vela forest graphs on an RTX 4070 Ti SUPER: a one-question request on the 0.8B goes from 67.7 to 17.7 ms, still bit for bit (4B: 1.9×).","All three PRs are merged. ROCm stays eager, since a live forest graph faults inside the release image's ROCm 7.2.3."]}},{"id":"sr-d3-quantized","date":"2026-10-18","dateLabel":"Oct 13 – 18, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #5066 · vllm-sr/d3-FP8 · vllm-sr/d3-W4A16-AWQ","status":"Merged upstream · packages published","title":"Decision 3.0 (27B) in FP8 and int4, loaded as stored","headline":"FP8 keeps 99.08% of BF16 answers in 31.9 GB instead of 52.2 GB; AWQ int4 keeps 97.76% in about 16.5 GB.","body":["#5066 (merged 18 Oct) lets the model runtime load FP8 (W8A8 FP8_DYNAMIC) and int4 (GPTQ, AWQ) compressed-tensors packages as stored instead of expanding them to BF16. For FP8, activations are quantized per row in one Triton kernel and multiplied in `torch._scaled_mm`. On MI300 the E4M3 bits are reinterpreted as e4m3fnuz with doubled scales; on Ada at 256 rows or more, under the approximate profile, the product is taken unscaled in FP32 and then scaled. On 14,031 test requests, FP8 keeps 99.08% of BF16's answers in 31.9 GB instead of 52.2 GB, and AWQ int4 keeps 97.76% in about 16.5 GB. On MI300X the FP8 median is 36.8 ms against 46.4 ms for BF16 with d3's forward replayed as graphs (#4999; record corrected in #5270); without graphs it is 80.9 against 67.1 ms, because per-layer host cost outweighs the smaller matmuls.","Two bugs were found and fixed inside the PR. The FP8 activation kernel indexed rows in 32 bits and returned NaN past 2^31 elements; it now uses 64-bit indexing. On Ada, PyTorch before 2.12 zeroed the stream-K workspace on the default stream (pytorch#177573), which hung the first graph replay; the runtime works around it.","I built and published the FP8 and AWQ packages on Hugging Face as vllm-sr/d3-FP8 and vllm-sr/d3-W4A16-AWQ. On the public Decision Index 0.3 suite, FP8 scores 65.40 against 65.45 for d3, with 99.15% top-1 agreement over 312,320 questions; AWQ scores 65.00."],"tags":["Quantization","FP8","Serving"],"links":[{"label":"PR #5066","url":"https://github.com/vllm-project/semantic-router/pull/5066"},{"label":"d3-FP8","url":"https://huggingface.co/vllm-sr/d3-FP8"},{"label":"d3-W4A16-AWQ","url":"https://huggingface.co/vllm-sr/d3-W4A16-AWQ"},{"label":"pytorch#177573","url":"https://github.com/pytorch/pytorch/issues/177573"}],"featured":true,"breakdown":{"problem":"d3 takes 52.2 GB in BF16, and the model runtime expanded FP8 and int4 compressed-tensors packages to BF16 instead of loading them as stored.","fix":["Let the model runtime load FP8 (W8A8 FP8_DYNAMIC) and int4 (GPTQ, AWQ) compressed-tensors packages as stored (#5066, merged 18 Oct). FP8 activations are quantized per row in one Triton kernel and multiplied in `torch._scaled_mm`.","Handled the hardware paths: on MI300 the E4M3 bits are reinterpreted as e4m3fnuz with doubled scales; on Ada at 256 rows or more, under the approximate profile, the product is taken unscaled in FP32 and then scaled.","Fixed two bugs inside the PR: the FP8 activation kernel indexed rows in 32 bits and returned NaN past 2^31 elements (now 64-bit indexing), and on Ada, PyTorch before 2.12 zeroed the stream-K workspace on the default stream (pytorch#177573), which hung the first graph replay (now worked around).","Built and published the FP8 and AWQ packages on Hugging Face as vllm-sr/d3-FP8 and vllm-sr/d3-W4A16-AWQ."],"result":["On 14,031 test requests, FP8 keeps 99.08% of BF16's answers in 31.9 GB instead of 52.2 GB; AWQ int4 keeps 97.76% in about 16.5 GB.","On MI300X with d3's forward replayed as graphs, the FP8 median is 36.8 ms against 46.4 ms for BF16 (#4999; record corrected in #5270). Without graphs it is 80.9 against 67.1 ms, because per-layer host cost outweighs the smaller matmuls.","On the public Decision Index 0.3 suite, FP8 scores 65.40 against 65.45 for d3, with 99.15% top-1 agreement over 312,320 questions; AWQ scores 65.00."]}},{"id":"sr-decision3-serving","date":"2026-10-20","dateLabel":"Oct 13 – 20, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #4999 · #4998 · #5009 · #5270","status":"4 merged upstream","title":"Serving Decision 3.0 as CUDA graphs","headline":"Decision 3.0 forwards as CUDA graphs: one question on d3-lite 29.4 → 7.3 ms on an RTX 4070 Ti SUPER, same answers.","body":["Decision 3.0 (d3) is the project's new family of decision models; the largest is 27B. Its forwards are noncausal, and the runtime ran every one of them eagerly. #4999 (merged) builds the noncausal attention mask from padding information the host already has and captures text forwards as CUDA graphs per exact shape: a one-question request on d3-lite (0.8B) went from 29.4 to 7.3 ms on an RTX 4070 Ti SUPER, with the same answers. Two graph-capture bugs came out of that work, both merged: every capture left a BLAS workspace behind (#4998), and the graph budget counted cache that large eager batches had grown, so a batching server stopped capturing after 16 graphs (#5009). #5270 (merged) corrected the latency record the model cards cite."],"tags":["CUDA graphs","Serving","Latency"],"links":[{"label":"PR #4999","url":"https://github.com/vllm-project/semantic-router/pull/4999"},{"label":"PR #4998","url":"https://github.com/vllm-project/semantic-router/pull/4998"},{"label":"PR #5009","url":"https://github.com/vllm-project/semantic-router/pull/5009"},{"label":"PR #5270","url":"https://github.com/vllm-project/semantic-router/pull/5270"}],"featured":true,"breakdown":{"problem":"Decision 3.0 (d3) is the project's new family of decision models, the largest 27B. Its forwards are noncausal, and the runtime ran every one of them eagerly.","fix":["Built the noncausal attention mask from padding information the host already has, and captured text forwards as CUDA graphs per exact shape (#4999).","Fixed a BLAS workspace that every capture left behind (#4998).","Fixed the graph budget counting cache that large eager batches had grown, which made a batching server stop capturing after 16 graphs (#5009).","Corrected the latency record the model cards cite (#5270)."],"result":["A one-question request on d3-lite (0.8B) went from 29.4 to 7.3 ms on an RTX 4070 Ti SUPER, with the same answers.","4 PRs merged upstream."]}},{"id":"sr-runtime-robustness","date":"2026-10-18","dateLabel":"Oct 17 – 20, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #5184 · #5183 · #5181 · #5179 · #5180 · #5185 · #5190","status":"7 merged · 4 open","title":"Model-runtime failures found bringing up the 27B model","headline":"Loading d3 on MI300X peaked at 49 GiB of host memory, now 6 GiB; an out-of-memory forward answers 413 or 429 instead of 500.","body":["While bringing the d3 packages up at three dtypes, and re-checking main afterwards, I found and fixed these runtime failures; the fixes merged on 17–18 Oct. #5184 stages a GPU model's checkpoint per tensor instead of as a full host copy: loading d3 on MI300X peaked at 49 GiB of anonymous host memory before and 6 GiB after. #5183 answers a forward that runs out of device memory with 413 when it ran alone and 429 when it shared the forward, instead of 500, and a later request no longer fails after it. #5181 releases cached GPU memory before a failed load is retried. Smaller fixes: kernel choices are pinned only at the published precision (#5179), media workers are waited for when the runtime stops (#5180), d3's CPU answers are recorded for AVX-512 hosts with native BF16 (#5185), and FP32 inputs on GPUs take the reference varlen attention (#5190). Still open: #5220 answers a full worker pool as 429 on the native gateway too, and #5274, #5286 and #5001 follow up on the smaller d3 packages, media preprocessing and padding."],"tags":["Reliability","Memory","Serving"],"links":[{"label":"PR #5184","url":"https://github.com/vllm-project/semantic-router/pull/5184"},{"label":"PR #5183","url":"https://github.com/vllm-project/semantic-router/pull/5183"},{"label":"PR #5181","url":"https://github.com/vllm-project/semantic-router/pull/5181"},{"label":"PR #5220","url":"https://github.com/vllm-project/semantic-router/pull/5220"}],"featured":true,"breakdown":{"problem":"Bringing up the d3 packages at three dtypes exposed runtime failures: loading d3 on MI300X peaked at 49 GiB of host memory, and a forward that ran out of device memory returned 500.","fix":["Staged a GPU model's checkpoint per tensor instead of as a full host copy (#5184).","Answered an out-of-memory forward with 413 when it ran alone and 429 when it shared the forward, instead of 500, so a later request no longer fails after it (#5183).","Released cached GPU memory before a failed load is retried (#5181).","Smaller fixes: kernel choices pinned only at the published precision (#5179), media workers waited for when the runtime stops (#5180), d3's CPU answers recorded for AVX-512 hosts with native BF16 (#5185), and FP32 inputs on GPUs sent to the reference varlen attention (#5190)."],"result":["Loading d3 on MI300X peaked at 49 GiB of anonymous host memory before #5184 and 6 GiB after.","An out-of-memory forward now answers 413 or 429 instead of 500.","7 merged on 17–18 Oct; 4 open."],"next":"Still open: #5220 answers a full worker pool as 429 on the native gateway too, and #5274, #5286 and #5001 follow up on the smaller d3 packages, media preprocessing and padding."}},{"id":"sr-kv-mapper","date":"2026-10-04","dateLabel":"Oct 1 – 4, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #4497 · #4526 · epic #2976","status":"1 merged · 1 in review","title":"Reusing a 14B model's KV cache in a 32B model","body":["The epic maps Qwen3-14B's KV cache into Qwen3-32B so the larger model can skip prefill. The first mapper was a ridge regression fitted to the cache values. #4497 (merged, +808 / −3) adds a rank-16 correction trained so the 32B model predicts the same next tokens it would from its own cache: KL against that falls from 0.088 to 0.036 on chat and from 0.142 to 0.082 on plain text. Refitting the ridge map on the same data does not get there, so the gain comes from the objective. I also verified the artifact through another contributor's connector PR (KL 0.055 vs 0.120 for v1). #4526 adds the KL evaluation, because HellaSwag cannot tell mappers apart. The latency gain in serving is not measured yet."],"tags":["KV cache","Distillation","Serving"],"links":[{"label":"PR #4497","url":"https://github.com/vllm-project/semantic-router/pull/4497"},{"label":"PR #4526","url":"https://github.com/vllm-project/semantic-router/pull/4526"},{"label":"Epic #2976","url":"https://github.com/vllm-project/semantic-router/issues/2976"}],"headline":"A 32B model reusing a 14B model's KV cache: KL to its own cache 0.088 → 0.036 on chat with a distilled rank-16 correction.","featured":true,"breakdown":{"problem":"The epic maps Qwen3-14B's KV cache into Qwen3-32B so the larger model can skip prefill. The first mapper was a ridge regression fitted to the cache values.","fix":["Added a rank-16 correction trained so the 32B model predicts the same next tokens it would from its own cache (#4497, +808 / −3).","Verified the artifact through another contributor's connector PR.","Added the KL evaluation, because HellaSwag cannot tell mappers apart (#4526)."],"result":["KL against the 32B model's own cache falls from 0.088 to 0.036 on chat and from 0.142 to 0.082 on plain text. Refitting the ridge map on the same data does not get there, so the gain comes from the objective.","Through the connector PR, KL is 0.055 vs 0.120 for v1.","1 merged (#4497), 1 in review."],"next":"One PR is in review, and the latency gain in serving is not measured yet."}},{"id":"sr-ranking","date":"2026-09-21","dateLabel":"Aug 31 – Sep 21, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"Proposal #3227 · 8 PRs · closes #3081","status":"9 merged upstream","title":"Making \"which route wins\" a contract instead of an accident","body":["Decision selection had three problems. It mixed policy rules into the confidence that picks the winner. It averaged over leaves, so adding evidence could lower a score: mean(.90, .90) beat mean(.90, .90, .88). And it ignored `routing.strategy` whenever a tier was set. On a maintainer's roadmap issue I wrote the contract: a rule is either policy or evidence, and confidence ranks only a pool whose members each report one score of one kind. Anything else falls back to priority. Eight PRs merged over two days (+1,513 / −183) collapsed three orderings into one comparator without changing the winner in any shipped config, and the ranking trace now appears in the eval API and the replay record. A follow-up calibration (#4550, in review) brings ECE on held-out domain scores from 0.263 to 0.033."],"tags":["Routing","Design","Merged upstream"],"links":[{"label":"Proposal #3227","url":"https://github.com/vllm-project/semantic-router/pull/3227"},{"label":"Issue #3081","url":"https://github.com/vllm-project/semantic-router/issues/3081"},{"label":"PR #4550","url":"https://github.com/vllm-project/semantic-router/pull/4550"}],"headline":"Route selection could rank a decision lower for having more evidence. Wrote the contract and shipped it in 8 merged PRs with no winner change in any shipped config.","featured":true,"breakdown":{"problem":"Route selection could rank a decision lower for having more evidence: mean(.90, .90) beat mean(.90, .90, .88).","cause":"Selection mixed policy rules into the confidence that picks the winner, averaged over leaves, and ignored `routing.strategy` whenever a tier was set.","fix":["Wrote the contract on a maintainer's roadmap issue (Proposal #3227): a rule is either policy or evidence, and confidence ranks only a pool whose members each report one score of one kind. Anything else falls back to priority.","Collapsed three orderings into one comparator in eight PRs merged over two days (+1,513 / −183).","Added the ranking trace to the eval API and the replay record."],"result":["The winner did not change in any shipped config.","9 merged upstream; closes #3081."],"next":"A follow-up calibration (#4550, in review) brings ECE on held-out domain scores from 0.263 to 0.033."}},{"id":"hf-47017","date":"2026-07-02","dateLabel":"Jul 2, 2026","kind":"pr","project":"huggingface/transformers","group":"transformers","ref":"PR #47017","status":"Open","title":"Route byte-level llama tokenizers to TokenizersBackend","body":["Byte-level checkpoints that declare `LlamaTokenizerFast`, DeepSeek-R1-Distill-Llama among them, decode with every space lost. The result is garbled output from a vLLM endpoint and no error. The accepted fix was a repo-id allowlist; this PR routes on whether `tokenizer.json` is actually byte-level. When asked to extend the allowlist instead, I swept the 50,000 most-downloaded llama repos. 1,866 repos across 929 orgs are affected, and the top 30 all return \"Helloworld.I'manAI.\" without a single log line. Users were still reporting it in October."],"tags":["Tokenizers","vLLM serving","Python"],"links":[{"label":"PR #47017","url":"https://github.com/huggingface/transformers/pull/47017"},{"label":"Issue #45488","url":"https://github.com/huggingface/transformers/issues/45488"}],"headline":"Byte-level Llama checkpoints decode with every space lost, silently. Measured the reach on the Hub: 1,866 repos across 929 orgs.","featured":true,"breakdown":{"problem":"Byte-level checkpoints that declare `LlamaTokenizerFast`, DeepSeek-R1-Distill-Llama among them, decode with every space lost. A vLLM endpoint returns garbled output and no error.","fix":["Routed tokenizers on whether `tokenizer.json` is actually byte-level, instead of the accepted repo-id allowlist (#47017).","When asked to extend the allowlist instead, swept the 50,000 most-downloaded llama repos."],"result":["1,866 repos across 929 orgs are affected, and the top 30 all return \"Helloworld.I'manAI.\" without a single log line.","Users were still reporting it in October."],"next":"The PR is open."}},{"id":"sr-decision-followups","date":"2026-10-09","dateLabel":"Oct 5 – 9, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #4778 · #4573 · issue #4572","status":"1 merged · 1 in review","title":"Decision questions that read the conversation, and answers checked against the question","body":["Decision signals read only the current user turn, so a follow-up like \"Rewrite your previous answer in fewer words\" routed on almost nothing. #4778 (in review) lets a decision rule read up to eight earlier user turns within a fixed 1,760-character budget, leaving out assistant turns, which scored no better at about twice the tokens. On MT-Bench's 80 two-turn conversations, category accuracy on the second turn rises from 41.3% to 77.5% on the 0.6B and from 51.2% to 87.5% on Vela 2.0 9B. #4573 (merged) closes the other side: the decision signal acted on any answer an attached engine returned, including a choice its own distribution contradicted or an option the question never declared. Such answers are now refused as `invalid_model_output` and the selector falls back; all 64 answers from the four built-in families pass unchanged."],"tags":["Routing","Multi-turn","Correctness"],"links":[{"label":"PR #4778","url":"https://github.com/vllm-project/semantic-router/pull/4778"},{"label":"PR #4573","url":"https://github.com/vllm-project/semantic-router/pull/4573"},{"label":"Issue #4572","url":"https://github.com/vllm-project/semantic-router/issues/4572"}],"headline":"Follow-up turns routed on almost nothing: reading prior user turns lifts MT-Bench category accuracy 41.3% → 77.5% on a 0.6B model.","breakdown":{"problem":"Decision signals read only the current user turn, so a follow-up like \"Rewrite your previous answer in fewer words\" routed on almost nothing.","fix":["Let a decision rule read up to eight earlier user turns within a fixed 1,760-character budget (#4778). Assistant turns are left out, since they scored no better at about twice the tokens.","Refused engine answers that the engine's own distribution contradicted or that the question never declared, as `invalid_model_output`, so the selector falls back (#4573)."],"result":["On MT-Bench's 80 two-turn conversations, second-turn category accuracy rises from 41.3% to 77.5% on the 0.6B and from 51.2% to 87.5% on Vela 2.0 9B.","#4573 is merged, and all 64 answers from the four built-in families pass unchanged."],"next":"#4778 is in review."}},{"id":"sr-decision-suite","date":"2026-10-04","dateLabel":"Sep 28 – Oct 4, 2026","kind":"analysis","project":"vllm-project/semantic-router","group":"semantic-router","ref":"Issue #4305 · #4306","status":"Completed · published","title":"One 0.6B decision model vs. eight per-signal encoders, on the same data","body":["The router runs a separate encoder per signal, and the question was whether one fine-tuned model could replace them. I built a corpus-matched suite (10 signals, 139 sources) that rebuilds byte for byte from an empty directory. Every model was trained on the same rows and compared with a grouped paired bootstrap and Benjamini–Hochberg correction. Decision-1.0-Route-0.6B wins 21, ties 30 and loses 6 against encoders on the same training budget, with the largest gains on held-out hallucination data (+0.06 to 0.19 AUC). The costs are reported alongside: 2.2× slower on CPU (409 vs 189 ms), faster on MI300X (47 vs 82 ms), and still weaker than the shipped guard on most safety sets."],"tags":["Evaluation","Serving cost","Datasets"],"links":[{"label":"Issue #4306","url":"https://github.com/vllm-project/semantic-router/issues/4306"},{"label":"Dataset","url":"https://huggingface.co/datasets/vllm-sr/router-signal-suite"},{"label":"Model","url":"https://huggingface.co/vllm-sr/Decision-1.0-Route-0.6B"}],"headline":"One 0.6B model vs. eight per-signal encoders on identical data: 21 wins, 30 ties, 6 losses, at 2.2× the CPU cost.","breakdown":{"problem":"The router runs a separate encoder per signal. The question was whether one fine-tuned model could replace them.","fix":["Built a corpus-matched suite (10 signals, 139 sources) that rebuilds byte for byte from an empty directory.","Trained every model on the same rows and compared them with a grouped paired bootstrap and Benjamini–Hochberg correction."],"result":["Decision-1.0-Route-0.6B wins 21, ties 30 and loses 6 against encoders on the same training budget, with the largest gains on held-out hallucination data (+0.06 to 0.19 AUC).","It is 2.2× slower on CPU (409 vs 189 ms), faster on MI300X (47 vs 82 ms), and still weaker than the shipped guard on most safety sets.","Completed and published (issue #4305, #4306)."]}},{"id":"sr-eval-integrity","date":"2026-10-07","dateLabel":"Aug 31 – Oct 7, 2026","kind":"analysis","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #3247 · #3248 · #4307 · #4308 · #4309 · #4565","status":"6 merged upstream","title":"Benchmarks that were measuring the training data","body":["The eval registry and the artifact listed labels in different orders, so four of fourteen classes were scored against the wrong logit. That dropped MMLU-Pro from 0.9445 to 0.6746, and no metric flagged it. I added versioned manifests with cross-reference validation (#3247) and a calibration-aware baseline harness (#3248). The same checks later found that Vela Domain had trained on the MMLU questions behind 6,810 of MMLU-Pro's 12,032 test rows: it scored 0.769 on those against 0.631 on the rest. They also showed that the fact-check split can be solved at AUC 0.999 from the source column alone. The baselines now score only rows the models never saw (#4308, #4309), and fact-check is scored on a corpus-matched split where the source predicts nothing (AUC 0.500, down from 0.999). On it the ranking flips: Vela FactCheck reaches 0.696 accuracy against 0.573 for the checkpoint the old split put first (#4565)."],"tags":["Evaluation","Provenance","Merged upstream"],"links":[{"label":"PR #3247","url":"https://github.com/vllm-project/semantic-router/pull/3247"},{"label":"PR #4308","url":"https://github.com/vllm-project/semantic-router/pull/4308"},{"label":"PR #4565","url":"https://github.com/vllm-project/semantic-router/pull/4565"},{"label":"Issue #4301","url":"https://github.com/vllm-project/semantic-router/issues/4301"}],"headline":"A label-order mismatch cost 27 accuracy points unseen; a domain benchmark was scoring the model's own training data.","breakdown":{"problem":"Benchmarks gave wrong scores with no metric flagging it: a label-order mismatch dropped MMLU-Pro from 0.9445 to 0.6746, and a domain benchmark was scoring the model's own training data.","cause":"The eval registry and the artifact listed labels in different orders, so four of fourteen classes were scored against the wrong logit.","fix":["Added versioned manifests with cross-reference validation (#3247) and a calibration-aware baseline harness (#3248).","The same checks later found that Vela Domain had trained on the MMLU questions behind 6,810 of MMLU-Pro's 12,032 test rows, and that the fact-check split can be solved at AUC 0.999 from the source column alone.","Made the baselines score only rows the models never saw (#4308, #4309), and moved fact-check to a corpus-matched split where the source predicts nothing (#4565)."],"result":["Vela Domain had scored 0.769 on the rows it trained on against 0.631 on the rest.","On the new fact-check split the source reaches AUC 0.500, down from 0.999, and the ranking flips: Vela FactCheck reaches 0.696 accuracy against 0.573 for the checkpoint the old split put first.","6 merged upstream."]}},{"id":"sr-jailbreak-retrain","date":"2026-10-04","dateLabel":"Sep 14 – Oct 4, 2026","kind":"analysis","project":"vllm-project/semantic-router","group":"semantic-router","ref":"Issue #3787 · PR #3924 · #3945","status":"2 merged · retrain in progress","title":"The jailbreak detector answers a different question than the router asks","body":["The prompt-injection rule asks whether a prompt attacks the system, but the shipped detector was trained to say whether content is harmful. 82% of its test positives carry no attack marker, and what it learned was mostly prompt length. I built a decontaminated 6,646-row set labelled for both questions. #3924 made the metric honest: the shipped model reads AUC 0.81, yet at a 1% false-positive budget it catches 3.9% of attacks. An attack-labelled candidate reaches AUC 0.96 in review. #3945 reports which window decided a guard score, which is what explained benign documents being blocked at 0.95 on their second window."],"tags":["Guardrails","Evaluation","Research"],"links":[{"label":"Issue #3787","url":"https://github.com/vllm-project/semantic-router/issues/3787"},{"label":"PR #3924","url":"https://github.com/vllm-project/semantic-router/pull/3924"},{"label":"PR #3945","url":"https://github.com/vllm-project/semantic-router/pull/3945"}],"headline":"The shipped detector catches 3.9% of attacks at a 1% false-positive budget; an attack-labelled retrain reaches AUC 0.96.","breakdown":{"problem":"The prompt-injection rule asks whether a prompt attacks the system, but the shipped detector was trained to say whether content is harmful.","cause":"82% of the detector's test positives carry no attack marker, and what it learned was mostly prompt length.","fix":["Built a decontaminated 6,646-row set labelled for both questions.","Made the metric honest (#3924).","Reported which window decided a guard score (#3945), which explained benign documents being blocked at 0.95 on their second window."],"result":["The shipped model reads AUC 0.81, yet at a 1% false-positive budget it catches 3.9% of attacks.","An attack-labelled candidate reaches AUC 0.96.","2 merged."],"next":"The retrain is in progress; the attack-labelled candidate is in review."}},{"id":"sr-replay-dataset","date":"2026-10-13","dateLabel":"Sep 18 – Oct 13, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #3912 · #3915 · #3998 · #4087 · #4089 · #4123","status":"6 merged upstream","title":"Reproducible datasets from shadow traffic, for comparing routes","body":["For a maintainer roadmap epic, I turned replayed traffic into a dataset that two routes can be compared on reproducibly (+3,336 / −14, merged). The manifest stores only hashes, and its digest is its identity. Splits follow a seeded hash, so record order cannot change them. A truncated or failed observation is excluded under a counted reason instead of being half-included, and no single decision can dominate the set. Blinded pairwise judge tasks sit on top, served from the dataset, and #4123 publishes a validated dataset and its judgments by digest; the judge-task route from #4116 landed with it. Building it showed that every shadow-dispatch outcome had been silently dropped since an earlier change deferred the replay start (#3914, fixed in #3915)."],"tags":["Data pipeline","Reproducibility","Merged upstream"],"links":[{"label":"PR #3912","url":"https://github.com/vllm-project/semantic-router/pull/3912"},{"label":"PR #3915","url":"https://github.com/vllm-project/semantic-router/pull/3915"},{"label":"PR #4089","url":"https://github.com/vllm-project/semantic-router/pull/4089"},{"label":"PR #4123","url":"https://github.com/vllm-project/semantic-router/pull/4123"}],"headline":"Hash-addressed, reproducible comparison datasets from shadow traffic, with a silent data-loss bug fixed on the way.","breakdown":{"problem":"A maintainer roadmap epic needed a dataset, built from replayed shadow traffic, that two routes can be compared on reproducibly.","fix":["Built the dataset (+3,336 / −14): the manifest stores only hashes, its digest is its identity, and splits follow a seeded hash so record order cannot change them.","Excluded truncated or failed observations under a counted reason instead of half-including them, and kept any single decision from dominating the set.","Added blinded pairwise judge tasks served from the dataset; #4123 publishes a validated dataset and its judgments by digest, and the judge-task route from #4116 landed with it.","Fixed every shadow-dispatch outcome being silently dropped since an earlier change deferred the replay start (#3914, fixed in #3915)."],"result":["Routes can be compared on a hash-addressed, reproducible dataset.","A silent data-loss bug was found and fixed along the way (#3915).","6 merged upstream."]}},{"id":"sr-response-stage","date":"2026-09-11","dateLabel":"Sep 2 – 11, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #3335 · #3531 · epic #3272","status":"3 merged upstream","title":"Response detection rebuilt as a composable routing signal","body":["Response jailbreak and hallucination detection were plugins that did their own detecting, so neither could be combined with request-side signals in a routing decision. I took the maintainer's epic and turned both into response-direction signals, scored once per response: #3335 (+2,597 / −230, 45 files) and #3531 (+1,919 / −332, 64 files), both merged. A recipe can now say \"route to a stronger model when the answer looks wrong\"."],"tags":["Architecture","Guardrails","Merged upstream"],"links":[{"label":"PR #3335","url":"https://github.com/vllm-project/semantic-router/pull/3335"},{"label":"PR #3531","url":"https://github.com/vllm-project/semantic-router/pull/3531"},{"label":"PR #3689","url":"https://github.com/vllm-project/semantic-router/pull/3689"},{"label":"Epic #3272","url":"https://github.com/vllm-project/semantic-router/issues/3272"}],"headline":"Response jailbreak and hallucination checks rebuilt as routing signals: +4.5k lines across 109 files, merged.","breakdown":{"problem":"Response jailbreak and hallucination detection were plugins that did their own detecting, so neither could be combined with request-side signals in a routing decision.","fix":["Took the maintainer's epic (#3272) and turned both into response-direction signals, scored once per response.","Shipped it in #3335 (+2,597 / −230, 45 files) and #3531 (+1,919 / −332, 64 files)."],"result":["A recipe can now say \"route to a stronger model when the answer looks wrong\".","+4.5k lines across 109 files, merged; 3 merged upstream in total."]}},{"id":"sr-silent-wrongness","date":"2026-10-13","dateLabel":"Aug 26 – Oct 13, 2026","kind":"pr","project":"vllm-project/semantic-router","group":"semantic-router","ref":"PR #3027 · #3141 · #3147 · #3080 · #3568 · #3694 · #4777","status":"7 merged upstream","title":"PII scores that were confidently wrong","body":["These are values that look reasonable, get compared against a threshold, and decide something. Entity confidence was a running fold rather than a mean, so whether PII got masked depended on how many subwords a name split into (#3027). Emails and domains were undetectable at any threshold, because the decoder required a `B-` tag the model never emits; EMAIL_ADDRESS recall went from 0/24 to 21/24 (#3141). In non-ASCII text, spans were byte offsets reported as character positions (#3147). Later I found that a value repeated in a prompt was masked only on its first copy: the second \"John Smith\" scored 0.49 against a 0.75 threshold and went to the provider in clear text. #4777 (merged) masks every whole-word copy, and the expansion stays fast: 20,000 copies within 2 s, where the old path took 16 s at 3,200."],"tags":["PII","Correctness","Merged upstream"],"links":[{"label":"PR #3027","url":"https://github.com/vllm-project/semantic-router/pull/3027"},{"label":"PR #3141","url":"https://github.com/vllm-project/semantic-router/pull/3141"},{"label":"PR #3147","url":"https://github.com/vllm-project/semantic-router/pull/3147"},{"label":"PR #3080","url":"https://github.com/vllm-project/semantic-router/pull/3080"},{"label":"PR #3568","url":"https://github.com/vllm-project/semantic-router/pull/3568"},{"label":"PR #3694","url":"https://github.com/vllm-project/semantic-router/pull/3694"},{"label":"PR #4777","url":"https://github.com/vllm-project/semantic-router/pull/4777"}],"headline":"PII masking decided by miscomputed scores; email recall 0/24 → 21/24.","breakdown":{"problem":"PII masking was decided by scores that looked reasonable but were miscomputed. Emails could not be detected, and a repeated name could reach the provider in clear text.","fix":["Fixed entity confidence, which was a running fold rather than a mean, so masking depended on how many subwords a name split into (#3027).","Fixed the decoder, which required a `B-` tag the model never emits, leaving emails and domains undetectable at any threshold (#3141).","Fixed non-ASCII spans, which were byte offsets reported as character positions (#3147).","Masked every whole-word copy of a repeated value, not only the first (#4777)."],"result":["EMAIL_ADDRESS recall went from 0/24 to 21/24 (#3141).","A second \"John Smith\" that scored 0.49 against a 0.75 threshold is now masked, and the expansion handles 20,000 copies within 2 s, where the old path took 16 s at 3,200.","7 merged upstream."]}},{"id":"sr-router-wg","date":"2026-10-21","dateLabel":"Aug 26 – Oct 21, 2026","kind":"analysis","project":"vllm-project/semantic-router","group":"semantic-router","ref":"Committer · WG lead","status":"Committer since Sep 17","title":"Committer and working-group lead","body":["I self-nominated to the Router Models & Inference Runtime working group on 26 August and was made a committer and WG lead on 17 September. I have opened 98 PRs to the project and 83 are merged; most close an issue I filed first with a reproduction. As lead I scope incoming work for the maintainers: the phasing of the built-in model runtime (#4496), the first slice of the Decision runtime follow-ups (#4472) and the lifecycle and release gaps it left (#4537), all five of which my own PRs closed, and per-area dependency projects for the training code (#3932). I also review incoming work (9 PRs from other contributors so far): for the cross-model KV connector (#4514, #4513) I asked for a shared validator and a guard that keeps approximate KV out of the prefix cache, and both went in; for serving an external NLI model as a decision model (#4765) I ran the GPU parity check."],"tags":["Committer","Working-group lead","Upstream process"],"links":[{"label":"Committers & team","url":"https://vllm-sr.ai/community/team"},{"label":"WG roster","url":"https://vllm-sr.ai/community/work-groups"},{"label":"Merged PRs","url":"https://github.com/vllm-project/semantic-router/pulls?q=is%3Apr+author%3Asubin9+is%3Amerged"}],"headline":"Committer and WG lead since Sep 17; 83 of 98 PRs merged.","breakdown":{"problem":"What the role covers: committer and lead of the Router Models & Inference Runtime working group, scoping incoming work for the maintainers and reviewing other contributors' PRs.","fix":["Self-nominated to the working group on 26 August; made a committer and WG lead on 17 September.","Scoped the phasing of the built-in model runtime (#4496), the first slice of the Decision runtime follow-ups (#4472) and the lifecycle and release gaps it left (#4537), plus per-area dependency projects for the training code (#3932).","Reviewed 9 PRs from other contributors: asked for a shared validator and a guard that keeps approximate KV out of the prefix cache on the cross-model KV connector (#4514, #4513), and ran the GPU parity check for serving an external NLI model as a decision model (#4765)."],"result":["83 of 98 PRs opened to the project are merged; most close an issue filed first with a reproduction.","The author's own PRs closed all five items from the runtime scoping work.","The shared validator and the prefix-cache guard both went in."]}},{"id":"mrs-2356","date":"2026-07-26","dateLabel":"Jul 26, 2026","kind":"pr","project":"EricLBuehler/mistral.rs","group":"mistral-rs","ref":"PR #2356","status":"Open","title":"feat(models): Support PaddleOCR-VL vision model (rework of #2320)","body":["End-to-end PaddleOCR-VL support: SigLIP NaViT vision encoder, projector and ERNIE-4.5 decoder with 2D RoPE / mRoPE (+3,780 / −34). It reaches token-for-token parity with transformers under greedy decoding across nine task types, on both CPU-f32 and GPU-bf16."],"tags":["VLM","Rust","Model support"],"links":[{"label":"PR #2356","url":"https://github.com/EricLBuehler/mistral.rs/pull/2356"},{"label":"Issue #2128","url":"https://github.com/EricLBuehler/mistral.rs/issues/2128"}],"breakdown":{"problem":"There was no end-to-end support for the PaddleOCR-VL vision model.","fix":["Added the SigLIP NaViT vision encoder, projector and ERNIE-4.5 decoder with 2D RoPE / mRoPE (#2356, +3,780 / −34)."],"result":["Token-for-token parity with transformers under greedy decoding across nine task types, on both CPU-f32 and GPU-bf16."],"next":"The PR is open."}},{"id":"mrs-2319","date":"2026-07-08","dateLabel":"Jul 8, 2026","kind":"pr","project":"EricLBuehler/mistral.rs","group":"mistral-rs","ref":"PR #2319 · llguidance #361","status":"Open","title":"fix(llg): honor tokenizer special flag in toktrie detok","body":["toktrie marked any `<...>`-shaped token as special and overrode the tokenizer's own flag, so content tokens disappeared on decode. PaddleOCR-VL tables, for example, collapsed into run-on text. I filed it upstream as llguidance #361, which led to a fix there, and this PR fixes it engine-side (+192)."],"tags":["Detokenization","Rust","Inference engine"],"links":[{"label":"PR #2319","url":"https://github.com/EricLBuehler/mistral.rs/pull/2319"},{"label":"llguidance #361","url":"https://github.com/guidance-ai/llguidance/issues/361"}],"breakdown":{"problem":"Content tokens disappeared on decode. PaddleOCR-VL tables, for example, collapsed into run-on text.","cause":"toktrie marked any `<...>`-shaped token as special and overrode the tokenizer's own flag.","fix":["Filed it upstream as llguidance #361, which led to a fix there.","Fixed it engine-side in this PR (#2319, +192)."],"result":["llguidance fixed it upstream after #361."],"next":"The engine-side PR is open."}},{"id":"mrs-2338","date":"2026-07-11","dateLabel":"Jul 11, 2026 · merged Aug 29","kind":"pr","project":"EricLBuehler/mistral.rs","group":"mistral-rs","ref":"PR #2338 · issue #2336","status":"Merged upstream","title":"fix(webui): send chat images as data URLs","body":["Images sent from the built-in web UI failed with HTTP 500, while the same image through the API worked. The UI was sending back a server-relative path that the media policy rejects; it now sends a data URL. Merged by the maintainer."],"tags":["Web UI","Rust","Merged upstream"],"links":[{"label":"PR #2338","url":"https://github.com/EricLBuehler/mistral.rs/pull/2338"},{"label":"Issue #2336","url":"https://github.com/EricLBuehler/mistral.rs/issues/2336"}],"breakdown":{"problem":"Images sent from the built-in web UI failed with HTTP 500, while the same image through the API worked.","cause":"The UI was sending back a server-relative path that the media policy rejects.","fix":["Made the UI send a data URL instead (#2338, issue #2336)."],"result":["Merged by the maintainer."]}},{"id":"mrs-2337","date":"2026-07-11","dateLabel":"Jul 11, 2026","kind":"pr","project":"EricLBuehler/mistral.rs","group":"mistral-rs","ref":"PR #2337 · issue #2335","status":"Open","title":"fix(mllama): CUDA image-inference crash, eager-only paged flag, vision head config","body":["Llama-3.2-11B-Vision failed with PagedAttention, the default, and crashed on CUDA without it, so there was no working path. The fix marks the model eager-only and corrects its vision-head config."],"tags":["CUDA","Rust","Bug fix"],"links":[{"label":"PR #2337","url":"https://github.com/EricLBuehler/mistral.rs/pull/2337"},{"label":"Issue #2335","url":"https://github.com/EricLBuehler/mistral.rs/issues/2335"}],"breakdown":{"problem":"Llama-3.2-11B-Vision failed with PagedAttention, the default, and crashed on CUDA without it, so there was no working path.","fix":["Marked the model eager-only and corrected its vision-head config (#2337, issue #2335)."],"result":["The fix is up as #2337."],"next":"The PR is open."}},{"id":"vllm-60211","date":"2026-10-06","dateLabel":"Oct 6, 2026","kind":"pr","project":"vllm-project/vllm","group":"vllm","ref":"PR #60211 · issue #60197","status":"Open","title":"LoRA adapters with multi-layer classification heads (RoBERTa, XLM-RoBERTa)","body":["PEFT sequence-classification adapters for RoBERTa failed to load in vLLM (`classifier.dense.bias is unsupported LoRA weight`, issue #60197), and 29 of the 40 most popular `roberta-base` adapters save their head that way. vLLM's LoRA code assumed a single-Linear head, but RoBERTa's is `dense → tanh → out_proj`, so a LoRA update on `classifier.dense` was dropped without an error (max abs diff 0.45). Models now declare their head linears in order, and RoBERTa and XLM-RoBERTa opt in. On an RTX 4070 Ti SUPER, vLLM in fp16 matches transformers + PEFT in fp32 within 4.5e-3 on real adapters. A reviewer would rather keep the scope narrow, so the PR waits for the code owner."],"tags":["LoRA","Rerankers","Python"],"links":[{"label":"PR #60211","url":"https://github.com/vllm-project/vllm/pull/60211"},{"label":"Issue #60197","url":"https://github.com/vllm-project/vllm/issues/60197"}],"breakdown":{"problem":"PEFT sequence-classification adapters for RoBERTa failed to load in vLLM (`classifier.dense.bias is unsupported LoRA weight`, issue #60197). 29 of the 40 most popular `roberta-base` adapters save their head that way.","cause":"vLLM's LoRA code assumed a single-Linear head, but RoBERTa's is `dense → tanh → out_proj`, so a LoRA update on `classifier.dense` was dropped without an error (max abs diff 0.45).","fix":["Made models declare their head linears in order, and opted RoBERTa and XLM-RoBERTa in (#60211)."],"result":["On an RTX 4070 Ti SUPER, vLLM in fp16 matches transformers + PEFT in fp32 within 4.5e-3 on real adapters."],"next":"A reviewer would rather keep the scope narrow, so the PR waits for the code owner."}},{"id":"vllm-49855","date":"2026-07-26","dateLabel":"Jul 26, 2026","kind":"pr","project":"vllm-project/vllm","group":"vllm","ref":"PR #49855","status":"Open","title":"[Rust Frontend] Support --allowed-media-domains","body":["The Rust frontend had no equivalent of the Python frontend's media-domain allowlist, so a multimodal deployment could be pointed at arbitrary URLs. I added `--allowed-media-domains` (+286 / −10) and closed both bypasses that automated review found: `file:` URLs and redirects to hosts off the list."],"tags":["Rust","Multimodal","Serving frontend"],"links":[{"label":"PR #49855","url":"https://github.com/vllm-project/vllm/pull/49855"}],"breakdown":{"problem":"The Rust frontend had no equivalent of the Python frontend's media-domain allowlist, so a multimodal deployment could be pointed at arbitrary URLs.","fix":["Added `--allowed-media-domains` (#49855, +286 / −10).","Closed both bypasses that automated review found: `file:` URLs and redirects to hosts off the list."],"result":["The Rust frontend can restrict media to listed domains, with the `file:` and redirect bypasses closed."],"next":"The PR is open."}},{"id":"vllm-47633","date":"2026-07-05","dateLabel":"Jul 5, 2026","kind":"pr","project":"vllm-project/vllm","group":"vllm","ref":"PR #47633","status":"Open","title":"Emit spec-required gen_ai.operation.name on request spans","body":["vLLM request spans omitted `gen_ai.operation.name`, which the OpenTelemetry GenAI conventions require, so its traces dropped out of any GenAI dashboard keyed on the convention. Adds it along with `gen_ai.request.model`."],"tags":["Observability","OpenTelemetry","Python"],"links":[{"label":"PR #47633","url":"https://github.com/vllm-project/vllm/pull/47633"}],"breakdown":{"problem":"vLLM request spans omitted `gen_ai.operation.name`, which the OpenTelemetry GenAI conventions require, so vLLM traces dropped out of any GenAI dashboard keyed on the convention.","fix":["Added `gen_ai.operation.name` to request spans, along with `gen_ai.request.model` (#47633)."],"result":["Request spans carry the attribute the conventions require."],"next":"The PR is open."}},{"id":"ollama-19159","date":"2026-10-20","dateLabel":"Oct 20, 2026","kind":"pr","project":"ollama/ollama","group":"ollama","ref":"PR #19159","status":"Open","title":"Give noncausal decision models a whole-prompt batch","body":["Noncausal decision-model GGUFs (vllm-d3) got the default batch of 512, so any prompt over 512 tokens failed. They now get a batch that holds the whole prompt. No reviews yet."],"tags":["GGUF","Batching","Decision models"],"links":[{"label":"PR #19159","url":"https://github.com/ollama/ollama/pull/19159"}],"breakdown":{"problem":"Noncausal decision-model GGUFs (vllm-d3) failed on any prompt over 512 tokens.","cause":"They got the default batch of 512.","fix":["Gave them a batch that holds the whole prompt (#19159)."],"result":["PR open, no reviews yet."],"next":"Waiting for a first review."}},{"id":"exaone-14","date":"2026-08-12","dateLabel":"Aug 12, 2026","kind":"pr","project":"LGAI-EXAONE/EXAONE-4.0-32B","group":"exaone","ref":"HF PR #14","status":"Open","title":"chat_template: guard tool_call arguments against pre-serialized JSON","body":["The chat template JSON-encoded tool-call arguments unconditionally, so arguments already serialized by the OpenAI API were encoded twice and agent loops broke silently. I scoped the impact rather than assume it: vLLM is unaffected, while llama.cpp and Ollama embed the same template in their GGUF builds."],"tags":["chat_template","Tool calling","Jinja"],"links":[{"label":"HF discussions/14","url":"https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B/discussions/14"}],"breakdown":{"problem":"The chat template JSON-encoded tool-call arguments unconditionally, so arguments already serialized by the OpenAI API were encoded twice and agent loops broke silently.","fix":["Guarded tool_call arguments in the chat template against pre-serialized JSON (HF PR #14).","Scoped the impact rather than assuming it."],"result":["vLLM is unaffected, while llama.cpp and Ollama embed the same template in their GGUF builds."],"next":"HF PR #14 is open."}},{"id":"branch-sampler-headroom","date":"2026-07-31","dateLabel":"Jul 31, 2026","kind":"repo","project":"rebel-sbpark/vllm-rbln","group":"rbln","ref":"Fork branch","status":"Prototype · used as an FDE deployment reference","title":"Why a model that compiles cleanly runs out of NPU memory","body":["With automatic KV sizing, the runtime binary-searches the largest KV block count that fits. The sampler is compiled later, in another process, and only on TP rank 0, so the search never saw it: the model compiled and then either ran out of memory or silently fell back to CPU. On Qwen3-8B with TP8 at batch 96, 188 blocks sent 70 graphs to CPU and the engine died on a prefix-cache hit; 164 blocks gave 0 fallbacks. Greedy decoding hides only the speed loss; with real sampling, throughput fell from 750 to 189 tok/s. I prototyped a compile helper that reserves the sampler's budget before the search runs, and presented the memory-budget issue at a company seminar. FDEs refer to the helper when they set deployment specs."],"tags":["KV cache","Memory budgeting","Decoding strategy"],"links":[{"label":"Branch","url":"https://github.com/rebel-sbpark/vllm-rbln/tree/soobin/suggest-sampler-headroom"},{"label":"Diff","url":"https://github.com/rebel-sbpark/vllm-rbln/compare/main...soobin/suggest-sampler-headroom"}],"headline":"Models that compiled cleanly ran out of memory on TP rank 0, because the KV block search never saw the sampler graph.","breakdown":{"problem":"With automatic KV sizing, models that compiled cleanly then ran out of memory on TP rank 0 or silently fell back to CPU.","cause":"The runtime binary-searches the largest KV block count that fits, but the sampler is compiled later, in another process, and only on TP rank 0, so the search never saw it.","fix":["Prototyped a compile helper that reserves the sampler's budget before the search runs.","Presented the memory-budget issue at a company seminar."],"result":["On Qwen3-8B with TP8 at batch 96, 188 blocks sent 70 graphs to CPU and the engine died on a prefix-cache hit; 164 blocks gave 0 fallbacks.","Greedy decoding hides only the speed loss; with real sampling, throughput fell from 750 to 189 tok/s.","FDEs refer to the helper when they set deployment specs."]}},{"id":"optimumrbln-649","date":"2026-07-28","dateLabel":"Jul 28, 2026","kind":"issue","project":"RBLN-SW/optimum-rbln","group":"rbln","ref":"Issue #649","status":"Closed","title":"Empty last_hidden_state when prompt length is a multiple of prefill_chunk_size","body":["When the prompt length was an exact multiple of `prefill_chunk_size`, `last_hidden_state` came back empty, silently degrading embeddings for those lengths only. Fixed upstream two days later."],"tags":["optimum-rbln","Embeddings","Chunked prefill"],"links":[{"label":"Issue #649","url":"https://github.com/RBLN-SW/optimum-rbln/issues/649"}],"breakdown":{"problem":"When the prompt length was an exact multiple of `prefill_chunk_size`, `last_hidden_state` came back empty, silently degrading embeddings for those lengths only.","fix":["Reported the bug (issue #649)."],"result":["Fixed upstream two days later; the issue is closed."]}},{"id":"vllmrbln-922","date":"2026-08-13","dateLabel":"Aug 13, 2026","kind":"issue","project":"RBLN-SW/vllm-rbln","group":"rbln","ref":"Issue #922","status":"Open","title":"KV-cache events are never published on the optimum-rbln path","body":["With `--kv-events-config` on the optimum-rbln path no KV-cache events are ever published, and anything built on them, such as cache-hit accounting, routing or autoscaling, reads a silent zero."],"tags":["vllm-rbln","KV cache","Observability"],"links":[{"label":"Issue #922","url":"https://github.com/RBLN-SW/vllm-rbln/issues/922"}],"breakdown":{"problem":"With `--kv-events-config` on the optimum-rbln path, no KV-cache events are ever published.","fix":["Reported the bug (issue #922)."],"result":["Anything built on these events, such as cache-hit accounting, routing or autoscaling, reads a silent zero."],"next":"Issue #922 is open."}},{"id":"repo-paddleocr-vl-rbln","date":"2026-07-21","dateLabel":"Jul 21, 2026","kind":"repo","project":"rebel-sbpark/paddleocr-vl-rbln","group":"rbln","ref":"Repository","status":"Public","title":"paddleocr-vl-rbln — PaddleOCR-VL recognition VLM on the NPU","body":["An optimum-rbln model class that runs the PaddleOCR-VL VLM on RBLN NPUs, with vision patch counts bucketed for AOT compilation. With rbln-paddlex, the pipeline scores 96.13 on the full OmniDocBench v1.6 (1,651 pages) against the published 96.33. Throughput across the host went from 0.048 to 0.61 pages/s once workers × threads matched the 36 cores; PaddleX's default of 10 CPU threads per worker had been contending."],"tags":["optimum-rbln","NPU","AOT compilation"],"links":[{"label":"rebel-sbpark/paddleocr-vl-rbln","url":"https://github.com/rebel-sbpark/paddleocr-vl-rbln"},{"label":"rbln-paddlex","url":"https://github.com/rebel-sbpark/rbln-paddlex"}],"breakdown":{"problem":"What the repo covers: running the PaddleOCR-VL recognition VLM on RBLN NPUs.","fix":["Wrote an optimum-rbln model class for PaddleOCR-VL, with vision patch counts bucketed for AOT compilation.","Matched workers × threads to the host's 36 cores; PaddleX's default of 10 CPU threads per worker had been contending."],"result":["With rbln-paddlex, the pipeline scores 96.13 on the full OmniDocBench v1.6 (1,651 pages) against the published 96.33.","Throughput across the host went from 0.048 to 0.61 pages/s.","The repository is public."]}},{"id":"repo-paddleocr-vl-rs","date":"2026-07-08","dateLabel":"Jul 8, 2026","kind":"repo","project":"subin9/paddleocr-vl-rs","group":"own-repos","ref":"Repository","status":"Public","title":"paddleocr-vl-rs — Python-free Rust document-parsing pipeline","body":["A Python-free document-parsing pipeline: layout detection on ONNX Runtime, then PaddleOCR-VL on mistral.rs. It scores 94.28 on the full OmniDocBench v1.5, against 94.50 published for the reference model."],"tags":["Rust","ONNX Runtime","OmniDocBench"],"links":[{"label":"subin9/paddleocr-vl-rs","url":"https://github.com/subin9/paddleocr-vl-rs"}],"breakdown":{"problem":"What the repo covers: a Python-free document-parsing pipeline.","fix":["Built layout detection on ONNX Runtime, followed by PaddleOCR-VL on mistral.rs."],"result":["Scores 94.28 on the full OmniDocBench v1.5, against 94.50 published for the reference model.","The repository is public."]}}]}