Machine-readable versions: llms.txt · llms-full.txt · JSON API · CV PDF

Soobin Park (박수빈): LLM Inference & Serving Engineer

Source: https://cv.subin.rs (generated 2026-10-21 from the same data the site and https://cv.subin.rs/api serve).

Summary

LLM serving engineer focused on throughput and correctness.

At Rebellions I work on serving performance for customer deployments: memory budgeting, batching, prefill/decode disaggregation and capacity planning. Upstream I am a committer on vLLM Semantic Router and lead its Router Models & Inference Runtime working group, and have fixed silent bugs in transformers and mistral.rs.

Contact

• Email: subin1031@gmail.com
• Location: Seoul, South Korea
• GitHub: https://github.com/subin9
• LinkedIn: https://www.linkedin.com/in/soobinpark1
• CV PDF: https://cv.subin.rs/resume.pdf

Highlights

• Serving performance: Found the causes of load-time OOM, silent CPU fallback and a dynamic-batching slowdown on NPU deployments; 2.4–2.7× throughput at the same batch size after the fix.
• Silent failures, fixed upstream: FP8 weights loading without their scales (transformers), guardrails reading 512 of 8K tokens, PII masking that missed every email address.
• Inference internals: FP8 and int4 weight loading, CUDA graphs, chunked attention, KV-cache transfer between models, prefill/decode disaggregation.
• Open-source standing: Committer and working-group lead on vLLM Semantic Router: 83 merged PRs since late August.

Experience

Forward Deployed Engineer, Rebellions (AI NPU Semiconductor) (June 2026 – Present, Seongnam, Korea)

• Memory-budget helper for load-time OOM and silent CPU fallback: Qwen3-8B went from 70 graphs on CPU to 0 fallbacks; greedy benchmarks had hidden a 4× throughput loss.
• Problem: Configs that compiled cleanly hit OOM at load time or fell back to CPU with no error.
• Cause: Auto KV sizing binary-searches the largest block count that fits, but the sampler is compiled later, in another process, on TP rank 0 only. Greedy decoding hides the speed loss.
• Fix: Built a compile helper that reserves the sampler's memory before the block search.
• Result: Qwen3-8B TP8 at batch 96: auto sizing chose 188 blocks, left 70 graphs on CPU, and the engine died on a prefix-cache hit. 164 blocks gave 0 fallbacks. With real sampling instead of greedy, throughput had fallen from 750 to 189 tok/s. FDEs refer to the helper when they set deployment specs.
• Dynamic batching throughput loss: 2.4–2.7× throughput at the same batch size, re-measured with the fix I reported.
• Problem: Dynamic batching lost throughput in a Qwen3 serving benchmark.
• Cause: It binary-searches memory to fill it, leaving about 32 MB free. A prefix-cache hit then fails the KV copy, and when the largest batch in a bucket fails the memory check, the whole bucket falls back to CPU.
• Fix: Found the cause and reported a fix. Later diagnostics showed a kernel timeout rather than OOM and measured the memory difference between static and dynamic batching. Proposed reserving memory instead of a block count.
• Result: Re-measured with the fix: 2.4–2.7× throughput at the same batch size (batch 8: 157 → 382 tok/s; batch 32: 161 → 434).
• Prefill/decode disaggregation on the NPU: ITL spikes went from 78–86× to within 2.7× of baseline; on one node, aggregated serving still carried more load.
• Problem: The optimum path prefills one request per engine step, so each new prefill stalls decode: ITL spiked to 78–86× of baseline.
• Fix: Compared disaggregated and aggregated serving across P:D ratios on 16 NPUs (Qwen3-VL-4B, TP2). Cut the NIXL KV read from 16.6 to 2.23 ms by enabling CMA in UCX_TLS, where it had been excluded and KV went over TCP. Fixed an all-layer single DMA that aborted on large prefills (SYS_TASK_ABORTED) by copying in 512-token chunks.
• Result: Disaggregation kept ITL within 2.7× of baseline. On a single node no static P:D split beat aggregated serving: 2.4 req/s within SLO aggregated vs 1.75 for 2P6D, with about 88% of the TTFT increase coming from the static split's prefill capacity. Conclusion: disaggregation bought decode-latency stability, not single-node capacity.
• Porting models the SDK did not support: PaddleOCR-VL within 0.2 points of the published OmniDocBench v1.6 score, at 12.7× throughput.
• Problem: Customers needed models the NPU SDK did not support: PaddleOCR-VL, and D-FINE and RTMPose for a vision PoC.
• Cause: PaddleOCR-VL throughput was low because PaddleX defaults to 10 CPU threads per worker, which contended on the 36-core host.
• Fix: Ported PaddleOCR-VL and ran it under PaddleX. Matched workers × threads to the 36-core host. Ported D-FINE and RTMPose for the vision PoC in four days.
• Result: PaddleOCR-VL scores 96.13 on the full OmniDocBench v1.6 (1,651 pages) against the published 96.33. Throughput rose 12.7× (0.048 → 0.61 pages/s). D-FINE and RTMPose reached TensorRT FP16 accuracy.
• Logging under a no-prompt-logging requirement: stdout, stderr and Docker logs at 0 bytes, with no vLLM source change.
• Problem: A deployment could not keep prompts in logs, and vLLM's own options still left 30 KB in stderr.
• Fix: Added a 7-level log control to the serving image, without changing vLLM source.
• Result: At off/quiet, stdout, stderr and Docker logs are 0 bytes.
• Customer deployments across the stack: Server hardware, the NPU compiler and runtime, and vLLM serving.
• Problem: Customer deployments span server hardware, the NPU compiler and runtime, and vLLM serving.
• Fix: Worked on server hardware (NIC topology, boot and storage, SR-IOV virtualization), the NPU compiler and runtime, and vLLM serving. Answered partner technical questions (virtualization, NIC configuration, security). Commented on draft NPU validation criteria and proposed an agentic benchmark.
• Stack: vLLM, KV cache, Dynamic batching, P/D disaggregation, Customer deployments, Benchmarking

Solution Engineer, Microsoft Korea (Jan 2026 – Mar 2026, Seoul, Korea)

• Technical lead contact for a Copilot helpdesk agent and admin dashboard across HR, IT and MCAPS; ran enterprise Copilot PoCs.
• Stack: Copilot Studio, Agents, PoC

Data Scientist, Munich Re Korea (Jan 2025 – Dec 2025, Seoul, Korea)

• Built Treaty Hub alone: a multi-service GenAI platform (RAG Q&A, VLM document parsing) on FastAPI and ChromaDB under enterprise network limits; presented across 4+ APAC offices.
• Stack: RAG, FastAPI, ChromaDB, VLM

AI Researcher, NCSoft, Audio AI Lab (Mar 2024 – Aug 2024, Pangyo, Korea)

• Voice-conversion and SFX-generation prototypes in PyTorch; an in-house tagging tool adopted by sound designers.
• Stack: PyTorch, Audio

Open source

Upstream work on the LLM serving stack: routing and guardrails, KV-cache reuse, attention kernels, tokenizers and quantized-weight loading. Most of it is one kind of bug. The system does not crash or log; it returns a confident answer that is wrong. Verified against the GitHub API on 21 Oct 2026.

Totals: 114 pull requests, 86 merged upstream, 71 issues filed, 10 projects

vLLM Semantic Router (vllm-project/semantic-router)

The routing layer of the vLLM project (not the vLLM engine itself). Committer and lead of the Router Models & Inference Runtime working group.

Guardrails that read only the first 512 tokens: fixed at every call site, then at the kernel

Aug 31 – Oct 6, 2026 · 11 PRs · #3206 · #3643 · #3402 · #3460 · #4567 · 11 merged upstream

• Problem: Guardrail classifiers truncated input at 512 tokens without saying so. A guardrail that scored only the opening of a prompt looked the same as one that read all of it.
• Cause: The classifiers built a full seq × seq attention matrix, which is why the limit existed.
• Fix: Found the truncation in the security API, the response jailbreak filter, the PII API, the hallucination detector and RAG retrieval. Fixed all six call sites with one overlapping-window walk, where a clean verdict requires every chunk to be scored (#3206, #3334, #3369, #3643, #3650, #3841). Moved the classifiers onto a chunked SDPA kernel to remove the reason for the limit. Fixed where chunked hallucination spans landed: in an answer that repeats itself, spans from later chunks had been reported on earlier copies (#4567).
• Result: Peak memory at 8,192 tokens fell from 15,789 MiB to 1,003 MiB for one global ModernBERT attention block on CPU in F32 (15.4 GiB to 1.0 GiB, about 16× lower; a local layer drops 62×). Qwen3 at 8K, previously not runnable, now fits in 3.4 GiB, and every site is pinned to its dense implementation within 1e-4. 11 PRs merged upstream.

Links: PR #3206, PR #3334, PR #3369, PR #3643, PR #3650, PR #3841, PR #3402, PR #3460, PR #4567

Serving-path failures that never showed up in the logs

Sep 1 – Oct 10, 2026 · PR #3303 · #3454 · #4425 · #4508 · #4629 · #4630 · 6 merged upstream

• Problem: Several serving-path failures left nothing in the logs: a CPU signal ran about 10× slower, malformed input returned 500, a missing device failed only after the model download, and half of the CLI unit tests never ran in CI.
• Cause: The slowdown came from model ranking never seeing the execution provider, so a signal pinned to CPU loaded the FP16 ONNX graph and paid for a cast around every operator.
• Fix: Fixed the CPU slowdown (#3303) and traced five published "fp16" files that held FP32 weights to the export tool, which CI had never tested (#3454). Made an unpaired surrogate in a request return 400 instead of 500 (#4508). Checked the device before the download and logged the supervisor's forced kills instead of leaving them silent (#4629). Installed the runtime in both images from one lock instead of from lower bounds (#4630), and ran all CLI unit tests in CI (#4425).
• Result: All 3,016 CLI unit tests now run in CI. A device the host lacks now fails before the download, not after it and again on every retry. 6 PRs merged upstream.

Links: PR #3303, PR #3454, PR #4425, PR #4629, PR #4630

GPU fast paths for the router's own models, admitted only where answers hold

Oct 9 – 13, 2026 · PR #4779 · #4811 · #4813 · 3 merged upstream

• Problem: The runtime's fused element-wise kernels were written for MI300X. On NVIDIA they do not reproduce the eager ops bit for bit (add_rmsnorm differs by one BF16 ulp).
• Fix: Registered the fused kernels on CUDA as approximate kernels, used only for a model family that opts in under a 99%-per-question-type agreement rule (#4779). max_speed selected them at first; the default --dtype auto does now. Measured each fast path switched off on an MI300X, which showed that Vela 2.0 never replayed a CUDA graph (#4811). Made Vela 2.0's decoder forest replay per shape on CUDA (#4813). After a reviewer's run found uncounted graph memory and kept results being overwritten, made the forest path return a copy and the graph budget charge each capture its device memory, up to 4 GiB or half of the free memory (#5009).
• Result: Decision 2.0 opts in: 11% to 48% faster on an RTX 4070 Ti SUPER, with none of 102 decisions changed. Vela 2.0 does not, because its 0.8B changed 23 of 336 span answers. Vela forest graphs on an RTX 4070 Ti SUPER: a one-question request on the 0.8B goes from 67.7 to 17.7 ms, still bit for bit (4B: 1.9×). All three PRs are merged. ROCm stays eager, since a live forest graph faults inside the release image's ROCm 7.2.3.

Links: PR #4779, PR #4811, PR #4813

Decision 3.0 (27B) in FP8 and int4, loaded as stored

Oct 13 – 18, 2026 · PR #5066 · vllm-sr/d3-FP8 · vllm-sr/d3-W4A16-AWQ · Merged upstream · packages published

• Problem: d3 takes 52.2 GB in BF16, and the model runtime expanded FP8 and int4 compressed-tensors packages to BF16 instead of loading them as stored.
• Fix: Let the model runtime load FP8 (W8A8 FP8_DYNAMIC) and int4 (GPTQ, AWQ) compressed-tensors packages as stored (#5066, merged 18 Oct). FP8 activations are quantized per row in one Triton kernel and multiplied in torch._scaled_mm. Handled the hardware paths: on MI300 the E4M3 bits are reinterpreted as e4m3fnuz with doubled scales; on Ada at 256 rows or more, under the approximate profile, the product is taken unscaled in FP32 and then scaled. Fixed two bugs inside the PR: the FP8 activation kernel indexed rows in 32 bits and returned NaN past 2^31 elements (now 64-bit indexing), and on Ada, PyTorch before 2.12 zeroed the stream-K workspace on the default stream (pytorch#177573), which hung the first graph replay (now worked around). Built and published the FP8 and AWQ packages on Hugging Face as vllm-sr/d3-FP8 and vllm-sr/d3-W4A16-AWQ.
• Result: On 14,031 test requests, FP8 keeps 99.08% of BF16's answers in 31.9 GB instead of 52.2 GB; AWQ int4 keeps 97.76% in about 16.5 GB. On MI300X with d3's forward replayed as graphs, the FP8 median is 36.8 ms against 46.4 ms for BF16 (#4999; record corrected in #5270). Without graphs it is 80.9 against 67.1 ms, because per-layer host cost outweighs the smaller matmuls. On the public Decision Index 0.3 suite, FP8 scores 65.40 against 65.45 for d3, with 99.15% top-1 agreement over 312,320 questions; AWQ scores 65.00.

Links: PR #5066, d3-FP8, d3-W4A16-AWQ, pytorch#177573

Serving Decision 3.0 as CUDA graphs

Oct 13 – 20, 2026 · PR #4999 · #4998 · #5009 · #5270 · 4 merged upstream

• Problem: Decision 3.0 (d3) is the project's new family of decision models, the largest 27B. Its forwards are noncausal, and the runtime ran every one of them eagerly.
• Fix: Built the noncausal attention mask from padding information the host already has, and captured text forwards as CUDA graphs per exact shape (#4999). Fixed a BLAS workspace that every capture left behind (#4998). Fixed the graph budget counting cache that large eager batches had grown, which made a batching server stop capturing after 16 graphs (#5009). Corrected the latency record the model cards cite (#5270).
• Result: A one-question request on d3-lite (0.8B) went from 29.4 to 7.3 ms on an RTX 4070 Ti SUPER, with the same answers. 4 PRs merged upstream.

Links: PR #4999, PR #4998, PR #5009, PR #5270

Model-runtime failures found bringing up the 27B model

Oct 17 – 20, 2026 · PR #5184 · #5183 · #5181 · #5179 · #5180 · #5185 · #5190 · 7 merged · 4 open

• Problem: Bringing up the d3 packages at three dtypes exposed runtime failures: loading d3 on MI300X peaked at 49 GiB of host memory, and a forward that ran out of device memory returned 500.
• Fix: Staged a GPU model's checkpoint per tensor instead of as a full host copy (#5184). Answered an out-of-memory forward with 413 when it ran alone and 429 when it shared the forward, instead of 500, so a later request no longer fails after it (#5183). Released cached GPU memory before a failed load is retried (#5181). Smaller fixes: kernel choices pinned only at the published precision (#5179), media workers waited for when the runtime stops (#5180), d3's CPU answers recorded for AVX-512 hosts with native BF16 (#5185), and FP32 inputs on GPUs sent to the reference varlen attention (#5190).
• Result: Loading d3 on MI300X peaked at 49 GiB of anonymous host memory before #5184 and 6 GiB after. An out-of-memory forward now answers 413 or 429 instead of 500. 7 merged on 17–18 Oct; 4 open.
• Next: Still open: #5220 answers a full worker pool as 429 on the native gateway too, and #5274, #5286 and #5001 follow up on the smaller d3 packages, media preprocessing and padding.

Links: PR #5184, PR #5183, PR #5181, PR #5220

Reusing a 14B model's KV cache in a 32B model

Oct 1 – 4, 2026 · PR #4497 · #4526 · epic #2976 · 1 merged · 1 in review

• Problem: The epic maps Qwen3-14B's KV cache into Qwen3-32B so the larger model can skip prefill. The first mapper was a ridge regression fitted to the cache values.
• Fix: Added a rank-16 correction trained so the 32B model predicts the same next tokens it would from its own cache (#4497, +808 / −3). Verified the artifact through another contributor's connector PR. Added the KL evaluation, because HellaSwag cannot tell mappers apart (#4526).
• Result: KL against the 32B model's own cache falls from 0.088 to 0.036 on chat and from 0.142 to 0.082 on plain text. Refitting the ridge map on the same data does not get there, so the gain comes from the objective. Through the connector PR, KL is 0.055 vs 0.120 for v1. 1 merged (#4497), 1 in review.
• Next: One PR is in review, and the latency gain in serving is not measured yet.

Links: PR #4497, PR #4526, Epic #2976

Making "which route wins" a contract instead of an accident

Aug 31 – Sep 21, 2026 · Proposal #3227 · 8 PRs · closes #3081 · 9 merged upstream

• Problem: Route selection could rank a decision lower for having more evidence: mean(.90, .90) beat mean(.90, .90, .88).
• Cause: Selection mixed policy rules into the confidence that picks the winner, averaged over leaves, and ignored routing.strategy whenever a tier was set.
• Fix: Wrote the contract on a maintainer's roadmap issue (Proposal #3227): a rule is either policy or evidence, and confidence ranks only a pool whose members each report one score of one kind. Anything else falls back to priority. Collapsed three orderings into one comparator in eight PRs merged over two days (+1,513 / −183). Added the ranking trace to the eval API and the replay record.
• Result: The winner did not change in any shipped config. 9 merged upstream; closes #3081.
• Next: A follow-up calibration (#4550, in review) brings ECE on held-out domain scores from 0.263 to 0.033.

Links: Proposal #3227, Issue #3081, PR #4550

Decision questions that read the conversation, and answers checked against the question

Oct 5 – 9, 2026 · PR #4778 · #4573 · issue #4572 · 1 merged · 1 in review

• Problem: Decision signals read only the current user turn, so a follow-up like "Rewrite your previous answer in fewer words" routed on almost nothing.
• Fix: Let a decision rule read up to eight earlier user turns within a fixed 1,760-character budget (#4778). Assistant turns are left out, since they scored no better at about twice the tokens. Refused engine answers that the engine's own distribution contradicted or that the question never declared, as invalid_model_output, so the selector falls back (#4573).
• Result: On MT-Bench's 80 two-turn conversations, second-turn category accuracy rises from 41.3% to 77.5% on the 0.6B and from 51.2% to 87.5% on Vela 2.0 9B. #4573 is merged, and all 64 answers from the four built-in families pass unchanged.
• Next: #4778 is in review.

Links: PR #4778, PR #4573, Issue #4572

One 0.6B decision model vs. eight per-signal encoders, on the same data

Sep 28 – Oct 4, 2026 · Issue #4305 · #4306 · Completed · published

• Problem: The router runs a separate encoder per signal. The question was whether one fine-tuned model could replace them.
• Fix: Built a corpus-matched suite (10 signals, 139 sources) that rebuilds byte for byte from an empty directory. Trained every model on the same rows and compared them with a grouped paired bootstrap and Benjamini–Hochberg correction.
• Result: Decision-1.0-Route-0.6B wins 21, ties 30 and loses 6 against encoders on the same training budget, with the largest gains on held-out hallucination data (+0.06 to 0.19 AUC). It is 2.2× slower on CPU (409 vs 189 ms), faster on MI300X (47 vs 82 ms), and still weaker than the shipped guard on most safety sets. Completed and published (issue #4305, #4306).

Links: Issue #4306, Dataset, Model

Benchmarks that were measuring the training data

Aug 31 – Oct 7, 2026 · PR #3247 · #3248 · #4307 · #4308 · #4309 · #4565 · 6 merged upstream

• Problem: Benchmarks gave wrong scores with no metric flagging it: a label-order mismatch dropped MMLU-Pro from 0.9445 to 0.6746, and a domain benchmark was scoring the model's own training data.
• Cause: The eval registry and the artifact listed labels in different orders, so four of fourteen classes were scored against the wrong logit.
• Fix: Added versioned manifests with cross-reference validation (#3247) and a calibration-aware baseline harness (#3248). The same checks later found that Vela Domain had trained on the MMLU questions behind 6,810 of MMLU-Pro's 12,032 test rows, and that the fact-check split can be solved at AUC 0.999 from the source column alone. Made the baselines score only rows the models never saw (#4308, #4309), and moved fact-check to a corpus-matched split where the source predicts nothing (#4565).
• Result: Vela Domain had scored 0.769 on the rows it trained on against 0.631 on the rest. On the new fact-check split the source reaches AUC 0.500, down from 0.999, and the ranking flips: Vela FactCheck reaches 0.696 accuracy against 0.573 for the checkpoint the old split put first. 6 merged upstream.

Links: PR #3247, PR #4308, PR #4565, Issue #4301

The jailbreak detector answers a different question than the router asks

Sep 14 – Oct 4, 2026 · Issue #3787 · PR #3924 · #3945 · 2 merged · retrain in progress

• Problem: The prompt-injection rule asks whether a prompt attacks the system, but the shipped detector was trained to say whether content is harmful.
• Cause: 82% of the detector's test positives carry no attack marker, and what it learned was mostly prompt length.
• Fix: Built a decontaminated 6,646-row set labelled for both questions. Made the metric honest (#3924). Reported which window decided a guard score (#3945), which explained benign documents being blocked at 0.95 on their second window.
• Result: The shipped model reads AUC 0.81, yet at a 1% false-positive budget it catches 3.9% of attacks. An attack-labelled candidate reaches AUC 0.96. 2 merged.
• Next: The retrain is in progress; the attack-labelled candidate is in review.

Links: Issue #3787, PR #3924, PR #3945

Reproducible datasets from shadow traffic, for comparing routes

Sep 18 – Oct 13, 2026 · PR #3912 · #3915 · #3998 · #4087 · #4089 · #4123 · 6 merged upstream

• Problem: A maintainer roadmap epic needed a dataset, built from replayed shadow traffic, that two routes can be compared on reproducibly.
• Fix: Built the dataset (+3,336 / −14): the manifest stores only hashes, its digest is its identity, and splits follow a seeded hash so record order cannot change them. Excluded truncated or failed observations under a counted reason instead of half-including them, and kept any single decision from dominating the set. Added blinded pairwise judge tasks served from the dataset; #4123 publishes a validated dataset and its judgments by digest, and the judge-task route from #4116 landed with it. Fixed every shadow-dispatch outcome being silently dropped since an earlier change deferred the replay start (#3914, fixed in #3915).
• Result: Routes can be compared on a hash-addressed, reproducible dataset. A silent data-loss bug was found and fixed along the way (#3915). 6 merged upstream.

Links: PR #3912, PR #3915, PR #4089, PR #4123

Response detection rebuilt as a composable routing signal

Sep 2 – 11, 2026 · PR #3335 · #3531 · epic #3272 · 3 merged upstream

• Problem: Response jailbreak and hallucination detection were plugins that did their own detecting, so neither could be combined with request-side signals in a routing decision.
• Fix: Took the maintainer's epic (#3272) and turned both into response-direction signals, scored once per response. Shipped it in #3335 (+2,597 / −230, 45 files) and #3531 (+1,919 / −332, 64 files).
• Result: A recipe can now say "route to a stronger model when the answer looks wrong". +4.5k lines across 109 files, merged; 3 merged upstream in total.

Links: PR #3335, PR #3531, PR #3689, Epic #3272

PII scores that were confidently wrong

Aug 26 – Oct 13, 2026 · PR #3027 · #3141 · #3147 · #3080 · #3568 · #3694 · #4777 · 7 merged upstream

• Problem: PII masking was decided by scores that looked reasonable but were miscomputed. Emails could not be detected, and a repeated name could reach the provider in clear text.
• Fix: Fixed entity confidence, which was a running fold rather than a mean, so masking depended on how many subwords a name split into (#3027). Fixed the decoder, which required a B- tag the model never emits, leaving emails and domains undetectable at any threshold (#3141). Fixed non-ASCII spans, which were byte offsets reported as character positions (#3147). Masked every whole-word copy of a repeated value, not only the first (#4777).
• Result: EMAIL_ADDRESS recall went from 0/24 to 21/24 (#3141). A second "John Smith" that scored 0.49 against a 0.75 threshold is now masked, and the expansion handles 20,000 copies within 2 s, where the old path took 16 s at 3,200. 7 merged upstream.

Links: PR #3027, PR #3141, PR #3147, PR #3080, PR #3568, PR #3694, PR #4777

Committer and working-group lead

Aug 26 – Oct 21, 2026 · Committer · WG lead · Committer since Sep 17

• Problem: What the role covers: committer and lead of the Router Models & Inference Runtime working group, scoping incoming work for the maintainers and reviewing other contributors' PRs.
• Fix: Self-nominated to the working group on 26 August; made a committer and WG lead on 17 September. Scoped the phasing of the built-in model runtime (#4496), the first slice of the Decision runtime follow-ups (#4472) and the lifecycle and release gaps it left (#4537), plus per-area dependency projects for the training code (#3932). Reviewed 9 PRs from other contributors: asked for a shared validator and a guard that keeps approximate KV out of the prefix cache on the cross-model KV connector (#4514, #4513), and ran the GPU parity check for serving an external NLI model as a decision model (#4765).
• Result: 83 of 98 PRs opened to the project are merged; most close an issue filed first with a reproduction. The author's own PRs closed all five items from the runtime scoping work. The shared validator and the prefix-cache guard both went in.

Links: Committers & team, WG roster, Merged PRs

Hugging Face transformers (huggingface/transformers)

Quantized-weight loading (merged) and tokenizer routing.

Silent corruption in FP8 weight loading: a fix, its revert, and 42 relabelled repos

Jul 8 – Aug 20, 2026 · PR #47152 · PR #48072 · 2 merged upstream

• Problem: FP8 checkpoints saved with status "frozen" loaded without their scales. There was no error; the model just generated noise.
• Cause: Raw fp8 weights were loaded and their scales never applied: q_proj max was 448 instead of ~0.68.
• Fix: Co-authored the fix, which loads these checkpoints with their weight_scale (#47152, merged 22 Jul). When that fix broke checkpoints with a mislabeled quantization_status, reproduced the break within a day and sampled 536 Hub configs to size it. Wrote the revert (#48072, merged 20 Aug), approved by both the transformers and Red Hat maintainers. Listed 42 nm-testing repos with the wrong quantization_status.
• Result: 2 PRs merged upstream: the fix (#47152) and its revert (#48072). Red Hat relabelled the 42 repos ("should be fixed now").

Links: Issue #47145, PR #47152, Revert #48072

Route byte-level llama tokenizers to TokenizersBackend

Jul 2, 2026 · PR #47017 · Open

• Problem: Byte-level checkpoints that declare LlamaTokenizerFast, DeepSeek-R1-Distill-Llama among them, decode with every space lost. A vLLM endpoint returns garbled output and no error.
• Fix: Routed tokenizers on whether tokenizer.json is actually byte-level, instead of the accepted repo-id allowlist (#47017). When asked to extend the allowlist instead, swept the 50,000 most-downloaded llama repos.
• Result: 1,866 repos across 929 orgs are affected, and the top 30 all return "Helloworld.I'manAI." without a single log line. Users were still reporting it in October.
• Next: The PR is open.

Links: PR #47017, Issue #45488

mistral.rs (EricLBuehler/mistral.rs)

Vision-model support and engine fixes.

feat(models): Support PaddleOCR-VL vision model (rework of #2320)

Jul 26, 2026 · PR #2356 · Open

• Problem: There was no end-to-end support for the PaddleOCR-VL vision model.
• Fix: Added the SigLIP NaViT vision encoder, projector and ERNIE-4.5 decoder with 2D RoPE / mRoPE (#2356, +3,780 / −34).
• Result: Token-for-token parity with transformers under greedy decoding across nine task types, on both CPU-f32 and GPU-bf16.
• Next: The PR is open.

Links: PR #2356, Issue #2128

fix(llg): honor tokenizer special flag in toktrie detok

Jul 8, 2026 · PR #2319 · llguidance #361 · Open

• Problem: Content tokens disappeared on decode. PaddleOCR-VL tables, for example, collapsed into run-on text.
• Cause: toktrie marked any <...>-shaped token as special and overrode the tokenizer's own flag.
• Fix: Filed it upstream as llguidance #361, which led to a fix there. Fixed it engine-side in this PR (#2319, +192).
• Result: llguidance fixed it upstream after #361.
• Next: The engine-side PR is open.

Links: PR #2319, llguidance #361

fix(webui): send chat images as data URLs

Jul 11, 2026 · merged Aug 29 · PR #2338 · issue #2336 · Merged upstream

• Problem: Images sent from the built-in web UI failed with HTTP 500, while the same image through the API worked.
• Cause: The UI was sending back a server-relative path that the media policy rejects.
• Fix: Made the UI send a data URL instead (#2338, issue #2336).
• Result: Merged by the maintainer.

Links: PR #2338, Issue #2336

fix(mllama): CUDA image-inference crash, eager-only paged flag, vision head config

Jul 11, 2026 · PR #2337 · issue #2335 · Open

• Problem: Llama-3.2-11B-Vision failed with PagedAttention, the default, and crashed on CUDA without it, so there was no working path.
• Fix: Marked the model eager-only and corrected its vision-head config (#2337, issue #2335).
• Result: The fix is up as #2337.
• Next: The PR is open.

Links: PR #2337, Issue #2335

RBLN · work at Rebellions (RBLN-SW)

NPU serving stack: issue reports and prototypes.

Why a model that compiles cleanly runs out of NPU memory

Jul 31, 2026 · Fork branch · Prototype · used as an FDE deployment reference

• Problem: With automatic KV sizing, models that compiled cleanly then ran out of memory on TP rank 0 or silently fell back to CPU.
• Cause: The runtime binary-searches the largest KV block count that fits, but the sampler is compiled later, in another process, and only on TP rank 0, so the search never saw it.
• Fix: Prototyped a compile helper that reserves the sampler's budget before the search runs. Presented the memory-budget issue at a company seminar.
• Result: On Qwen3-8B with TP8 at batch 96, 188 blocks sent 70 graphs to CPU and the engine died on a prefix-cache hit; 164 blocks gave 0 fallbacks. Greedy decoding hides only the speed loss; with real sampling, throughput fell from 750 to 189 tok/s. FDEs refer to the helper when they set deployment specs.

Links: Branch, Diff

Empty last_hidden_state when prompt length is a multiple of prefill_chunk_size

Jul 28, 2026 · Issue #649 · Closed

• Problem: When the prompt length was an exact multiple of prefill_chunk_size, last_hidden_state came back empty, silently degrading embeddings for those lengths only.
• Fix: Reported the bug (issue #649).
• Result: Fixed upstream two days later; the issue is closed.

Links: Issue #649

KV-cache events are never published on the optimum-rbln path

Aug 13, 2026 · Issue #922 · Open

• Problem: With --kv-events-config on the optimum-rbln path, no KV-cache events are ever published.
• Fix: Reported the bug (issue #922).
• Result: Anything built on these events, such as cache-hit accounting, routing or autoscaling, reads a silent zero.
• Next: Issue #922 is open.

Links: Issue #922

paddleocr-vl-rbln — PaddleOCR-VL recognition VLM on the NPU

Jul 21, 2026 · Repository · Public

• Problem: What the repo covers: running the PaddleOCR-VL recognition VLM on RBLN NPUs.
• Fix: Wrote an optimum-rbln model class for PaddleOCR-VL, with vision patch counts bucketed for AOT compilation. Matched workers × threads to the host's 36 cores; PaddleX's default of 10 CPU threads per worker had been contending.
• Result: With rbln-paddlex, the pipeline scores 96.13 on the full OmniDocBench v1.6 (1,651 pages) against the published 96.33. Throughput across the host went from 0.048 to 0.61 pages/s. The repository is public.

Links: rebel-sbpark/paddleocr-vl-rbln, rbln-paddlex

My own repos (subin9)

Public NPU and Rust inference work.

paddleocr-vl-rs — Python-free Rust document-parsing pipeline

Jul 8, 2026 · Repository · Public

• Problem: What the repo covers: a Python-free document-parsing pipeline.
• Fix: Built layout detection on ONNX Runtime, followed by PaddleOCR-VL on mistral.rs.
• Result: Scores 94.28 on the full OmniDocBench v1.5, against 94.50 published for the reference model. The repository is public.

Links: subin9/paddleocr-vl-rs

vLLM (vllm-project/vllm)

LoRA classification heads, a Rust frontend and OpenTelemetry conformance.

LoRA adapters with multi-layer classification heads (RoBERTa, XLM-RoBERTa)

Oct 6, 2026 · PR #60211 · issue #60197 · Open

• Problem: PEFT sequence-classification adapters for RoBERTa failed to load in vLLM (classifier.dense.bias is unsupported LoRA weight, issue #60197). 29 of the 40 most popular roberta-base adapters save their head that way.
• Cause: vLLM's LoRA code assumed a single-Linear head, but RoBERTa's is dense → tanh → out_proj, so a LoRA update on classifier.dense was dropped without an error (max abs diff 0.45).
• Fix: Made models declare their head linears in order, and opted RoBERTa and XLM-RoBERTa in (#60211).
• Result: On an RTX 4070 Ti SUPER, vLLM in fp16 matches transformers + PEFT in fp32 within 4.5e-3 on real adapters.
• Next: A reviewer would rather keep the scope narrow, so the PR waits for the code owner.

Links: PR #60211, Issue #60197

[Rust Frontend] Support --allowed-media-domains

Jul 26, 2026 · PR #49855 · Open

• Problem: The Rust frontend had no equivalent of the Python frontend's media-domain allowlist, so a multimodal deployment could be pointed at arbitrary URLs.
• Fix: Added --allowed-media-domains (#49855, +286 / −10). Closed both bypasses that automated review found: file: URLs and redirects to hosts off the list.
• Result: The Rust frontend can restrict media to listed domains, with the file: and redirect bypasses closed.
• Next: The PR is open.

Links: PR #49855

Emit spec-required gen_ai.operation.name on request spans

Jul 5, 2026 · PR #47633 · Open

• Problem: vLLM request spans omitted gen_ai.operation.name, which the OpenTelemetry GenAI conventions require, so vLLM traces dropped out of any GenAI dashboard keyed on the convention.
• Fix: Added gen_ai.operation.name to request spans, along with gen_ai.request.model (#47633).
• Result: Request spans carry the attribute the conventions require.
• Next: The PR is open.

Links: PR #47633

Ollama (ollama/ollama)

Batch size for noncausal decision-model GGUFs.

Give noncausal decision models a whole-prompt batch

Oct 20, 2026 · PR #19159 · Open

• Problem: Noncausal decision-model GGUFs (vllm-d3) failed on any prompt over 512 tokens.
• Cause: They got the default batch of 512.
• Fix: Gave them a batch that holds the whole prompt (#19159).
• Result: PR open, no reviews yet.
• Next: Waiting for a first review.

Links: PR #19159

LG AI EXAONE (LGAI-EXAONE/EXAONE-4.0-32B)

Chat-template fix on the model repo.

chat_template: guard tool_call arguments against pre-serialized JSON

Aug 12, 2026 · HF PR #14 · Open

• Problem: The chat template JSON-encoded tool-call arguments unconditionally, so arguments already serialized by the OpenAI API were encoded twice and agent loops broke silently.
• Fix: Guarded tool_call arguments in the chat template against pre-serialized JSON (HF PR #14). Scoped the impact rather than assuming it.
• Result: vLLM is unaffected, while llama.cpp and Ollama embed the same template in their GGUF builds.
• Next: HF PR #14 is open.

Links: HF discussions/14

Projects

Treaty Hub: RAG-Based Document Intelligence Ecosystem (2025)

Multi-service GenAI platform for reinsurance operations with version management and auto-update system, earning regional recognition and escalation to APAC offices (Singapore) for continued development.

• Reduced treaty document search time from hours to minutes
• Enabled non-technical users to access treaty information independently
• Delivered 10+ presentations across 4+ APAC offices; escalated to Region office (Singapore)
• Created unified knowledge ecosystem connecting treaty documents and email communications
• Built modular architecture enabling rapid feature development
• Developed comprehensive dashboard displaying treaty history, document versions, and classification metadata in timeline views
• Connected with Deep Scan service to enable fast, accurate article-level parsing of PDFs including text and image content
• Implemented dual-mode LLM summaries (bullet-point and detailed formats) for each article
• Created Forum service for email management with automatic content parsing and tagging
• Developed Treaty Copilot with optional Forum inclusion, enabling search across formal contracts and informal communications
• Created Excel export feature for structured article-level data extraction
• Stack: FastAPI, React, LLM/RAG, ChromaDB

Railog: Seoul Subway Real-Time Delay Prediction & Visualization (Ongoing)

End-to-end ML pipeline for Seoul Metro delay prediction — real-time ingestion, dbt feature engineering, RandomForest + MLflow model serving, 4 Airflow DAGs on K3s, and a live MapLibre map across 754 stations.

• Trained on 9.62M continuously-updating records; MAE 60s / R² 0.45 for delay prediction
• Identified upstream_delay_seconds as top feature (34.6% importance), validating delay-propagation hypothesis
• Delivered real-time train position map and per-station congestion visualization across 754 stations
• Demonstrated NAS-aware data architecture separating heavy DuckDB computation from storage I/O
• Orchestrated 4 Apache Airflow DAGs: 2-min realtime ingestion, daily timetable sync, 10-min dbt transforms, and weekly model retraining
• Designed NAS-aware data architecture: raw tables extracted via PostgreSQL COPY STDOUT → local parquet, then all window functions executed in DuckDB — decoupling heavy computation from NAS I/O
• Trained RandomForestRegressor on 9.62M (continuously updating) train-position records; achieved MAE 60s / R² 0.45 for delay prediction
• Validated delay-propagation hypothesis: top feature upstream_delay_seconds at 34.6% importance
• Served prediction intervals (RF estimator percentiles) via FastAPI REST API, visualized in interactive MapLibre GL JS live map showing real-time train positions and per-station congestion severity
• Stack: Airflow, dbt, DuckDB, MapLibre GL JS

Taxi Driver Allocation: Real-time Demand Prediction & Optimal Dispatch System (2024)

XGBoost-based demand prediction with Hungarian algorithm optimization for taxi driver revenue increase. KISTI Director's Award winner.

• Achieved 3.1% revenue increase on weekdays, 11.3% on weekends, 12.7% on holidays
• Demand prediction model accuracy: MAE 1.93 for hourly predictions across 80 clusters
• Won KISTI Director's Award at 2024 DATA·AI Analysis Competition
• Provided explainable AI recommendations using SHAP analysis
• Implemented K-means clustering to divide Daejeon into ~80 meaningful zones
• Developed XGBoost demand prediction model achieving MAE 1.93
• Integrated SHAP analysis for transparent demand predictions
• Applied Hungarian matching algorithm for optimal taxi-cluster assignment
• Built Flutter-based mobile app with intuitive UI/UX
• Created web-based management dashboard
• Stack: XGBoost, K-means, SHAP, Flutter
• Links: KISTI Director's Award Winner

Self-Hosted Homelab: Hybrid Infrastructure — Bare Metal, K3s & Mesh VPN (Ongoing)

Multi-host homelab: bare-metal Docker Compose (GMKtec K12), K3s cluster (Firebat T9), and Synology NAS — Nginx reverse proxy with wildcard TLS and Headscale mesh VPN for zero-trust access.

• Operate multi-host production homelab with K3s cluster, bare-metal services, and NAS storage backend
• Zero-trust mesh VPN via self-hosted Headscale — no public port exposure across any node
• Wildcard TLS automation via DNS-01 challenges for *.subin.rs and *.subin.pro
• Externally accessible Mattermost workspace with Claude Code–triggered bot and browser-based SSH gateway
• Self-hosted an LLM serving stack with vLLM and AWQ quantization on a consumer GPU, with hands-on inference benchmarking — early groundwork that led into the NPU serving work at Rebellions.
• Architected multi-host setup: GMKtec K12 (Docker Compose production), Firebat T9 K3s cluster (MetalLB L2), Synology DS224+ NAS as storage backend
• Built Nginx Proxy Manager reverse proxy with wildcard TLS automation via acme-dns–driven DNS-01 challenges for *.subin.rs and *.subin.pro
• Deployed self-hosted Headscale coordination server for zero-trust mesh VPN access across all nodes in lieu of public port exposure
• Exposed Mattermost workspace (with Claude Code–triggered bot) and browser-based SSH gateway (Termix) as externally accessible self-hosted services
• Stack: K3s, Headscale, Nginx, SSL/TLS

ChulhaGil: Knowledge-Graph–Based Agricultural Shipment Timing Advisor (Ongoing)

Knowledge-graph shipment-timing advisor for MAFRA Agriculture + AI Competition — dlt → PostgreSQL → Pydantic v2 → NetworkX pipeline over weather, price, and customs data.

• Competing in 11th MAFRA Agriculture + AI Startup Competition
• Multi-source ETL pipeline integrating weather, price, and trade statistics
• Knowledge-graph approach providing explainable shipment-timing recommendations
• Designed knowledge-graph pipeline: dlt for ingestion → PostgreSQL → Pydantic v2 data models → NetworkX graph for relational reasoning
• Integrated KMA weather data, aT auction/wholesale/retail price data, and customs import statistics as graph nodes and edges
• Built explainable recommendation layer surfacing graph-traversal reasoning for shipment timing decisions
• Stack: NetworkX, dlt, PostgreSQL, Pydantic v2

Data Flow: Data Reconciliation Platform (2025)

End-to-end automation platform streamlining reinsurance data processing with intelligent validation, standardization, and reconciliation workflows.

• Reduced data processing time by 80% through automation
• Enabled precise settlement of 60 billion KRW through automated reconciliation
• Created benefit code mapping system for cross-insurer product identification
• Built comprehensive validation engine analyzing uploaded files for structural anomalies
• Developed intelligent mapping interface for normalizing heterogeneous column names
• Created auto-mapping algorithm using fuzzy matching and pattern recognition
• Developed reconciliation engine processing standardized data against cedent statements
• Stack: FastAPI, SQLAlchemy, Pandas, Luigi

Trend Radar: Market Intelligence Platform (2025)

AI-powered news aggregation with LLM-based filtering, translation, and semantic search for Korean insurance market.

• Reduced information processing time through automated filtering and translation
• Bridged language barrier providing English-accessible insights from Korean sources
• Delivered end-to-end intelligence platform
• Implemented two-stage AI pipeline: LLM-based content classifier filtering promotional content
• Automated translation and summarization converting Korean articles to concise English summaries
• Built ChromaDB vector database for semantic search
• Created Market Pulse dashboard for trend visualization
• Stack: LLM, ChromaDB, RAG, FastAPI

Doc Compass: Hybrid Search Engine (2025)

Enterprise email search combining BM25 and vector similarity with adaptive weighting for precise retrieval.

• Addressed both precision and recall use cases effectively
• Reduced email search time from hours to minutes
• Created searchable institutional memory
• Built hybrid search combining BM25 and Vector Database approaches
• Implemented adaptive weighting mechanism based on query characteristics
• Created RAG pipeline with LLM-powered answer synthesis
• Stack: BM25, ChromaDB, RAG, FastAPI

Cancer Risk Prediction: Survival Analysis Model (2025)

Interpretable survival models with SurvSHAP(t) for cancer claim predictions using medical reimbursement data.

• Successfully addressed left censoring through survival analysis
• C-index of 0.71 with MAE 2.7 years in cancer prediction
• Solution scheduled for introduction as official product
• Shifted from classification to regression, predicting time until cancer diagnosis
• Achieved MAE of 2.7 years and c-index of 0.71
• Implemented SurvSHAP(t) for interpretable risk factor insights
• Implemented Ranking-to-Learn methodology
• Stack: XGBoost, SurvSHAP(t), FastAPI, SQL Server

Speech Recognition: Age-Specific ASR Model (2023)

DeepSpeech2-based model for elderly and child speech. NIA Director's Award winner.

• Won NIA Director's Award (7th out of 302 teams)
• Led team as captain and main AI engineer
• Successfully addressed demographic-specific ASR problem
• Implemented DeepSpeech2-based model with extensive data augmentation
• Created custom preprocessing for target demographics
• Built lightweight deployment package for edge devices
• Stack: PyTorch, DeepSpeech2, Librosa, Docker

Sign Language Interpreter: Real-Time Emergency Medical Sign Language Translation (2023)

1st place at SNU Ambient AI Competition. MediaPipe + GRU pipeline for real-time emergency medical sign language translation, deployed as a Flutter tablet app for first responders.

• Won 1st place at Seoul National University Ambient AI Bootcamp & Competition
• Led team (KOKO AI) as team leader across dataset curation, model development, and app deployment
• Built specialized emergency medical sign language dataset combining AI Hub data and 46 self-recorded clips
• Demonstrated real-time sign language translation on tablet hardware for first responder use
• Built two-stage pipeline: Google MediaPipe extracts skeletal landmarks (hands, arms, face) from video frames, generating structured motion data
• Replaced LSTM with GRU layers for improved accuracy under limited training data conditions
• Curated specialized dataset from AI Hub's non-face-to-face medical consultation data, supplemented with 46 self-recorded emergency sign language clips by team members
• Deployed Flutter-based tablet app with Flask backend for real-time inference result display
• Stack: MediaPipe, GRU, Flutter, Flask
• Links: 1st Place — SNU Ambient AI Competition, Watch Presentation

Education

Korea University (Mar 2018 – Feb 2025)

• B.A. in Korean Language and Literature
• B.S. in Computer Science (Double Major, CS GPA 4.29 / 4.5)
• Research assistant, Center for Lexicography (2018–2023)
• Overall GPA: 4.17 / 4.50

Awards

• 3rd Place, Fire Safety Bigdata Competition (2025): Fire Prevention & Safety Analytics
• KISTI Director's Award, DATA AI Competition (2024): Daejeon Taxi Driver Location Recommendation Track
• Advanced to 3rd Phase, 4th LG Aimers (2024): B2B Sales Opportunity Model (99th out of 1100+ teams)
• Encouragement Award, Korea Institute of Communication Sciences (2024): Academic Conference Idea Competition
• Encouragement Award, Jeju Technopark Skin Precision Diagnosis AI (2023)
• NIA Director's Award, Korean AI Competition Track 1 (2023): 7th out of 302 teams
• 1st Place, Seoul National University Ambient AI Bootcamp & Competition (2023): Sign language interpreter in Emergency situations
• 2nd Place Overall, 5th Goorm NLP Bootcamp (2022): 2nd out of 50+ people

Skills

• Serving & inference: vLLM, mistral.rs, Continuous batching, Paged KV cache, Prefix caching, Prefill/decode disaggregation, KV-cache transfer, FP8 / AWQ / INT4, TTFT / ITL benchmarking, Capacity planning
• Accelerators & kernels: CUDA GPUs, AMD Instinct MI300X (ROCm), CUDA graphs, RBLN NPU, AOT compilation, Tensor parallelism, Chunked / flash attention, ONNX Runtime, torch.compile, Perfetto
• Platform & ops: Docker, Kubernetes / K3s, Airflow, MLflow, OpenTelemetry, Grafana, CI/CD
• Languages: Python, Rust, TypeScript, PyTorch, FastAPI

Languages and certifications

• Korean (Native)
• English (TEPS 504, Business Fluent)
• Baekjoon Online Judge Top 1.5% Rating (Active Problem Solver)
• Microsoft Certified: Azure Fundamentals (AZ-900, 2025)
• TEPS 504 – Business Fluent English Proficiency (2024)