{"name":"Soobin Park","title":"LLM Inference & Serving Engineer","summary":"LLM serving engineer focused on throughput and correctness.\n\nAt Rebellions I work on serving performance for customer deployments: memory budgeting, batching, prefill/decode disaggregation and capacity planning. Upstream I am a committer on vLLM Semantic Router and lead its Router Models & Inference Runtime working group, and have fixed silent bugs in transformers and mistral.rs.","stats":[{"number":"2.4–2.7×","label":"Throughput at the same batch size","context":"Dynamic batching on NPU · Rebellions"},{"number":"83","label":"Merged PRs in vLLM Semantic Router","context":"Committer · 86 merged upstream in all","url":"https://github.com/vllm-project/semantic-router/pulls?q=is%3Apr+author%3Asubin9+is%3Amerged"},{"number":"99.08%","label":"BF16 answers kept in FP8","context":"Decision 3.0 27B · 31.9 GB instead of 52.2 GB","url":"https://github.com/vllm-project/semantic-router/pull/5066"},{"number":"−27 pts","label":"Hidden accuracy loss, found and fixed","context":"Label-order mismatch in a shipped router model","url":"https://github.com/vllm-project/semantic-router/pull/3247"}],"highlights":[{"label":"Serving performance","text":"Found the causes of load-time OOM, silent CPU fallback and a dynamic-batching slowdown on NPU deployments; 2.4–2.7× throughput at the same batch size after the fix."},{"label":"Silent failures, fixed upstream","text":"FP8 weights loading without their scales (transformers), guardrails reading 512 of 8K tokens, PII masking that missed every email address."},{"label":"Inference internals","text":"FP8 and int4 weight loading, CUDA graphs, chunked attention, KV-cache transfer between models, prefill/decode disaggregation."},{"label":"Open-source standing","text":"Committer and working-group lead on vLLM Semantic Router: 83 merged PRs since late August."}],"education":[{"institution":"Korea University","location":"Seoul, Korea","degrees":["B.A. in Korean Language and Literature","B.S. in Computer Science (Double Major, CS GPA 4.29 / 4.5)","Research assistant, Center for Lexicography (2018–2023)"],"period":"Mar 2018 – Feb 2025","gpa":"Overall GPA: 4.17 / 4.50"}],"certifications":["Baekjoon Online Judge Top 1.5% Rating (Active Problem Solver)","Microsoft Certified: Azure Fundamentals (AZ-900, 2025)","TEPS 504 – Business Fluent English Proficiency (2024)"],"languages":["Korean (Native)","English (TEPS 504, Business Fluent)"],"experience":[{"company":"Rebellions (AI NPU Semiconductor)","location":"Seongnam, Korea","position":"Forward Deployed Engineer","period":"June 2026 – Present","logo":"/images/rebellions-logo.jpg","logoPadding":"3px","achievements":["Configs that compiled cleanly hit OOM or silent CPU fallback at runtime. Auto KV sizing binary-searches the largest block count that fits, but the sampler is compiled later, in another process, on TP rank 0 only. On Qwen3-8B TP8 at batch 96, 188 blocks left 70 graphs on CPU and the engine died on a prefix-cache hit; 164 blocks gave 0 fallbacks. Greedy decoding hides only the speed loss: with real sampling, throughput fell from 750 to 189 tok/s. FDEs refer to my memory-budget helper when they set deployment specs.","Found why dynamic batching lost throughput: it binary-searches memory to fill it, leaving about 32 MB free, so a prefix-cache hit fails the KV copy, and when the largest batch in a bucket fails the memory check the whole bucket falls back to CPU. Re-measured with the fix I reported, throughput at the same batch size was 2.4–2.7× (batch 8: 157 → 382 tok/s; batch 32: 161 → 434). Later diagnostics showed a kernel timeout rather than OOM and measured the memory difference between static and dynamic batching; I proposed reserving memory instead of a block count.","Built prefill/decode disaggregation on the NPU. The optimum path prefills one request per engine step, which stalls decode (ITL spikes of 78–86×); in a 16-card NPU study (Qwen3-VL-4B, TP2), disaggregation kept ITL within 2.7× of baseline. On a single node no static P:D split beat aggregated serving: 2.4 req/s within SLO aggregated vs 1.75 for 2P6D, with about 88% of the TTFT increase coming from the static split's prefill capacity. Disaggregation bought decode-latency stability, not single-node capacity.","Cut the NIXL KV read from 16.6 to 2.23 ms by enabling CMA in UCX_TLS, where it had been excluded and KV went over TCP. An all-layer single DMA aborted on large prefills (SYS_TASK_ABORTED); copying in 512-token chunks fixed it.","Work across the full stack on customer deployments: server hardware (NIC topology, boot and storage, SR-IOV virtualization), the NPU compiler and runtime, and vLLM serving. Answered partner technical questions (virtualization, NIC configuration, security), commented on draft NPU validation criteria and proposed an agentic benchmark.","Ported models the SDK did not support. PaddleOCR-VL scores 96.13 on the full OmniDocBench v1.6 (1,651 pages) against the published 96.33; throughput rose 12.7× (0.048 → 0.61 pages/s) once workers × threads matched the 36-core host, where PaddleX's default of 10 CPU threads per worker had been contending. Also D-FINE and RTMPose for a customer vision PoC in four days, at TensorRT FP16 accuracy.","Added a 7-level log control to the serving image: at off/quiet, stdout, stderr and Docker logs are 0 bytes, against 30 KB in stderr with vLLM's own options, with no vLLM source change."],"tags":["vLLM","KV cache","Dynamic batching","P/D disaggregation","Customer deployments","Benchmarking"],"details":[{"title":"Memory-budget helper for load-time OOM and silent CPU fallback","headline":"Qwen3-8B went from 70 graphs on CPU to 0 fallbacks; greedy benchmarks had hidden a 4× throughput loss.","breakdown":{"problem":"Configs that compiled cleanly hit OOM at load time or fell back to CPU with no error.","cause":"Auto KV sizing binary-searches the largest block count that fits, but the sampler is compiled later, in another process, on TP rank 0 only. Greedy decoding hides the speed loss.","fix":["Built a compile helper that reserves the sampler's memory before the block search."],"result":["Qwen3-8B TP8 at batch 96: auto sizing chose 188 blocks, left 70 graphs on CPU, and the engine died on a prefix-cache hit. 164 blocks gave 0 fallbacks.","With real sampling instead of greedy, throughput had fallen from 750 to 189 tok/s.","FDEs refer to the helper when they set deployment specs."]}},{"title":"Dynamic batching throughput loss","headline":"2.4–2.7× throughput at the same batch size, re-measured with the fix I reported.","breakdown":{"problem":"Dynamic batching lost throughput in a Qwen3 serving benchmark.","cause":"It binary-searches memory to fill it, leaving about 32 MB free. A prefix-cache hit then fails the KV copy, and when the largest batch in a bucket fails the memory check, the whole bucket falls back to CPU.","fix":["Found the cause and reported a fix.","Later diagnostics showed a kernel timeout rather than OOM and measured the memory difference between static and dynamic batching.","Proposed reserving memory instead of a block count."],"result":["Re-measured with the fix: 2.4–2.7× throughput at the same batch size (batch 8: 157 → 382 tok/s; batch 32: 161 → 434)."]}},{"title":"Prefill/decode disaggregation on the NPU","headline":"ITL spikes went from 78–86× to within 2.7× of baseline; on one node, aggregated serving still carried more load.","breakdown":{"problem":"The optimum path prefills one request per engine step, so each new prefill stalls decode: ITL spiked to 78–86× of baseline.","fix":["Compared disaggregated and aggregated serving across P:D ratios on 16 NPUs (Qwen3-VL-4B, TP2).","Cut the NIXL KV read from 16.6 to 2.23 ms by enabling CMA in UCX_TLS, where it had been excluded and KV went over TCP.","Fixed an all-layer single DMA that aborted on large prefills (SYS_TASK_ABORTED) by copying in 512-token chunks."],"result":["Disaggregation kept ITL within 2.7× of baseline.","On a single node no static P:D split beat aggregated serving: 2.4 req/s within SLO aggregated vs 1.75 for 2P6D, with about 88% of the TTFT increase coming from the static split's prefill capacity.","Conclusion: disaggregation bought decode-latency stability, not single-node capacity."]}},{"title":"Porting models the SDK did not support","headline":"PaddleOCR-VL within 0.2 points of the published OmniDocBench v1.6 score, at 12.7× throughput.","breakdown":{"problem":"Customers needed models the NPU SDK did not support: PaddleOCR-VL, and D-FINE and RTMPose for a vision PoC.","cause":"PaddleOCR-VL throughput was low because PaddleX defaults to 10 CPU threads per worker, which contended on the 36-core host.","fix":["Ported PaddleOCR-VL and ran it under PaddleX.","Matched workers × threads to the 36-core host.","Ported D-FINE and RTMPose for the vision PoC in four days."],"result":["PaddleOCR-VL scores 96.13 on the full OmniDocBench v1.6 (1,651 pages) against the published 96.33.","Throughput rose 12.7× (0.048 → 0.61 pages/s).","D-FINE and RTMPose reached TensorRT FP16 accuracy."]}},{"title":"Logging under a no-prompt-logging requirement","headline":"stdout, stderr and Docker logs at 0 bytes, with no vLLM source change.","breakdown":{"problem":"A deployment could not keep prompts in logs, and vLLM's own options still left 30 KB in stderr.","fix":["Added a 7-level log control to the serving image, without changing vLLM source."],"result":["At off/quiet, stdout, stderr and Docker logs are 0 bytes."]}},{"title":"Customer deployments across the stack","headline":"Server hardware, the NPU compiler and runtime, and vLLM serving.","breakdown":{"problem":"Customer deployments span server hardware, the NPU compiler and runtime, and vLLM serving.","fix":["Worked on server hardware (NIC topology, boot and storage, SR-IOV virtualization), the NPU compiler and runtime, and vLLM serving.","Answered partner technical questions (virtualization, NIC configuration, security).","Commented on draft NPU validation criteria and proposed an agentic benchmark."],"result":[]}}]},{"company":"Microsoft Korea","location":"Seoul, Korea","position":"Solution Engineer","period":"Jan 2026 – Mar 2026","logo":"/images/microsoft-logo.png","logoPadding":"3px","internship":true,"achievements":["Technical lead contact for a Copilot helpdesk agent and admin dashboard across HR, IT and MCAPS; ran enterprise Copilot PoCs."],"tags":["Copilot Studio","Agents","PoC"]},{"company":"Munich Re Korea","location":"Seoul, Korea","position":"Data Scientist","period":"Jan 2025 – Dec 2025","internship":true,"logo":"/images/munich-re-logo.png","logoPadding":"3px","achievements":["Built Treaty Hub alone: a multi-service GenAI platform (RAG Q&A, VLM document parsing) on FastAPI and ChromaDB under enterprise network limits; presented across 4+ APAC offices."],"tags":["RAG","FastAPI","ChromaDB","VLM"]},{"company":"NCSoft, Audio AI Lab","location":"Pangyo, Korea","position":"AI Researcher","period":"Mar 2024 – Aug 2024","internship":true,"logo":"/images/ncsoft-logo.png","logoPadding":"3px","achievements":["Voice-conversion and SFX-generation prototypes in PyTorch; an in-house tagging tool adopted by sound designers."],"tags":["PyTorch","Audio"]}],"skills":[{"category":"Serving & inference","icon":"brain","items":["vLLM","mistral.rs","Continuous batching","Paged KV cache","Prefix caching","Prefill/decode disaggregation","KV-cache transfer","FP8 / AWQ / INT4","TTFT / ITL benchmarking","Capacity planning"]},{"category":"Accelerators & kernels","icon":"tools","items":["CUDA GPUs","AMD Instinct MI300X (ROCm)","CUDA graphs","RBLN NPU","AOT compilation","Tensor parallelism","Chunked / flash attention","ONNX Runtime","torch.compile","Perfetto"]},{"category":"Platform & ops","icon":"cloud","items":["Docker","Kubernetes / K3s","Airflow","MLflow","OpenTelemetry","Grafana","CI/CD"]},{"category":"Languages","icon":"code","items":["Python","Rust","TypeScript","PyTorch","FastAPI"]}],"awards":[{"title":"3rd Place","organization":"Fire Safety Bigdata Competition","year":"2025","description":"Fire Prevention & Safety Analytics","image":"/images/fire-safety-award.jpg"},{"title":"KISTI Director's Award","organization":"DATA AI Competition","year":"2024","description":"Daejeon Taxi Driver Location Recommendation Track","image":"/images/data-ai-award.jpg"},{"title":"Advanced to 3rd Phase","organization":"4th LG Aimers","year":"2024","description":"B2B Sales Opportunity Model (99th out of 1100+ teams)","image":"/images/lg-aimers-competition.png"},{"title":"Encouragement Award","organization":"Korea Institute of Communication Sciences","year":"2024","description":"Academic Conference Idea Competition","image":"/images/academic-conference-award.jpg"},{"title":"Encouragement Award","organization":"Jeju Technopark Skin Precision Diagnosis AI","year":"2023","description":"","image":"/images/skin-competition.png"},{"title":"NIA Director's Award","organization":"Korean AI Competition Track 1","year":"2023","description":"7th out of 302 teams","image":"/images/korean-ai-competition-award.png"},{"title":"1st Place","organization":"Seoul National University Ambient AI Bootcamp & Competition","year":"2023","description":"Sign language interpreter in Emergency situations","image":"/images/snu-ambient-ai-competition-award.png"},{"title":"2nd Place Overall","organization":"5th Goorm NLP Bootcamp","year":"2022","description":"2nd out of 50+ people","image":"/images/goorm-nlp-bootcamp-certificate.jpg"}],"contact":{"email":"subin1031@gmail.com","phone":"+82 10-9199-9786","location":"Seoul, South Korea","linkedin":"https://www.linkedin.com/in/soobinpark1","github":"https://github.com/subin9"},"openSourceSpotlight":{"eyebrow":"Open source","project":"vLLM Semantic Router","projectUrl":"https://github.com/vllm-project/semantic-router","role":"Committer · Lead, Router Models & Inference Runtime WG","badge":"83 merged","lead":"The routing layer of the vLLM project. Guardrails that silently read only part of their input, a route-ranking contract, cross-model KV-cache reuse, GPU fast paths admitted only where answers hold, serving the new 27B decision model as CUDA graphs and in FP8 and int4, and evaluations that had been measuring their own training data.","entries":[{"ref":"#4497","url":"https://github.com/vllm-project/semantic-router/pull/4497","text":"14B → 32B KV-cache mapper: KL to the target's own cache 0.088 → 0.036"},{"ref":"#4779","url":"https://github.com/vllm-project/semantic-router/pull/4779","text":"Fused kernels on CUDA where answers hold: 11–48% faster, 0 of 102 decisions changed"},{"ref":"#5066","url":"https://github.com/vllm-project/semantic-router/pull/5066","text":"Decision 3.0 (27B) loads FP8 and int4 packages as stored: FP8 keeps 99.08% of BF16 answers in 31.9 GB instead of 52.2 GB"},{"ref":"#4999","url":"https://github.com/vllm-project/semantic-router/pull/4999","text":"Decision 3.0 forwards as CUDA graphs: one question 29.4 → 7.3 ms on d3-lite, same answers"},{"ref":"#5184","url":"https://github.com/vllm-project/semantic-router/pull/5184","text":"Checkpoint staged per tensor: host memory loading d3 on MI300X 49 GiB → 6 GiB"},{"ref":"#3402","url":"https://github.com/vllm-project/semantic-router/pull/3402","text":"Chunked attention removed the 512-token limit's reason: 15.4 GiB → 1.0 GiB at 8K tokens (global layer)"},{"ref":"#3227","url":"https://github.com/vllm-project/semantic-router/pull/3227","text":"Decision ranking contract, implemented in 8 merged PRs"},{"ref":"#4306","url":"https://github.com/vllm-project/semantic-router/issues/4306","text":"One 0.6B decision model vs. eight encoders: 21 wins, 6 losses, cost reported"},{"ref":"#3787","url":"https://github.com/vllm-project/semantic-router/issues/3787","text":"Jailbreak detector catches 3.9% of attacks at 1% FPR; retrain reaches AUC 0.96"}],"links":[{"label":"Committers & team","url":"https://vllm-sr.ai/community/team"},{"label":"Working-group roster","url":"https://vllm-sr.ai/community/work-groups"},{"label":"Merged PRs","url":"https://github.com/vllm-project/semantic-router/pulls?q=is%3Apr+author%3Asubin9+is%3Amerged"}]},"nameKo":"박수빈"}