Abhinandan
I do RL post-training on reasoning models, & build the inference systems that serve them.
i like working when the world is sleeping. my work cycle is generally 12pm to 4am. And I usually write my thoughts in my artifacts .
The part I live is generally after the launch day. When the reasoning breaks, when the cost skyrockets, when the first traffic hits - all the similar thrills :)
Artifacts
- 05
I do not know if it is my poor luck or my mind, but after failing to measure spec-decoding myself, I realized that SGLang has already merged adaptive speculative decoding.
https://abhinandan.one/artifacts/measuring-adaptive-speculative-decoding-in-sglang · serial 05 · published 2026-09-24T08:00:00+00:00 · updated 2026-09-24T10:14:34.144+00:00 - 04
Building a Mini Inference Engine from Scratch
Sep 202657viewsMost of us know that LLMs are auto-regressive, i.e., they generate token one by one. We also know that they have a concept of KV caching and can process multiple requests…
GitHub ↗https://github.com/awesome-pro/miniserveYT Series ↗https://www.youtube.com/playlist?list=PLIi0G24WvbAAhttps://abhinandan.one/artifacts/building-a-mini-inference-engine-from-scratch · serial 04 · published 2026-09-21T10:38:43.277+00:00 · updated 2026-09-23T11:25:27.735+00:00 - 03
Building a Semantic Cache that Does Not Trust Cosine
Jul 202631viewsUsually semantic caches for LLMs work on embeddings and cosine similarity. You take the new prompt, turn it into a vector, find the closest old prompt, and if the score is high…
https://abhinandan.one/artifacts/building-a-semantic-cache-that-does-not-trust-cosine · serial 03 · published 2026-07-03T07:37:17.016+00:00 · updated 2026-09-23T08:34:02.344+00:00
Open source
Projects
Versioned RL rollout runtime for vLLM with NCCL hot weight updates, version-pure rollouts, and cache-coherent transitions.
keywords: LLM inference, serving runtime, continuous batching, chunked prefill, paged attention, KV cache, prefix caching, preemption, request scheduling, vLLM, TTFT, throughput
date: 2026-09-23 · case study: none
INT8 hierarchical KV caching for SGLang HiCache, delivering 43.75% lower host KV memory and 1.78× more L2 cache capacity for Qwen3-8B.
keywords: LLM inference, serving runtime, continuous batching, chunked prefill, paged attention, KV cache, prefix caching, preemption, request scheduling, vLLM, TTFT, throughput
date: 2026-09-23 · case study: none
a from-scratch LLM serving runtime with continuous batching, chunked prefill, block-based KV management, preemption, and prefix caching.
keywords: LLM inference, serving runtime, continuous batching, chunked prefill, paged attention, KV cache, prefix caching, preemption, request scheduling, vLLM, TTFT, throughput
date: 2026-09-23 · case study: none
Process-supervised RL that made an 8B model reason better, and the gain carried over to a domain it never trained on.
keywords: AgentFlow, process reward model, DAPO, reinforcement learning, agentic reasoning, Qwen3, LoRA fine-tuning, GPQA, AIME, RLHF
date: 2026-05-20 · case study: https://abhinandan.one/agentflow-pro
A semantic cache for LLM agents where a trained classifier decides if a cached answer is safe to reuse, instead of raw cosine similarity.
keywords: semantic cache, LLM cache, FAISS, embeddings, sentence-transformers, pairwise classifier, semantic equivalence, prompt caching, MiniLM
date: 2026-03-15 · case study: https://abhinandan.one/smartmemo
A dependency-free Python framework for readable multi-agent pipelines: sequential, parallel, conditional, and resumable flows.
keywords: multi-agent, orchestration, asyncio, pipeline, LiteLLM, workflow framework, checkpoint resume, dependency-free
date: 2026-02-20 · case study: https://abhinandan.one/orchflow
Behavioral eval for agents: replaces brittle exact-match asserts with repeated-run pass-rate scoring for CI gates.
keywords: agent evaluation, LLM testing, pass rate, CI, behavioral assertions, agent tracing, non-deterministic testing, regression tracking
date: 2026-01-25 · case study: https://abhinandan.one/agenteval
Work
Amazon ML Summer School 2025 · IYMC Gold Honour · Top 1% TypeScript Engineer, Algora