Start with the workload.
Define what the system needs to do before choosing a tool or turning a knob. Latency, throughput, quality, and cost need a context.
LLM inference · Serving engines · GPUs
I’m Sachin, an inference engineer who leads a team of eight across six products. I read LLM serving engines from source, starting with vLLM, then benchmark whether the code behaves the way it reads, to make inference faster, cheaper, and easier to reason about.
Software & distributed systems15 years of engineering experience
Engineering leadershipSenior Engineering Manager II · Guidepoint
Current technical focusvLLM · SGLang · TensorRT-LLM · FlashInfer
01 / How I think
A model is part of a larger system. The interesting work is understanding how its pieces interact—and which decision changes the outcome.
Define what the system needs to do before choosing a tool or turning a knob. Latency, throughput, quality, and cost need a context.
Follow the request through queues, compute, and memory. An optimization matters when it improves the bottleneck that matters.
Document the evidence, assumptions, and trade-offs. A result becomes useful when someone else can understand and check it.
02 / The scheduler
vLLM’s V1 scheduler doesn’t run prefill and decode as separate batches. Each step hands out one token budget: requests already answering take one token each, and new prompts get what’s left. Change the budget and watch who waits.
max_num_batched_tokensScheduling follows my notes on the V1 scheduler: running requests first, then waiting ones, in arrival order. Step time uses a simple cost model, 8 ms per step plus 0.06 ms per token, roughly an 8B model on one modern GPU. Benchmark 001 will replace it with measured numbers.
03 / The lab
Notes on serving engines, each pinned to the version and source files it was read against. Benchmarks that record the GPU, model, engine version, and exact command.
How one token budget per step explains chunked prefill, preemption by recompute, and why prefix-cache hits save compute. Source files: scheduler.py, kv_cache_manager.py, request.py.
Chunked prefill isn’t a mode.
It falls out of one rule: give each request some of its uncomputed tokens. A long prompt simply spreads across several steps.
Preemption order shapes tail latency.
Under memory pressure, the most recently admitted requests are preempted first and pay a full recompute, because V1 drops their KV cache rather than swapping it to CPU.
Prefix-cache hits save compute, not just memory.
Cached tokens advance a request for free and never enter the step’s token budget.
How does max_num_batched_tokens trade time to first token against inter-token latency in vLLM?
TTFT p50/p99 · inter-token latency p50/p99 · tokens/s · GPU memory. Tests the simulator above against real hardware.
How much throughput does prefix caching add on a shared-system-prompt workload?
Tokens/s · cache hit rate · TTFT
vLLM vs. SGLang on the same model, GPU, and workload.
TTFT · inter-token latency · tokens/s
FP16 vs. FP8 vs. AWQ-INT4: what do latency, throughput, and quality each give up?
Latency · tokens/s · eval score
Small CUDA and Triton kernels, each built on the last, profiled with Nsight Compute.
The next engine notes, in roughly this order.
04 / The longer view
I’ve spent 15 years building software and distributed platforms, and leading the engineers who run them.
That work teaches you where systems really break: under load, at the edge of memory, in the queue nobody was watching. LLM serving has all three, plus a GPU bill that makes every inefficiency visible.
So I’m going engine by engine. I read the source, write down what it does, then benchmark whether the code behaves the way the reading predicts. Every note is pinned to the version I read, so it can be checked.
05 / Writing
Before inference, I worked through classical ML from first principles, one concept per post on DEV Community: optimization, ensembles, evaluation, and data.
06 / Get in touch
Tell me what you serve, at what load, and where the latency goes. I’m glad to compare notes, and open to roles leading inference and ML platform teams.