sr·Sachin RajputLLM inference engineer

LLM inference · Serving engines · GPUs

Engineering the
space between
model & production.

I’m Sachin, an inference engineer who leads a team of eight across six products. I read LLM serving engines from source, starting with vLLM, then benchmark whether the code behaves the way it reads, to make inference faster, cheaper, and easier to reason about.

THE INFERENCE ENGINE
Context enters the system.Move your pointer to explore
Requests
Based in IndiaA closer look 01 — 06

Software & distributed systems15 years of engineering experience

Engineering leadershipSenior Engineering Manager II · Guidepoint

Current technical focusvLLM · SGLang · TensorRT-LLM · FlashInfer

01 / How I think

Performance is a
systems problem.

A model is part of a larger system. The interesting work is understanding how its pieces interact—and which decision changes the outcome.

01

Start with the workload.

Define what the system needs to do before choosing a tool or turning a knob. Latency, throughput, quality, and cost need a context.

02

Trace before tuning.

Follow the request through queues, compute, and memory. An optimization matters when it improves the bottleneck that matters.

03

Make the reasoning visible.

Document the evidence, assumptions, and trade-offs. A result becomes useful when someone else can understand and check it.

02 / The scheduler

One token budget.
Every request shares it.

vLLM’s V1 scheduler doesn’t run prefill and decode as separate batches. Each step hands out one token budget: requests already answering take one token each, and new prompts get what’s left. Change the budget and watch who waits.

Token budget per step max_num_batched_tokens
New long prompt
Longest pause for running requests–
Long prompt, first token–
Short prompt, first token–

Scheduling follows my notes on the V1 scheduler: running requests first, then waiting ones, in arrival order. Step time uses a simple cost model, 8 ms per step plus 0.06 ms per token, roughly an 8B model on one modern GPU. Benchmark 001 will replace it with measured numbers.

03 / The lab

Read the source.
Then measure it.

Notes on serving engines, each pinned to the version and source files it was read against. Benchmarks that record the GPU, model, engine version, and exact command.

Engine internalsPublished
NOTE 001 / vLLM V1

The scheduler,
one step at a time.

How one token budget per step explains chunked prefill, preemption by recompute, and why prefix-cache hits save compute. Source files: scheduler.py, kv_cache_manager.py, request.py.

Read the note on GitHub

What reading the code showed

Chunked prefill isn’t a mode.

It falls out of one rule: give each request some of its uncomputed tokens. A long prompt simply spreads across several steps.

Preemption order shapes tail latency.

Under memory pressure, the most recently admitted requests are preempted first and pay a full recompute, because V1 drops their KV cache rather than swapping it to CPU.

Prefix-cache hits save compute, not just memory.

Cached tokens advance a request for free and never enter the step’s token budget.

Still openWhat does the curve between first-token and inter-token latency look like as the budget goes from 512 to 8,192? That’s benchmark 001.
  1. 001

    How does max_num_batched_tokens trade time to first token against inter-token latency in vLLM?

    TTFT p50/p99 · inter-token latency p50/p99 · tokens/s · GPU memory. Tests the simulator above against real hardware.

    Up next
  2. 002

    How much throughput does prefix caching add on a shared-system-prompt workload?

    Tokens/s · cache hit rate · TTFT

    Queued
  3. 003

    vLLM vs. SGLang on the same model, GPU, and workload.

    TTFT · inter-token latency · tokens/s

    Queued
  4. 004

    FP16 vs. FP8 vs. AWQ-INT4: what do latency, throughput, and quality each give up?

    Latency · tokens/s · eval score

    Queued

Kernel ladder

CUDA folder

Small CUDA and Triton kernels, each built on the last, profiled with Nsight Compute.

  1. Vector add and bandwidth testAchieved vs. peak bandwidth, coalescing
  2. Parallel reductionWarp shuffles, shared memory, bank conflicts
  3. Matmul: naive to tiled to register-blockedArithmetic intensity, the roofline
  4. Softmax: naive vs. onlineThe trick behind FlashAttention
  5. The same matmul in TritonHow Triton maps blocks to hardware

Reading queue

The next engine notes, in roughly this order.

  • vLLMPagedAttention and the KV-cache block manager
  • vLLMAutomatic prefix caching: block hashing and eviction
  • SGLangRadixAttention vs. vLLM prefix caching
  • TensorRT-LLMIn-flight batching and FP8 paths
  • FlashInferAttention kernels for decode vs. prefill
Portrait of Sachin Rajput
Sachin RajputInference engineer · India
THE PERSON BEHIND THE WORK↗

04 / The longer view

Fifteen years of platforms.
Now, the engine underneath.

I’ve spent 15 years building software and distributed platforms, and leading the engineers who run them.

That work teaches you where systems really break: under load, at the edge of memory, in the queue nobody was watching. LLM serving has all three, plus a GPU bill that makes every inefficiency visible.

So I’m going engine by engine. I read the source, write down what it does, then benchmark whether the code behaves the way the reading predicts. Every note is pinned to the version I read, so it can be checked.

Current roleSenior Engineering Manager II · Guidepoint
Team8 engineers · 6 products
FoundationDistributed platforms on AWS
FocusLLM serving, KV cache, GPU performance
PROFESSIONAL BACKGROUNDSachin Rajput on LinkedInExperience, engineering leadership, and current focus.

05 / Writing

Writing it down
is how I learn it.

Before inference, I worked through classical ML from first principles, one concept per post on DEV Community: optimization, ensembles, evaluation, and data.

82posts publishedExplore my DEV profile ↗

ML groundwork

82 posts on classical ML, on DEV Community

06 / Get in touch

Serving models
in production?

Tell me what you serve, at what load, and where the latency goes. I’m glad to compare notes, and open to roles leading inference and ML platform teams.

Let’s talk