Inside LLM Inference: From Silicon to Served Token - A Measured Guide to How Language Models Run on GPUs

By The AI Singularity

How does a language model actually become served tokens on a GPU - and why is it so much slower than the hardware's headline TFLOPS promise?

Inside LLM Inference follows a single request through the entire stack: from a PyTorch call down through the CUDA runtime, the streaming multiprocessor, the warp, and the memory system, to the arithmetic cores that finally do the work - and back up to the throughput and cost you pay for.

Every claim is measured, not asserted. A production model runs on 2026's newest silicon - NVIDIA's Blackwell generation - across four inference engines (vLLM, SGLang, TensorRT-LLM, and Hugging Face Transformers), profiled to individual GPU kernels with hardware counters and cross-checked against the physics that predicts them.

You start from zero - Chapter 0 defines every term - and finish able to read a live server: classify a kernel as memory- or compute-bound, and prove it with a clock-sensitivity testsize a KV-cache pool and the batch it admits from a model's own dimensionsread sm_active and sm_occupancy off a running GPU and know what they meanchoose an inference engine on evidence, not vibes
The closing chapters reach the 2026 state of the art: speculative decoding and self-drafting (EAGLE-3, multi-token prediction), the compressed KV cache of multi-head latent attention, prefill/decode disaggregation, mixture-of-experts routing, and the Blackwell-era FlashAttention-4 kernel.

One thesis runs throughout: serving language models is fundamentally a memory-bandwidth problem, and batch size is the price of admission to compute you have already paid for.

Whether you deploy models, optimize inference, or simply want to understand what your GPU is really doing between tokens, this book turns the black box into something you can measure, predict, and tune.

• Format: Paperback
• Language: English
• ISBN: B0H94RZVNV
• ISBN13: 9798187180769
• Release Date: July 2026
• Publisher: Independently Published
• Length: 180 Pages
• Weight: 0.71 lbs.
• Dimensions: 0.4" x 7.0" x 10.0"