From Tokens to Agents: Understanding, Building, and Scaling Modern AI Systems

By Sameer Shukla

This book is a deep dive into how modern AI systems truly work, moving far beyond surface-level tool usage to uncover the underlying mechanics of large language models (LLMs). It begins at first principles, tracing the full journey of a prompt from raw text input to tokenization, meaning extraction, and next-token prediction. Readers gain an intuitive understanding of concepts like embeddings, context windows, neural networks, and self-attention, building a mental model of how transformers process language. Rather than treating AI as a black box, the book equips readers with clarity on why models behave the way they do, making every later concept easier to grasp and apply.

From this foundation, the book shifts into building practical knowledge systems using Retrieval-Augmented Generation (RAG). It covers semantic search, chunking strategies, vector databases, and the complete RAG pipeline, followed by advanced techniques such as hybrid search, re-ranking, query transformation, and contextual retrieval. Readers then move into creating real-world applications: mastering prompt engineering, enabling tool use through function calling, managing memory, and leveraging frameworks like LangChain. The later sections introduce agentic AI, where systems reason, act, and iterate. Topics include the ReAct pattern, LangGraph workflows, multi-tool and multi-agent systems, human-in-the-loop safety, and reflection-based self-improvement, all supported by working code and practical examples.

By the end of this book, readers will not just know how to use AI tools, they will understand how to design, build, and deploy intelligent systems from the ground up. They will be able to create robust agents, evaluate performance, optimize cost and latency, and secure their systems against real-world risks. More importantly, they will possess a clear mental model of how everything fits together, enabling them to adapt as the field evolves and confidently build production-grade AI applications.

What will you learn?

Inner workings of LLMs (tokens, embeddings, transformers, attention)Building production-ready RAG systems end-to-endDesigning and scaling autonomous AI agentsUsing LangChain and LangGraph for structured workflowsDeploying, monitoring, optimizing, and securing AI systems

Who is it for?

This book is for software engineers, data engineers, data scientists, AI engineers, and technical leaders who want to build real-world AI systems. It is also ideal for career changers with basic Python knowledge who want to deeply understand how AI works. No prior ML or AI framework experience is required.

• Format: Paperback
• Language: English
• ISBN: B0HD6CCVKR
• ISBN13: 9798868832987
• Release Date: February 2027
• Publisher: Apress