Engin Bozaba
Türkçe

Engineering Notes

In-depth technical writing on AI architecture, large language models and deep learning systems, organised as a learning path.

Orientation · Module 01 · The Big Picture

See how an LLM works end to end in one sitting; the deep-dive modules then open up each part of this map.

  1. An LLM, End to End — tokens, transformer, training, inference, and why it makes things upEvery part of an LLM in one read, from tokens and attention to training, inference and why models make things up.16 min read

Foundations · Module 02 · Building Blocks of Neural Networks

The five pieces that make a neural network trainable: activations, losses, backpropagation, optimizers and normalization.

  1. Activation Functions — the geometry of non-linearity, vanishing gradients, and the SwiGLU revolutionWhy layers collapse into a single matrix without non-linearity, and the road from ReLU to SwiGLU.18 min read
  2. Loss Functions — the geometry of error, Log-Sum-Exp stability, and the Cross-Entropy simplificationWhy MSE fails for classification, how the Cross-Entropy gradient simplifies to p − y, and the Log-Sum-Exp trick that keeps it stable.20 min read
  3. Backpropagation and Autodiff — computational graphs, activation memory, and chain rule anatomyHow the chain rule runs over a computation graph, and why backpropagation fills memory with activations.20 min read
  4. Optimization Algorithms — SGD oscillations, the AdamW paradox, and the Muon revolutionWhy SGD oscillates in narrow ravines, why AdamW decouples weight decay and what Muon changes, worked out step by step.22 min read
  5. Weight Initialization and Normalization — Xavier, Kaiming, LayerNorm, and RMSNorm mathematicsHow Xavier and Kaiming initialization, LayerNorm and RMSNorm keep the signal alive through a deep network, shown with the variance arithmetic.20 min read

Architecture · Module 03 · A Prompt's Journey

Follow a prompt step by step, from the tokenizer through the transformer block to the final output layer.

  1. Tokenization — from text to integers, BPE mechanics, and fertilityHow text becomes integers before it reaches the model, how BPE builds its vocabulary, and why some languages spend more tokens.12 min read
  2. The Embedding Layer — from token IDs to continuous vectors, lookup tables, and RoPEThe lookup table that turns token IDs into vectors, the weights it shares with lm_head, and where RoPE comes in.16 min read
  3. Semantic Embeddings — the coordinates of meaning, cosine distance, and HNSW search spaceHow a sentence becomes a single point in meaning space, what cosine similarity really tells you, and how HNSW finds neighbours fast.11 min read
  4. Inside Self-Attention — Q, K, V mechanics, step-by-step matrix arithmetic, and context vectorsWhy Q, K and V are kept separate, computed by hand on a three-word example without skipping a step.18 min read
  5. Inside Causal Attention — The Arrow of Time, Lower Triangular Matrices, and the Math of the VoidHow the mask matrix lets each token look only at the past, and why training depends on it.22 min read
  6. Inside Multi-Head Attention — subspaces, symmetry breaking, and head pruningHow heads split the model dimension into subspaces to track different relationships in parallel, and which of them can be pruned.18 min read
  7. The Transformer Block — the gradient highway, the convex hull, and FFN associative memoryHow residual connections, normalization and the FFN work together inside a single transformer block.22 min read
  8. The Output Layer (LM Head) — Logits, the Memory Wall, Megatron-LM, and Speculative DecodingHow the hidden vector from the final transformer block becomes a word on the screen, smashing the memory wall and accelerating inference.24 min read
  9. Inside the Context Window — From the 2K Wall to FlashAttention, RoPE/YaRN, and 2026 Hybrid Mamba ArchitecturesFollow the context window from the 2K wall through FlashAttention, RoPE/YaRN, and into 2026 hybrid Mamba topologies and hardware memory walls.26 min read

Systems · Module 04 · LLM Systems in Production

What it takes to keep an LLM fast and affordable in production, from cost and latency down to the vLLM engine.

  1. Cost & Latency Anatomy — TTFT, TPOT, prompt caching, and model routingHow to bring down the bill and the latency of LLM calls with TTFT and TPOT metrics, prompt caching and model routing.18 min read
  2. Inside the vLLM Engine — continuous batching, prefix caching, and KV memory managementHow vLLM pages KV memory into blocks, and how continuous batching and the prefix cache work, traced from inside the engine.14 min read

Systems · Module 05 · From Prompting to Autonomous Agents

From prompting techniques to an agent's system prompt and its long-term memory.

  1. Reasoning & Prompting — Chain-of-Thought, ReAct, and the era of reasoning modelsWhere techniques such as Chain-of-Thought and ReAct still earn their place now that models reason on their own.6 min read
  2. Anatomy of an Agent Prompt — system contracts, tool boundaries, and 8 building blocksBuilding an agent's system prompt as a contract of eight blocks, from its role to its tool boundaries.11 min read
  3. Agent Memory — write and read paths, supersession, dreaming, and multi-agent scopeThe system around a stateless model that decides what to remember, what to update and what to forget.35 min read

Systems · Module 06 · RAG & Retrieval Architecture

The retrieval layer, from inside the vector database to RAG patterns that survive production.

  1. Vector Database Architecture — from ANN indexing to metadata filtering and live dataWhat a vector database builds around its ANN index to handle deletes, metadata filtering and live data.11 min read
  2. Production RAG Patterns — diagnosing 4 root failures, HyDE, hybrid search, and rerankingRAG failures fall into four families; this guide picks the fix by symptom, from HyDE to hybrid search and reranking.8 min read

Standalone

  1. Understanding GPU Architecture and LLM Scaling — From Silicon Compute to NVSwitch Fabrics, GPUDirect Storage to Parallelism RooflinesCompute, Network, and Storage: Modern GPU scaling anatomy from silicon Streaming Multiprocessors to NVSwitch fabrics, collective primitives, GPUDirect Storage, and 4D parallelism.52 min read

Loading articles…