#Inference

Articles about Inference — exploring patterns, best practices, and real-world implementations in production systems.

8 posts tagged with inference. ← All posts

#A/B Testing (3)#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Agents (311)#AI Architecture (22)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (254)#AI Evaluation (9)#AI Gateway (8)#AI Governance (39)#AI Red Teaming (9)#AI Research (9)#AI Safety (10)#AI Security (29)#AI in Production (12)#AML (3)#API Design (10)#API Security (8)#APIs (57)#AWS (17)#Accountability (8)#Accounting (9)#Agent Skills (3)#Agentic AI (25)#Agentic Commerce (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (40)#Attention (3)#Audit (4)#Authentication (11)#Authorization (3)#Automation (9)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (310)#Benchmarks (3)#Best Practices (3)#BigQuery (6)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (16)#Caching (11)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (25)#Causal Inference (8)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (8)#Collaboration (5)#Communication (9)#Compliance (54)#Computer Networking (9)#Computer Science (32)#Computer Vision (13)#Concurrency (40)#Consulting (3)#Containers (10)#Context Engineering (10)#Conversational AI (8)#Cost Optimisation (5)#Cost Optimization (3)#Credit (14)#Credit Risk (14)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (9)#Custody (9)#DSPy (8)#Data (14)#Data Engineering (15)#Data Science (8)#Data Structures (10)#Databases (40)#Deep Learning (20)#Deployment (5)#Design Patterns (10)#DevOps (24)#DevSecOps (22)#Developer Experience (5)#Developer Tools (5)#Differential Privacy (3)#Distributed Systems (98)#Documentation (3)#Edge AI (8)#Embeddings (21)#Emotional Intelligence (8)#Energy (8)#Engineering (13)#Engineering Culture (3)#Engineering Judgment (5)#Engineering Practices (16)#Error Handling (4)#Evaluation (67)#Event-Driven Architecture (8)#Explainable AI (8)#FREE-AI (8)#FX (5)#Feature Engineering (3)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Engineering (8)#Financial Systems (129)#Fine-Tuning (12)#Fintech (131)#Flutter (8)#Forecasting (9)#Foreign Exchange (5)#Forward Deployed Engineer (20)#Forward Deployed Engineering (8)#Fraud (10)#Function Tools (5)#Functional Programming (3)#Fundamentals (18)#Fundraising (8)#GCP (5)#GPU (8)#Gemma (4)#Generative AI (3)#Git (8)#Go (220)#Go-to-Market (11)#Google ADK (36)#Governance (61)#Granite (6)#Graph Neural Networks (8)#GraphQL (3)#Growth (3)#Guardrails (34)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (8)#Hugging Face (8)#Human-in-the-Loop (9)#IBM watsonx (8)#Identity (11)#Inference (8)#Information Retrieval (8)#Integration (3)#Intellectual Property (8)#Interfaces (3)#Interpretability (8)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (10)#Knowledge Graphs (10)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM Infrastructure (8)#LLM-as-Judge (3)#LLMOps (8)#LLMs (190)#LangChain (8)#LangGraph (11)#Leadership (28)#Ledger (12)#Legal (8)#Lending (14)#Linux (9)#LlamaIndex (8)#Load Balancing (3)#MCP (22)#MLOps (35)#Machine Learning (108)#Marketing (16)#Markets (4)#Mathematical Finance (8)#Memory (15)#Memory Management (5)#Metrics (7)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Mixture of Experts (5)#Monitoring (4)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (73)#Multimodal (3)#Multimodal AI (10)#NIM (5)#NVIDIA (8)#Networking (3)#OAuth (3)#OWASP (7)#Observability (50)#On-Device AI (8)#Ontology (8)#Open Source (7)#OpenTelemetry (5)#Operating Systems (9)#Operations (11)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (54)#People (8)#Performance (56)#Personalization (9)#Platform Engineering (9)#Policy as Code (3)#PreSales (8)#Privacy (13)#Privacy Engineering (3)#Process (4)#Product (30)#Product Management (8)#Production (21)#Programming (10)#Programming Languages (48)#Prompt Engineering (78)#Prompt Injection (14)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#Quantitative Finance (8)#RAG (69)#RBI (3)#REST (5)#Rails (16)#Reasoning Models (8)#Recommender Systems (8)#Reconciliation (3)#RegTech (8)#Regulation (10)#Reinforcement Learning (9)#Reliability (53)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (14)#Risk Management (10)#Rust (32)#SLSA (3)#SRE (23)#Sales (9)#Scalability (5)#Search (10)#Security (101)#Security Engineering (8)#Self-Evolving Agents (16)#Semantic Web (8)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Delivery (9)#Software Engineering (144)#Spanner (4)#Speech (8)#Startups (30)#Statistics (8)#Strands (8)#Strategy (6)#Streaming (32)#Structured Output (4)#Supply Chain Security (9)#Sustainability (8)#System Design (32)#Systems Programming (56)#Testing (59)#Threat Modeling (3)#Time Series (8)#Tool Use (22)#Tooling (5)#Tools (3)#Trading (8)#Treasury (6)#Type System (3)#Type Systems (11)#TypeScript (8)#Vector Databases (22)#Vector Search (11)#Venture Capital (8)#Version Control (9)#Voice AI (9)#Web Development (6)#Workflow (3)#Workflows (14)#eBPF (8)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·5 min read

Profiling and the Optimization Workflow — Measure, Don't Guess

Everything in this series converges on one discipline: you cannot optimize what you haven't measured. GPU performance is unintuitive — the bottleneck is rarely where you'd guess — so the engineers who make inference fast are the ones who profile first, identify the actual limit, apply the one lever that moves it, and measure again. This closing post ties the series into a workflow you can actually run, and looks at where inference engineering is heading.

You can't optimize what you haven't measured. The optimization loop — baseline, profile, find the dominant limit, apply the matching lever, repeat — plus knowing when to stop, leaning on the ecosystem, and the durable mental model behind it all.

#GPU#Profiling#Performance#Inference
Pratik Dhanave · ·5 min read

Memory Management for LLM Inference — The KV Cache Problem

The single largest consumer of GPU memory during inference often isn't the model — it's the KV cache, the growing store of attention state that lets a model generate text without recomputing the past. Managing it well is the difference between serving a handful of users and serving thousands on the same hardware. This post explains what the KV cache is, why it's such a brutal memory problem, and how PagedAttention borrowed a 1960s operating-systems idea to solve it.

The KV cache often exceeds the model weights in HBM, and naive contiguous allocation wrecks concurrency with fragmentation. PagedAttention applies operating-system paging to fix it — a core inference bottleneck solved by borrowing a classic systems idea.

#GPU#KV Cache#PagedAttention#Inference#Performance#LLMs#Concurrency
Pratik Dhanave · ·5 min read

The Attention Bottleneck — and How FlashAttention Fixed It

Attention is the operation that makes transformers powerful, and for years it was also their worst performance bottleneck — not because it does too much math, but because it moved too much data. FlashAttention is the single most important GPU optimization of the LLM era, and it's a perfect case study in everything this series has built: it's memory-bound, the roofline says so, and the fix is to raise arithmetic intensity by never writing the big intermediate to HBM. Understanding it is understanding how real kernel optimization works.

Naive attention materializes an N×N matrix in HBM — memory-bound and quadratic. FlashAttention fixes it with pure roofline reasoning: tile into shared memory, use an online softmax, fuse the kernel, never touch HBM. The archetype of all inference optimization.

#GPU#FlashAttention#Transformers#Inference#Performance
Pratik Dhanave · ·5 min read

Numeric Precision — Why Lower Precision Makes Inference Faster

The fastest way to speed up LLM inference is often not a better algorithm but fewer bits per number. Precision — how many bits represent each weight and activation — is a direct lever on both of the limits this series has built up: it halves or quarters the data you move (bandwidth, post 2) and unlocks the GPU's fastest arithmetic units (compute, post 3). Understanding floating-point formats is understanding why fp16, bf16, and int8 are everywhere in modern inference.

FP32, FP16, BF16, and INT8/INT4 — how a float splits bits between range and precision, why bf16 won, and why fewer bits is a double win: less data to stream from HBM and faster tensor-core math. Quantization and why LLMs tolerate it.

#GPU#Quantization#Precision#Inference#Performance#LLMs
Pratik Dhanave · ·5 min read

The CUDA Execution Model — Threads, Warps, and Occupancy

To make a GPU fast you have to express your problem the way the hardware wants to run it: as thousands of near-identical threads marching in lockstep. CUDA is the programming model that exposes this, and its abstractions — threads, warps, blocks, grids — map directly onto the SMs and cores from post 1. You don't need to write kernels to be an inference engineer, but you do need this model to understand why some operations fly and others crawl.

Threads, warps, blocks, and grids; SIMT and branch divergence; and the cleverest trick — hiding slow memory by oversubscribing warps (occupancy). Why a GPU can be underutilized without being either compute- or memory-bound.

#GPU#CUDA#Performance#Inference
Pratik Dhanave · ·5 min read

The Roofline Model — Are You Compute-Bound or Memory-Bound?

Posts 1 and 2 kept drawing a line between two worlds: workloads limited by how fast the GPU can compute, and workloads limited by how fast it can move data. The roofline model is the single tool that tells you which world you're in for any given operation — and therefore which optimizations will help and which are a waste of time. It is the most useful mental model in performance engineering, and it fits on one chart.

The roofline model answers the one question that governs every optimization — compute-bound or memory-bound? — from arithmetic intensity (FLOPs per byte). Why LLM prefill is compute-bound and decode is memory-bound, and why batching is the master throughput lever.

#GPU#Roofline#Performance#Inference#LLMs
Pratik Dhanave · ·5 min read

The Memory Hierarchy — Why Bandwidth Rules Inference

Post 1 ended on a reversal: GPUs are fast at arithmetic, but arithmetic is rarely the bottleneck — moving data is. This post makes that concrete by walking the GPU's memory hierarchy, from the tiny ultra-fast registers next to the compute units down to the big slow memory that holds the model's weights. Once you see the orders-of-magnitude gaps between these levels, every performance decision in LLM inference starts to make sense.

The GPU memory hierarchy from registers to HBM, the difference between capacity and bandwidth limits, and the defining fact of LLM generation: it's memory-bandwidth-bound because every token reads the whole model from HBM — which is why batching and quantization work.

#GPU#Memory Bandwidth#Inference#Performance#LLMs
Pratik Dhanave · ·5 min read

Why GPUs Power AI — The Hardware Underneath Inference

Every large language model runs on a GPU, but most engineers treat the GPU as a black box that "does the math fast." Understanding why it's fast — and, more usefully, why it's often not as fast as its spec sheet promises — is the foundation of inference engineering. This series works below the serving layer: not how to run an inference server, but how the silicon actually executes a model and where the real bottlenecks live.

Every LLM runs on a GPU, but the spec-sheet FLOPS number is a near-fiction in practice. Why neural nets map onto GPUs (matrix multiply, SMs, tensor cores), and the governing intuition of the series: compute is cheap, data movement is expensive.

#GPU#Inference#CUDA#Performance#LLMs

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.