#Streaming

Articles about Streaming — exploring patterns, best practices, and real-world implementations in production systems.

32 posts tagged with streaming. ← All posts

#A/B Testing (3)#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Agents (311)#AI Architecture (22)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (254)#AI Evaluation (9)#AI Gateway (8)#AI Governance (39)#AI Red Teaming (9)#AI Research (9)#AI Safety (10)#AI Security (29)#AI in Production (12)#AML (3)#API Design (10)#API Security (8)#APIs (57)#AWS (17)#Accountability (8)#Accounting (9)#Agent Skills (3)#Agentic AI (25)#Agentic Commerce (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (40)#Attention (3)#Audit (4)#Authentication (11)#Authorization (3)#Automation (9)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (310)#Benchmarks (3)#Best Practices (3)#BigQuery (6)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (16)#Caching (11)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (25)#Causal Inference (8)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (8)#Collaboration (5)#Communication (9)#Compliance (54)#Computer Networking (9)#Computer Science (32)#Computer Vision (13)#Concurrency (40)#Consulting (3)#Containers (10)#Context Engineering (10)#Conversational AI (8)#Cost Optimisation (5)#Cost Optimization (3)#Credit (14)#Credit Risk (14)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (9)#Custody (9)#DSPy (8)#Data (14)#Data Engineering (15)#Data Science (8)#Data Structures (10)#Databases (40)#Deep Learning (20)#Deployment (5)#Design Patterns (10)#DevOps (24)#DevSecOps (22)#Developer Experience (5)#Developer Tools (5)#Differential Privacy (3)#Distributed Systems (98)#Documentation (3)#Edge AI (8)#Embeddings (21)#Emotional Intelligence (8)#Energy (8)#Engineering (13)#Engineering Culture (3)#Engineering Judgment (5)#Engineering Practices (16)#Error Handling (4)#Evaluation (67)#Event-Driven Architecture (8)#Explainable AI (8)#FREE-AI (8)#FX (5)#Feature Engineering (3)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Engineering (8)#Financial Systems (129)#Fine-Tuning (12)#Fintech (131)#Flutter (8)#Forecasting (9)#Foreign Exchange (5)#Forward Deployed Engineer (20)#Forward Deployed Engineering (8)#Fraud (10)#Function Tools (5)#Functional Programming (3)#Fundamentals (18)#Fundraising (8)#GCP (5)#GPU (8)#Gemma (4)#Generative AI (3)#Git (8)#Go (220)#Go-to-Market (11)#Google ADK (36)#Governance (61)#Granite (6)#Graph Neural Networks (8)#GraphQL (3)#Growth (3)#Guardrails (34)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (8)#Hugging Face (8)#Human-in-the-Loop (9)#IBM watsonx (8)#Identity (11)#Inference (8)#Information Retrieval (8)#Integration (3)#Intellectual Property (8)#Interfaces (3)#Interpretability (8)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (10)#Knowledge Graphs (10)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM Infrastructure (8)#LLM-as-Judge (3)#LLMOps (8)#LLMs (190)#LangChain (8)#LangGraph (11)#Leadership (28)#Ledger (12)#Legal (8)#Lending (14)#Linux (9)#LlamaIndex (8)#Load Balancing (3)#MCP (22)#MLOps (35)#Machine Learning (108)#Marketing (16)#Markets (4)#Mathematical Finance (8)#Memory (15)#Memory Management (5)#Metrics (7)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Mixture of Experts (5)#Monitoring (4)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (73)#Multimodal (3)#Multimodal AI (10)#NIM (5)#NVIDIA (8)#Networking (3)#OAuth (3)#OWASP (7)#Observability (50)#On-Device AI (8)#Ontology (8)#Open Source (7)#OpenTelemetry (5)#Operating Systems (9)#Operations (11)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (54)#People (8)#Performance (56)#Personalization (9)#Platform Engineering (9)#Policy as Code (3)#PreSales (8)#Privacy (13)#Privacy Engineering (3)#Process (4)#Product (30)#Product Management (8)#Production (21)#Programming (10)#Programming Languages (48)#Prompt Engineering (78)#Prompt Injection (14)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#Quantitative Finance (8)#RAG (69)#RBI (3)#REST (5)#Rails (16)#Reasoning Models (8)#Recommender Systems (8)#Reconciliation (3)#RegTech (8)#Regulation (10)#Reinforcement Learning (9)#Reliability (53)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (14)#Risk Management (10)#Rust (32)#SLSA (3)#SRE (23)#Sales (9)#Scalability (5)#Search (10)#Security (101)#Security Engineering (8)#Self-Evolving Agents (16)#Semantic Web (8)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Delivery (9)#Software Engineering (144)#Spanner (4)#Speech (8)#Startups (30)#Statistics (8)#Strands (8)#Strategy (6)#Streaming (32)#Structured Output (4)#Supply Chain Security (9)#Sustainability (8)#System Design (32)#Systems Programming (56)#Testing (59)#Threat Modeling (3)#Time Series (8)#Tool Use (22)#Tooling (5)#Tools (3)#Trading (8)#Treasury (6)#Type System (3)#Type Systems (11)#TypeScript (8)#Vector Databases (22)#Vector Search (11)#Venture Capital (8)#Version Control (9)#Voice AI (9)#Web Development (6)#Workflow (3)#Workflows (14)#eBPF (8)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·6 min read

Operating in Production — Cost, Latency, Incidents, and the Tightening Loop

The final discipline of LLMOps is running the thing as a product that costs real money, has to be fast enough to use, breaks in new ways, and is expected to get better over time. Cost and latency aren't afterthoughts you optimize later — in an LLM app they're first-class constraints that shape the architecture. And incidents aren't failures of planning; they're a certainty you prepare for. This closing post ties the series together into the operating posture that keeps a living system healthy.

Running an LLM app means treating cost and latency as first-class constraints (model routing, token hygiene, caching, streaming), preparing for incidents as certainties (provider failures, instant rollback, blameless postmortems), and the closing idea: operating is continuous improvement — a tightening loop, because an LLM app is never done.

#LLMOps#Cost Optimization#SRE#Production#AI Engineering#LLMs#Streaming
Pratik Dhanave · ·6 min read

Latency: The Make-or-Break Constraint

Everything about voice AI comes down to one number: how long the user waits to hear a reply. Get it under the threshold where conversation feels natural and the agent is a delight; miss it and no amount of intelligence saves the experience. This post is about the latency budget — where the milliseconds go, and how streaming the entire pipeline turns an additive delay into something that feels instant.

Everything about voice AI comes down to one number: how long the user waits to hear a reply. Get it under the threshold where conversation feels natural and the agent is a delight; miss it and no intelligence saves the experience. The latency budget — where the milliseconds go, and how streaming the entire pipeline turns an additive delay into something that feels instant. With an interactive turn diagram.

#Voice AI#Latency#Performance#AI Engineering#Conversational AI#Speech#Streaming
Pratik Dhanave · ·5 min read

Text-to-Speech: Giving the Agent a Voice

The last stage of the pipeline is where the agent finally speaks, and it's where an interaction either sounds human or sounds like a robot reading a menu. Modern text-to-speech is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and it must pronounce the messy real world correctly. This post covers TTS for real-time voice.

The last stage of the pipeline is where the agent finally speaks — and where an interaction either sounds human or sounds like a robot reading a menu. Modern TTS is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and pronounce the messy real world correctly. TTS for real-time voice.

#Voice AI#Text-to-Speech#TTS#Speech#Conversational AI#LLMs#Streaming
Pratik Dhanave · ·5 min read

The LLM Turn: The Brain in the Loop

The language model is where a voice agent stops being a transcription toy and becomes something you can actually talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: it must respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming its answer so the user isn't left waiting. This post is about the LLM stage, adapted for voice.

The language model is where a voice agent stops being a transcription toy and becomes something you can talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming so the user isn't left waiting. The LLM stage, adapted for voice.

#Voice AI#LLMs#Conversational AI#AI Engineering#Speech#Streaming
Pratik Dhanave · ·6 min read

Messages, History, and Streaming

A single agent run answers one question; a conversation needs memory, and a good user experience needs the answer to appear as it's generated. Pydantic AI handles both through its message system — the record of what was said that you pass between runs — and streaming, which delivers typed output progressively. Together they turn one-shot agents into conversational, responsive ones.

A single run answers one question; a conversation needs memory, and good UX needs the answer to appear as it's generated. Pydantic AI handles both through its message system — the record you pass between runs — and streaming that delivers typed output progressively.

#Pydantic AI#AI Agents#Python#AI Engineering#Software Engineering#Streaming
Pratik Dhanave · ·5 min read

Delivery Semantics and Exactly-Once

"Exactly-once" is the most misunderstood phrase in streaming — it is real in Kafka, but only within a specific boundary, and outside that boundary the honest and usually-correct answer is at-least-once plus idempotent consumers.

'Exactly-once' is the most misunderstood phrase in streaming — it's real in Kafka, but only within a specific boundary, and outside it the honest answer is at-least-once plus idempotent consumers.

#Kafka#Distributed Systems#Backend Engineering#Event-Driven Architecture#Streaming
Pratik Dhanave · ·6 min read

LCEL and Runnables

The pipe operator that lets you write `prompt | model | parser` is not syntactic sugar — it's LangChain's core composition model, and everything you pipe together shares one standard interface that gives you streaming, batching, and async for free. Understanding Runnables and LCEL is understanding how LangChain applications are actually built.

The pipe operator that lets you write prompt | model | parser is not syntactic sugar — it's LangChain's core composition model, and everything you pipe together shares one interface that gives you streaming, batching, and async for free.

#LangChain#AI Engineering#Python#Software Architecture#Software Engineering#Prompt Engineering#Streaming
Pratik Dhanave · ·5 min read

Streaming and Push Notifications

Long-running agent work needs a way to report progress without the client holding its breath, and A2A offers two: stream the updates live, or register a webhook and get called back.

Long-running agent work needs a way to report progress without the client holding its breath. A2A offers two: stream the updates live over SSE, or register a webhook and get called back.

#A2A#AI Agents#Streaming#APIs#Multi-Agent Systems#Protocols
Pratik Dhanave · ·11 min read

Beyond REST: GraphQL and gRPC

When REST is the wrong shape for the problem, GraphQL and gRPC each fix a different pain — and each buys that fix with a new cost you have to design around.

When REST isn't the right shape: GraphQL (client-selected fields, and its N+1 / caching / complexity costs) and gRPC (Protobuf, HTTP/2, streaming, codegen) — plus a decision framework for REST vs GraphQL vs gRPC.

#API Design#GraphQL#gRPC#Architecture#Streaming#APIs
Pratik Dhanave · ·16 min read

Asynchronous Processing and Messaging

How queues, pub/sub, and log-based streaming let systems stay responsive under load — the delivery semantics, ordering rules, backpressure, and outbox patterns that decide whether async saves you or sinks you.

Decoupling with queues and events: message queues vs pub/sub vs log-based streaming, delivery semantics (why exactly-once delivery is a myth — do idempotent processing), ordering, backpressure, the transactional outbox, and dead-letter queues.

#System Design#Messaging#Event-Driven#Kafka#Distributed Systems#Streaming
Pratik Dhanave · ·6 min read

Streaming and Backpressure

Streaming is gRPC's most powerful feature and its most misused. Sending a stream of messages sounds simple until one side produces faster than the other can consume — then, without flow control, you get unbounded memory growth and crashes. gRPC inherits HTTP/2's flow control to prevent exactly this. Understanding backpressure is the difference between streaming that scales and streaming that falls over under load.

Streaming is gRPC's most powerful feature and its most misused. Sending a stream sounds simple until one side produces faster than the other consumes — then, without flow control, you get unbounded memory growth and crashes. gRPC inherits HTTP/2's flow control to prevent exactly this. Backpressure is the difference between streaming that scales and streaming that falls over.

#gRPC#Streaming#Performance#Backend Engineering#APIs
Pratik Dhanave · ·13 min read

Bedrock Agents

How to invoke a managed Agent for Amazon Bedrock from Go — where the server owns the reason-act loop, and your job is to call InvokeAgent, range the event stream, accumulate the answer chunks, and read the trace for observability.

Agents for Amazon Bedrock from Go: the managed reason-act loop that runs server-side (vs the DIY Converse loop), invoking an agent alias with InvokeAgent, streaming the response and trace events, and keeping multi-turn state with a SessionId.

#Amazon Bedrock#AWS#AI Agents#Go#AI Engineering#Backend Engineering#Streaming
Pratik Dhanave · ·5 min read

The Four Types of RPC

gRPC isn't just request-and-response. Because it rides on HTTP/2, it offers four distinct call patterns: unary, server streaming, client streaming, and bidirectional streaming. Each fits a different shape of problem, and choosing the right one is a real design decision — it changes how your API feels, how it performs, and how it handles data that arrives over time rather than all at once.

gRPC isn't just request-and-response. Riding on HTTP/2, it offers four call patterns: unary, server streaming, client streaming, and bidirectional. Each fits a different shape of problem, and choosing the right one is a real design decision — it changes how your API feels, performs, and handles data that arrives over time rather than all at once.

#gRPC#Streaming#APIs#Backend Engineering
Pratik Dhanave · ·10 min read

Calling watsonx.ai from Python

Your first real inference calls with the ibm-watsonx-ai SDK — credentials, ModelInference, the generate and chat APIs, streaming, token usage, error handling, and the langchain-ibm path.

Make your first watsonx.ai calls in Python: credentials (IAM key + project_id + regional url), ModelInference with generate_text vs the modern chat API, decoding methods and parameters, streaming, token usage, and the langchain-ibm ChatWatsonx path.

#IBM watsonx#Granite#LLMs#Python#AI Engineering#Software Engineering#Streaming#APIs
Pratik Dhanave · ·10 min read

Calling NIM from Python

Your first real NVIDIA NIM calls in Python, two idiomatic ways: the plain OpenAI SDK pointed at NVIDIA's endpoint, and the official LangChain integration — with error handling, streaming, and the one change that makes the same code run against a self-hosted model.

Make your first NIM calls in Python two ways: the standard openai client pointed at the API Catalog with an nvapi- key, and the official ChatNVIDIA from langchain-nvidia-ai-endpoints — with streaming, token usage, typed errors, and the base_url swap to a self-hosted NIM.

#NVIDIA#NIM#LLMs#Python#AI Engineering#Software Engineering#Streaming#APIs
Pratik Dhanave · ·9 min read

Streaming and Token Usage

How to stream Amazon Bedrock responses token-by-token with the aws-sdk-go-v2 Converse API, decode the event stream with a double type-switch, and account for tokens and cost from the metadata event — accurately, in Go.

Streaming responses and accounting for tokens and cost on Bedrock in Go: ranging the ConverseStream event stream, the nested delta unions, checking stream.Err(), and computing cost from the metadata usage event with a formula you fill in.

#Amazon Bedrock#AWS#LLMs#Go#AI Engineering#Backend Engineering#Streaming
Pratik Dhanave · ·11 min read

Calling an LLM from Go

Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming — no framework required.

Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming.

#AI Engineering#LLMs#Go#APIs#Backend Engineering#Streaming
Pratik Dhanave · ·11 min read

Google ADK Glossary: Every Core Concept in One Place

The reference capstone for the 26-part series — every canonical ADK term, defined concisely.

The capstone of the series: every core ADK concept defined in one place — agents and orchestration, tools, sessions/state/memory, context and callbacks, runtime and streaming, models, grounding, evaluation, protocols, and deployment.

#Google ADK#AI Agents#Agentic AI#Evaluation#Multi-Agent Systems#Streaming
Pratik Dhanave · ·7 min read

Runtime & Events in ADK: The Runner, the Invocation, and the Event Stream

How an agent actually runs — a Runner drives an invocation and hands you back a stream of events, not a single answer.

How ADK runs an agent: the Runner drives an invocation that yields a stream of Event objects — content, tool calls, state deltas, control signals. The event loop explains streaming, callbacks, and state.

#Google ADK#AI Agents#Python#Go#Software Engineering#Backend Engineering#Streaming
Pratik Dhanave · ·7 min read

Designing a Deterministic Order-Book Matching Engine

How price-time priority, a single-threaded sequencer, and gap-free event streaming combine into an exchange core you can replay byte-for-byte.

Teaches how to build a deterministic matching engine: price-time priority order book, limit/market/IOC/FOK order types, single-threaded sequencer for determinism, and gap-free trade/execution event streaming.

#Fintech#Trading#Markets#Financial Systems#Capital Markets#Streaming
Pratik Dhanave · ·4 min read

Workflow Mechanics — Microsoft Agent Framework in Python

The graph model underneath every multi-agent app: executors as nodes, edges as data flow, and typed events streaming out as it runs.

The Microsoft Agent Framework workflow model in Python: executors as nodes, edges as data flow, switch-case routing, and typed streaming events - learned model-free.

#Microsoft Agent Framework#Python#AI Agents#Workflows#Orchestration#Software Engineering#Multi-Agent Systems#Streaming
Pratik Dhanave · ·7 min read

Batch vs Streaming

How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in large chunks on a schedule — simpler, cheaper, and fine when yesterday's data is good enough. Stream processing handles data continuously as it arrives — more complex and costly, but necessary when you need to know now. Choosing between them (and knowing when each fits) is one of the most consequential architectural decisions in a data platform, and it's driven by real requirements, not by which sounds more impressive.

How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in chunks on a schedule — simpler and cheaper. Stream processing handles data continuously as it arrives — more complex, but necessary when you need to know now.

#Data Engineering#Streaming#Backend Engineering#Distributed Systems#Data

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.