DEV Community: Neeraj Sharma

Python 3.15 lazy imports took my cold start from 512 ms to 32 ms

Neeraj Sharma — Sun, 11 Oct 2026 13:50:29 +0000

Python 3.15.0 came out on 9 October. The release notes list a dozen headline changes, but for anyone running Python behind an HTTP endpoint, a CLI or a serverless function, one matters more than the rest: PEP 810, explicit lazy imports.

There were no 3.15 builds available through my usual tooling yet, so I built it from the python.org source tarball (SHA-256 checked against the release page) and measured what lazy imports do to a realistic handler.

The test

A small handler of the kind every backend has: it imports boto3, httpx, Jinja2, Rich and SQLAlchemy at the top, but the health check path only needs json.

import json
import sys
import boto3
import httpx
from jinja2 import Environment
from rich.console import Console
from sqlalchemy import create_engine


def handler(event):
    if event["path"] == "/health":
        return {"status": 200, "body": json.dumps({"ok": True})}
    s3 = boto3.client("s3", region_name="us-east-1")
    return {"status": 200, "body": s3.meta.region_name}


if __name__ == "__main__":
    print(handler({"path": sys.argv[1]}), len(sys.modules))

The lazy version changes five lines and nothing else:

lazy import boto3
lazy import httpx
lazy from jinja2 import Environment
lazy from rich.console import Console
lazy from sqlalchemy import create_engine

I ran each case 25 times as a fresh process on a 2 vCPU Linux container, with the bytecode cache already warm, and took the median.

Case	Median	Modules loaded
Eager imports, /health	512 ms	608
`lazy` keyword, /health	32 ms	57
`-X lazy_imports=all`, /health	23 ms	53
Eager imports, /s3	614 ms	613
`lazy` keyword, /s3	361 ms	404

The health check went from 512 ms to 32 ms, 16 times faster, because Python no longer loads 551 modules it never uses on that path.

Where the time was going

python -X importtime shows the culprits. boto3 alone takes about 210 ms cumulative to import, and SQLAlchemy about 155 ms, before your code runs a single line.

python -X importtime handler.py /health 2> imports.log
sort -t'|' -k2 -n -r imports.log | head

The /s3 row shows the limit. That path really does use boto3, so it still pays for boto3. It dropped from 614 ms to 361 ms only because httpx, Jinja2, Rich and SQLAlchemy stayed unloaded. Lazy imports move the cost to the first use. They do not delete it.

Three ways to turn it on

The keyword. lazy import x or lazy from x import y, per import. Explicit and easy to review.
Globally. python -X lazy_imports=all or PYTHON_LAZY_IMPORTS=all makes every import lazy. I checked both with sys.get_lazy_imports(), which returns "all". Good for an experiment in staging, risky as a default on day one.
Across versions. A module can set __lazy_modules__ = ["boto3", "sqlalchemy"] and keep plain import statements, so the same file still runs on 3.14.

For a middle ground, combine lazy_imports=all with sys.set_lazy_imports_filter(). The filter is only consulted for imports that would be lazy, and returning False keeps a module eager, so you can make everything lazy except the modules whose import side effects you rely on.

What will bite you

Errors move. A missing module now raises at first use, not at startup. A typo in a rarely used code path can hide until production hits that path. Keep an import smoke test in CI.
Import side effects stop running. Plugin registration, model registries, monkeypatching libraries and logging configuration that happens at import time will not happen until something touches the module. Keep those imports eager.
Not everything can be lazy. lazy from x import * and lazy from __future__ import ... are syntax errors.

The other change you will hit: UTF-8 by default

PEP 686 makes UTF-8 the default encoding for open() and friends when no encoding is given, whatever the system locale says. That changes behaviour on machines whose locale was not UTF-8, which mostly means Windows. Find the affected calls before upgrading:

python -X warn_default_encoding -m pytest
# EncodingWarning: 'encoding' argument not specified

If you need the old behaviour while you fix things, -X utf8=0 restores the locale encoding. With LC_ALL=C and locale coercion off, that gave me ASCII again, which is exactly the kind of difference that corrupts a file quietly.

Upgrade checklist

Add 3.15 to the CI matrix and run the suite with -X warn_default_encoding.
Check binary wheels exist for your compiled dependencies. When I checked on 11 October, pydantic-core and psycopg-binary already had cp315 wheels on PyPI, and numpy and orjson did not:

pip download --no-deps --only-binary=:all: --python-version 3.15 \
  --platform manylinux_2_28_x86_64 --implementation cp numpy -d /tmp/whl

Profile startup with -X importtime and mark the heaviest third-party imports lazy first.
Try PYTHON_LAZY_IMPORTS=all in staging and watch for side-effect breakage before making anything global.

Does it save money?

A little, and latency matters more. AWS Lambda has billed the init phase since August 2025, so a slow import is now billed time as well as a slow first response. But the arithmetic is modest: one million cold starts a month, 0.48 seconds saved each, at 1 GB of memory, is 480,000 GB-seconds, about 8 dollars at the x86 rate of 0.0000166667 per GB-second. The real win is a first request that answers in tens of milliseconds instead of half a second, for users and for health checks with tight timeouts.

This is the kind of change I look for when I review a Python backend's cloud cost and latency at Axionry. If you are starting a new AI product build, 3.15 is worth targeting from day one.

Sources: Python 3.15.0 release, What's new in Python 3.15, PEP 810, PEP 686, AWS: Lambda standardizes billing for the INIT phase, AWS Lambda pricing. Benchmarks run on a 2 vCPU Linux container, 11 October 2026. I used an AI assistant while drafting this.

An offline voice agent for my terminal in 52 MB (Whistle + Needle)

Neeraj Sharma — Sat, 10 Oct 2026 12:29:34 +0000

Two of the biggest Hacker News threads this week were about Whistle, a 16.9 MB speech-to-text model, and DeepSeek V4.1 Flash. I wanted to see what you can build with the first one, so I put a voice front end on my terminal, with the second as a fallback.

Say "show me the last fifteen minutes of logs for the checkout service" and it prints kubectl logs deploy/checkout --since=15m. Speech recognition and tool choice both run on the CPU. Your audio never leaves the machine. If the local model returns no usable call, the transcript goes to a cloud model for one try.

The stack

Whistle (Cactus Compute, Apache-2.0): speech to text in a single 16.9 MB file. It handles English, German, French, Spanish, Italian, Dutch and Polish.
Needle, from the same package: a small tool-calling model (121M parameters, per its README). The needle3 file it downloaded for me was 35.3 MB.
DeepSeek V4.1 Flash as the fallback, used only when Needle returns no usable call.

The two local model files add up to 52 MB.

Both local models come from one package, pip install cactus-needle, which downloads the weights on first use. Add the [mic] extra for microphone input, and pip install openai for the fallback.

The code

The script has six tools, a confirmation gate and a cloud fallback. Run it with a WAV file as the argument, or with no argument to record four seconds from the mic. It prints each command instead of running it.

import json, os, sys
from typing import Literal
os.environ.setdefault("NEEDLE_TELEMETRY", "0")  # the package sends anonymous usage counts unless this is 0
import needle

Service = Literal["checkout", "billing", "payments", "search", "api"]
Env = Literal["staging", "production"]

@needle.tool
def run_tests(service: Service):
    "Run the test suite of one service."
    return ["pytest", f"services/{service}/tests"]

@needle.tool
def tail_logs(service: Service, minutes: int):
    "Show the recent log lines of a service."
    return ["kubectl", "logs", f"deploy/{service}", f"--since={minutes}m"]

@needle.tool
def scale_service(service: Service, replicas: int):
    "Change how many replicas of a service are running."
    return ["kubectl", "scale", f"deploy/{service}", f"--replicas={replicas}"]

@needle.tool
def deploy(service: Service, environment: Env):
    "Deploy the latest build of a service to an environment."
    return ["./deploy.sh", service, environment]

@needle.tool
def rollback(service: Service, environment: Env):
    "Roll a service back to its previous release in an environment."
    return ["./rollback.sh", service, environment]

@needle.tool
def git_status():
    "Show uncommitted changes in the repository."
    return ["git", "status", "--short"]

TOOLS = {f.__name__: f for f in [run_tests, tail_logs, scale_service, deploy, rollback, git_status]}
RISKY = {"deploy", "rollback", "scale_service"}
WORDS = ["checkout", "billing", "payments", "search", "staging", "production", "replicas"]

ear = needle.Whistle()
brain = needle.Needle(tools=list(TOOLS.values()), stateless=True)

def hear(wav=None):
    if wav is None:
        import sounddevice as sd  # pip install "cactus-needle[mic]"
        wav = sd.rec(4 * 16000, samplerate=16000, channels=1, dtype="float32")
        sd.wait()
        wav = wav[:, 0]
    return ear.transcribe(wav, keywords=WORDS)["text"]

def ask_cloud(text):
    from openai import OpenAI
    client = OpenAI(api_key=os.environ["DEEPSEEK_API_KEY"], base_url="https://api.deepseek.com")
    reply = client.chat.completions.create(
        model="deepseek-flash",
        messages=[{"role": "user", "content": text}],
        tools=[{"type": "function", "function": needle.build_schema(f)} for f in TOOLS.values()],
        extra_body={"thinking": {"type": "disabled"}},
    )
    calls = reply.choices[0].message.tool_calls or []
    return [{"name": c.function.name, "arguments": json.loads(c.function.arguments)} for c in calls], 0.0

def handle(text):
    plan = brain.complete(text)
    calls, sure = plan.get("function_calls") or [], plan.get("confidence") or 0.0
    if plan.get("validation", {}).get("ungrounded"):
        calls = []  # it filled in a value you never said
    if not calls and os.environ.get("DEEPSEEK_API_KEY"):
        calls, sure = ask_cloud(text)
    if not calls:
        print(f"Heard {text!r}. Not sure what to run, say it again.")
    for call in calls:
        name = call["name"]
        if name not in TOOLS:
            continue
        cmd = TOOLS[name](**(call.get("arguments") or {}))
        if (name in RISKY or sure < 0.5) and input(f"{' '.join(cmd)}  (yes to run) ") != "yes":
            continue
        print("would run:", " ".join(cmd))  # swap for subprocess.run(cmd) once you trust it

if __name__ == "__main__":
    text = hear(sys.argv[1] if len(sys.argv) > 1 else None)
    print("heard:", text)
    handle(text)

Four details matter:

Literal types become enums in the tool schema, and Needle's decoder is constrained by the schema, so it can only pick a service that exists. With plain str types it returned Checkout and search service, neither of which is a deployment name.
keywords biases Whistle toward your service names and the other words you expect to say.
The gate. Anything risky, or anything under 0.5 confidence, asks before it runs. Cloud answers carry no confidence score, so they always ask. If validation.ungrounded flags an argument value you never said, the plan is dropped.
Telemetry. By default the package sends anonymous usage counts (function name, versions, OS and a random install ID, no prompts or outputs). The script turns that off with NEEDLE_TELEMETRY=0. Even then, each model load fetches a small config file from Hugging Face, which counts as a download.

What I measured

I ran six spoken commands on a 2 vCPU cloud VM (Intel Xeon at 2.8 GHz, no GPU), five runs each, and took the median. The audio came from Piper TTS, so treat the accuracy part as a smoke test, not a benchmark.

Speech to text took 133 to 290 ms for 1.3 to 3.2 seconds of audio.
Picking the tool took 320 to 700 ms.
Startup, which loads both models, took 2.65 seconds, and the process peaked at 144 MB of memory.

Whistle misheard three of the six commands. "Show me" became "Jomy", "Scale" became "Gale", and "in" became "and". Here is what Needle made of them:

"Jomy the last 15 minutes of logs for the Checkout service" became tail_logs(checkout, 15) at 0.81 confidence.
"Gale the API service to three replicas" became the right scale_service call plus an extra run_tests, at 0.34 confidence. Both calls stop at the gate, so the extra one does not run unless I say yes.
"Run the tests and the payments folder" got no call (0.06 confidence). That is the case the cloud fallback is for.

The three clean transcripts all mapped to the right tool and arguments, at 0.64 to 0.86 confidence.

Cactus reports 11.1 ms to first token for a 10 second clip on an Apple M4 Pro CPU. My numbers cover the whole transcription on a small cloud VM, so the two do not compare directly.

What the fallback costs

The fallback sends the six tool schemas plus your sentence. DeepSeek wraps the schemas in about 210 tokens of its own tool instructions. Counted with DeepSeek's tokenizer in the prompt format published with the model, a short command comes to about 700 input tokens. The tool call that comes back is about 40 tokens with one argument and 56 with two, so call it 50.

At DeepSeek's off-peak rates for deepseek-flash (0.15 USD per million input tokens on a cache miss, 0.60 per million output):

input:  700 x 0.15 / 1,000,000 = 0.000105 USD
output:  50 x 0.60 / 1,000,000 = 0.000030 USD
total per fallback             = 0.000135 USD, about 13.5 cents per 1,000

Peak hours (01:00 to 04:00 and 06:00 to 10:00 UTC, Monday to Friday, except Chinese public holidays) cost double, so about 27 cents per 1,000. Thinking mode is on by default for this model, which is why the code sends "thinking": {"type": "disabled"}. A tool call this simple does not need a chain of thought.

Commands handled locally cost nothing per call. For a personal tool, what matters more to me is that the audio stays on the machine and the local path works with no network at all.

Where it breaks

Seven languages only.
It records a fixed four seconds per run. Whistle has a streaming mode (Whistle.stream) that I have not wired in yet.
Needle picks tools. It is not a chat model, so it will not answer questions.
I only tested synthetic speech. Real voices, accents and room noise will change the error rate, so test it with your own voice before you trust it near production.

A phone agent is a different problem: it has to listen continuously and answer fast enough to hold a conversation. The production voice agent I built runs on LiveKit at about 2.5 cents a minute. If you are picking speech to text for something real-time, I compared the main options in the best STT for voice agents in 2026, and the AI voice agent page covers what we build for clients.

Sources: Cactus Compute's Whistle announcement and the Cactus-Compute/whistle model card, the cactus-needle README and source, DeepSeek's API pricing and thinking mode docs, and the DeepSeek-V4.1-Flash model card, tokenizer and prompt encoding reference. Latency and accuracy numbers are from my own runs on 10 October 2026.

I used an AI assistant while drafting this.

Find the AWS waste in one afternoon: 7 read-only CLI checks

Neeraj Sharma — Fri, 09 Oct 2026 14:33:06 +0000

If you're at a seed or Series A company, nobody owns the AWS bill until it gets big enough to scare someone. By then the waste has been sitting there for months. It's rarely one big mistake. It's a pile of small things: a volume nobody deleted, a load balancer from a demo, logs kept forever.

This is the afternoon audit I run first on a startup's AWS account. Every check is a read-only CLI command, and each one comes with what it costs if you find something. Prices below are us-east-1 list prices. Check your own region before quoting numbers to anyone.

Before you start

Use a role with read-only access (the ReadOnlyAccess managed policy covers everything below; if the Cost Explorer call returns AccessDenied, add ce:GetCostAndUsage). Most resources are regional, so run each check in every region you use:

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  echo "== $r"
  # paste any check below here, with --region "$r"
done

0. Where the money actually goes

Start with last month's bill by service so you know which checks matter most:

aws ce get-cost-and-usage \
  --time-period Start=2026-09-01,End=2026-10-01 \
  --granularity MONTHLY --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' \
  --output table

Cost Explorer API calls cost $0.01 each, so run this once, not in a loop.

1. Volumes attached to nothing

An EBS volume in the available state isn't attached to any instance, but you still pay for every gigabyte.

aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' --output table

At $0.08 per GB-month for gp3, 500 GB of forgotten volumes is $40 a month. Snapshot anything you're unsure about, then delete the volume.

2. gp2 volumes that should be gp3

gp3 costs $0.08 per GB-month against $0.10 for gp2, which is 20% cheaper, and its baseline performance is at least as good for most workloads. The change happens online, without detaching anything.

aws ec2 describe-volumes --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[].[VolumeId,Size,Iops]' --output table

# per volume, after checking its IOPS needs:
aws ec2 modify-volume --volume-id vol-0123456789abcdef0 --volume-type gp3

2,000 GB moved from gp2 to gp3 saves 2,000 × 0.02 = $40 a month. Before you convert a large gp2 volume, look at its IOPS: gp2 gets 3 IOPS per GB, and gp3 starts at a flat 3,000.

3. Snapshots nobody will restore

aws ec2 describe-snapshots --owner-ids self \
  --query "Snapshots[?StartTime<='2026-04-01'].[SnapshotId,VolumeSize,StartTime,Description]" \
  --output table

Standard snapshot storage is $0.05 per GB-month. VolumeSize overstates the real cost because snapshots are incremental, so read the "EBS:SnapshotUsage" line in Cost Explorer for the true figure. For snapshots you must keep but won't touch, the archive tier is $0.0125 per GB-month, with a 90-day minimum and a fee to restore.

4. Public IPv4 addresses

Since February 2024 every public IPv4 address costs $0.005 an hour, whether it's in use or idle. That's 0.005 × 730 = $3.65 a month each. Start with Elastic IPs that aren't associated with anything:

aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==null].[PublicIp,AllocationId]' --output table

Then list every network interface with a public IP. You'll often find instances in public subnets that don't need one:

aws ec2 describe-network-interfaces \
  --query 'NetworkInterfaces[?Association.PublicIp].[NetworkInterfaceId,Association.PublicIp,Description]' \
  --output table

5. Load balancers with no traffic

An Application Load Balancer costs $0.0225 an hour before any traffic, so 0.0225 × 730 = $16.43 a month just for existing. Find the ones that served no requests in the last 14 days:

for arn in $(aws elbv2 describe-load-balancers --query 'LoadBalancers[?Type==`application`].LoadBalancerArn' --output text); do
  name=$(echo "$arn" | cut -d/ -f2-4)
  sum=$(aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB \
    --metric-name RequestCount --dimensions Name=LoadBalancer,Value="$name" \
    --start-time "$(date -u -d '14 days ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
    --period 1209600 --statistics Sum --query 'Datapoints[0].Sum' --output text)
  echo "$name $sum"
done

None or 0 means nothing has hit it in two weeks. On macOS, use date -u -v-14d instead of -d '14 days ago'.

6. Log groups that keep everything forever

New CloudWatch log groups default to "never expire". Storage is about $0.03 per GB-month, but ingestion at $0.50 per GB is usually the bigger number. This lists groups with no retention and their stored size; for ingestion, check each large group's IncomingBytes metric:

aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' --output table

# set a retention period per group:
aws logs put-retention-policy --log-group-name /aws/lambda/my-fn --retention-in-days 30

If one group is ingesting tens of GB a day, the fix is less logging, not shorter retention.

7. NAT gateways doing more work than they should

A NAT gateway costs $0.045 an hour (0.045 × 730 = $32.85 a month) plus $0.045 for every GB it processes. The usual culprit is S3 traffic (ECR image layers included, since they're pulled from S3) that should go through a free gateway endpoint. I wrote up that fix separately: why your NAT gateway bill keeps growing.

aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,VpcId,State]' --output table

What a typical first pass adds up to

Here's an illustrative example of findings, not a specific account:

Finding	Monthly saving (USD)
500 GB of unattached gp3 volumes	40.00
2,000 GB of gp2 moved to gp3	40.00
8 idle public IPv4 addresses	29.20
2 load balancers with no traffic	32.85
3 TB of old standard snapshots	150.00
Total	292.05 USD a month

That's about 292.05 × 12 = 3,505 dollars a year from an afternoon of reading. None of this needs a reserved instance or a Savings Plan. Those are bigger levers, but they lock you in, so do this cleanup first and commit to the smaller bill afterwards.

Make it stick

Tag everything with an owner and a purpose. Untagged resources are the ones nobody deletes.
Turn on AWS Cost Anomaly Detection. It's free, and it emails you when a service jumps.
Put a monthly calendar reminder on someone's name to run this list again.

If you want a rough number before you start, our cloud cost calculator estimates how much of a bill is usually recoverable. For a full review across AWS, GCP and Azure, that's the cloud cost optimization work I do at Axionry.

Prices: AWS EBS, VPC, Elastic Load Balancing and CloudWatch pricing pages (us-east-1), checked October 2026. I used an AI assistant while drafting this.

What an investor's engineer checks in your codebase before a round

Neeraj Sharma — Thu, 08 Oct 2026 12:31:39 +0000

If your startup is raising, at some point someone like me gets read access to your repo, your cloud account and a call with your lead engineer. I do technical due diligence for investors, and the review is more predictable than founders think. It is not a code style audit. It answers one question: is the technology worth what the deck says, and what will it cost to keep it working?

Here is what I actually look at, and the commands I would run on your repo first.

1. Architecture and scaling

I want to know what breaks at 10x the current load and how expensive the fix is. A monolith is fine. A monolith where every request does a full table scan is not.

What I ask for: a one-page architecture diagram, the three slowest endpoints, and the current peak load. If nobody can draw the diagram, that is the finding.

2. Delivery

How code gets from a laptop to production tells me more than the code itself.

# the 20 most recent release tags with dates, if you tag releases
git tag --sort=-creatordate --format='%(creatordate:short) %(refname:short)' | head -20

# commits in the last 90 days, by author
git shortlog -sn --since="90 days ago" HEAD

I look for CI on every pull request, tests that run, and a rollback that someone has actually done. Weekly or faster deploys are a good sign. "We deploy when Raj is around" is not.

3. Security basics

This is where most fixable red flags live, and you can find them yourself in an afternoon.

# secrets committed anywhere in git history, all branches
gitleaks git -v --log-opts="--all" .

# known vulnerable dependencies across lockfiles
osv-scanner scan source -r .

I also check who has production access, whether it is behind SSO and MFA, and whether customer data is encrypted at rest. If you sell to enterprises, I check how far you are from SOC 2, because the next buyer will ask.

4. Infrastructure cost

I open the cloud bill and divide it by customers. Gross margin problems hide here: a product that costs $40 a month to serve a customer paying $50 is a different company from one that costs $4.

Common findings are idle databases, NAT gateway data charges and logs nobody reads. A cleanup here often improves margin before the round closes, which is worth doing on its own.

5. AI defensibility (if you are an AI company)

The question is what is left if a competitor calls the same model API tomorrow. I look for:

proprietary data or feedback loops the model gets better from
an evaluation set and a record of how quality changed between model versions
cost per request, and what happens to margin if the model price doubles
how hard it would be to switch model vendor

A thin prompt over a single API is not automatically a red flag, but the deck should not call it a moat.

6. Team and bus factor

# who actually knows the payments code
git shortlog -sn --since="12 months ago" HEAD -- src/payments

If one name owns 90% of a critical directory and that person is a contractor, it goes in the report. Documentation, runbooks and an on-call rotation all reduce this risk.

How findings get graded

Nothing is pass or fail. Each finding gets a grade:

Green: fine as is.
Amber: needs work, but the cost is known and fits in the plan. Most findings land here.
Red: expensive or risky enough to change the valuation or the terms, such as leaked customer data, no backups, or a core system only one departed engineer understood.

Investors expect ambers. What hurts a round is a red the founders did not know about.

A one-week prep list

Run the commands above and fix what they find.
Draw the architecture on one page.
Write down your cloud cost per customer.
Make sure two people can deploy and roll back.
Turn on MFA everywhere production lives.

I keep a longer technical due diligence checklist on our blog, and if you are an investor who needs an independent review of an AI or SaaS company, that is the technical due diligence work I do at Axionry.

Showing live voice agent transcripts in a Flutter app with LiveKit

Neeraj Sharma — Fri, 02 Oct 2026 13:53:20 +0000

If you build a voice agent with LiveKit Agents and a Flutter frontend, at some point you want the conversation on screen: what the user said and what the agent is saying, updating live. The docs cover it, but the pieces are spread out and the older API in many examples is deprecated. Here's what works with the current SDKs.

Where the text comes from

When AgentSession runs STT, it publishes transcriptions to the room on the lk.transcription text stream topic. The agent's own speech goes out on the same topic, synced with audio playback, so it can show up word by word as the agent talks. If the user interrupts, the agent text is cut to what was actually spoken.

The sender identity on each stream is the participant who was transcribed, so you can tell user and agent apart without extra metadata.

The older TranscriptionReceived event and publish_transcription() are deprecated. They use a separate delivery path, so anything written to lk.transcription never reaches a TranscriptionEvent listener. Build on text streams.

The Flutter handler

import 'dart:convert';

final transcript = <String, ({String speaker, String text, bool isFinal})>{};

void listenForTranscripts(Room room) {
  final me = room.localParticipant?.identity;

  room.registerTextStreamHandler('lk.transcription',
      (TextStreamReader reader, String participantIdentity) async {
    final info = reader.info!;
    if (info.attributes['lk.transcribed_track_id'] == null) return; // chat message, not a transcript

    final segmentId = info.attributes['lk.segment_id'] ?? info.id;
    final speaker = participantIdentity == me ? 'You' : 'Agent';
    var text = '';

    void update(bool isFinal) {
      transcript[segmentId] = (speaker: speaker, text: text, isFinal: isFinal);
      onTranscriptChanged(); // setState, notifyListeners, etc.
    }

    try {
      await for (final chunk in reader) {
        text += utf8.decode(chunk.content);
        update(false);
      }
    } catch (_) {
      return; // stream aborted, e.g. during a reconnect
    }

    // read this after the stream closes: agent streams set it in the trailer
    update(info.attributes['lk.transcription_final'] == 'true');
  });
}

The docs example uses reader.readAll(). That waits for the stream to close, and an agent segment stays open until the agent stops talking, so the whole reply lands at once. Reading chunks gives you the word-by-word version.

Interim vs final

User speech and agent speech arrive differently. For user speech you get a new stream for each interim result, then a final one, all with the same lk.segment_id. Key your map by that id and replace the entry every time, or the same sentence shows up three times.

Agent speech is one stream per segment that grows as the agent talks. It opens with lk.transcription_final set to false and flips it to true in the trailer when it closes, which is why the handler checks it after the loop.

If you only want finished sentences, skip entries where isFinal is false.

Register once

registerTextStreamHandler throws if a handler for that topic is already set. In a widget that can rebuild or reconnect, register once after connecting and call room.unregisterTextStreamHandler('lk.transcription') in dispose.

If the transcript still feels laggy after this, look at end-of-turn detection on the agent side. I wrote about tuning that in turn detection and barge-in for voice agents, and if you are estimating what a production voice agent costs per minute, there's a voice AI cost calculator on our site.

The code was checked against the client-sdk-flutter source and the LiveKit docs. Originally published on Axionry Engineering.

Estimate your AI feature's model bill before you build it

Neeraj Sharma — Fri, 02 Oct 2026 11:42:19 +0000

Most AI MVP budgets I see cover the build and forget the monthly line. That line is the model bill, and at a few thousand users it can pass what the build cost.

One formula gets you a usable estimate.

The formula

monthly cost = requests per month x (input tokens x input price + output tokens x output price)

Requests per month is users times how often they use the feature. Input tokens include everything you send: system prompt, retrieved context, chat history. Resending all of that on every request is why input usually dominates.

A worked example

10,000 monthly users, 40 requests each, 4,000 input and 400 output tokens per request. That's 400,000 requests, 1.6B input tokens and 160M output tokens a month.

Model (list price per 1M tokens)	Input	Output	Monthly
Claude Sonnet 5 ($2 / $10)	$3,200	$1,600	$4,800
Gemini 3.1 Flash-Lite ($0.25 / $1.50)	$400	$240	$640

Same feature, 7.5x apart.

Routing and caching

Routing. Most requests don't need the big model. Send the easy 80% to Flash-Lite and the hard 20% to Sonnet, and the $4,800 bill drops to about $1,470.

Caching. If 3,000 of those 4,000 input tokens are the same system prompt and reference docs every time, cache them. Gemini 3.1 Flash-Lite bills cached input at $0.025 per 1M instead of $0.25, which takes the Flash-Lite bill from $640 to about $370 a month. Storage is $0.50 per 1M tokens per hour, so keeping a 3,000-token cache alive all month adds about $1.

Both are build decisions. Adding them after launch means rewriting the request path.

What I'd do before writing code

Write down users, requests per user and token counts per request, even rough ones.
Run the formula for a big and a small model.
If the big-model number scares you, design routing and caching in from day one.

If you want the build cost and the run cost side by side, our AI product cost estimator does both, including the monthly bill at your user count. I also wrote up how we scope and price AI product development in checkpoints.

Prices are list prices from the Anthropic and Google pricing pages, October 2026.

Our voice agent costs 2.5¢ a minute. Here's the whole bill.

Neeraj Sharma — Fri, 02 Oct 2026 11:38:48 +0000

We run an AI interviewer that talks to candidates over phone calls. It started on a managed voice platform at roughly 10¢ a minute. The same interviews now run on a custom LiveKit pipeline at about 2.5¢ a minute, all in. For reference, Retell's pricing calculator defaults to 11¢ a minute and LiveKit Cloud's own calculator to about 4.8¢.

The stack

Telnyx SIP trunk into LiveKit SIP
Silero VAD plus LiveKit's turn detector (v1-mini) running on CPU
xAI streaming STT, Gemini 2.5 Flash-Lite, xAI TTS
Self-hosted agent workers and Redis for session state

Where the money goes

Per minute of an inbound call, at published list prices, assuming 600 characters of agent speech and about 3,000 input / 175 output tokens a minute:

Line item	Rate	Per minute
xAI TTS	$15 per 1M characters	$0.0090
LiveKit SIP fee (Ship plan)	$0.004 per minute	$0.0040
xAI STT (streaming)	$0.20 per hour	$0.0033
Telnyx inbound local	$0.0032 per minute	$0.0032
Gemini 2.5 Flash-Lite	$0.10 in / $0.40 out per 1M tokens	$0.0004
Turn detection (local CPU)	none	$0.0000
Workers, Redis, logs	self-hosted	~$0.0051
Total		~$0.025

A few things I didn't expect:

TTS is the biggest line. Almost a cent a minute, billed by character. Shorter agent turns save more than any model swap would.

The LLM barely matters for cost. Four hundredths of a cent. Pick it for latency and quality, not price.

Rate cards only get you to 2¢. The last half cent is workers, Redis and logging. No vendor lists that.

When it isn't worth building

On per-minute cost alone, building wins easily. Count engineer time and it gets closer. With about $600 a month of fixed infra and two engineer-days a month of upkeep (at $150k a year, 260 working days), a custom pipeline only beats an 11¢ vendor above roughly 20,600 minutes a month. Below that, stay on a managed platform and spend the time on the product.

The full migration, with the numbers behind the 75% cut, is in our 2.5¢ voice AI case study. If you want to run your own volumes, there's a voice AI cost calculator on our site.

Rates are from the xAI, Google, Telnyx and LiveKit pricing pages. Originally published on Axionry Engineering.

Why your NAT gateway bill keeps growing (and the 20 minute fix)

Neeraj Sharma — Fri, 02 Oct 2026 08:22:14 +0000

If your workloads run in private subnets, a big part of your NAT gateway bill is probably traffic to S3. The fix takes a few minutes and costs nothing.

NAT gateway pricing in us-east-1 is $0.045 per hour plus $0.045 per GB processed. The hourly part is small. The per-GB part grows with every image pull, backup and log upload that leaves a private subnet.

Check if this is you

In Cost Explorer, filter Service to "EC2-Other" and group by Usage Type. Look for NatGateway-Bytes. In us-east-1 it has no prefix; other regions get one, like USW2-NatGateway-Bytes. If it's near the top, keep reading.

To see where the bytes go, turn on VPC Flow Logs for the NAT gateway's network interface and sum bytes by destination. If most destinations fall in the S3 ranges from ip-ranges.amazonaws.com/ip-ranges.json, you found it.

Fix 1: S3 and DynamoDB gateway endpoints (free)

Gateway endpoints for S3 and DynamoDB have no hourly or data charge. Create one and attach your private route tables:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0abc123 \
  --vpc-endpoint-type Gateway \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-0priv1 rtb-0priv2

It only covers buckets in the same region. Creating it drops open TCP connections to S3, so don't do it mid-backup, and check that your clients reconnect. S3 will also see your private IPs instead of the NAT's public IP, so bucket policies that allow by source IP need updating. ECR pulls get cheaper too, because image layers are served from S3.

Fix 2: interface endpoints, but only above break-even

ECR API, STS, CloudWatch Logs and most other services need interface endpoints instead. These are not free: $0.01 per hour per AZ plus $0.01 per GB. Across two AZs that is $14.60 a month (2 x 730 hours x $0.01) before any traffic.

Every GB you move off NAT saves $0.035 ($0.045 minus $0.01). So one endpoint in two AZs pays for itself at about 420 GB a month. Below that, leave the traffic on NAT.

Fix 3: watch cross-AZ traffic to the NAT

With one NAT gateway for the whole VPC, traffic from the other AZs pays inter-AZ transfer just to reach it: $0.01/GB in each direction, so $0.02 for every GB that crosses. A second NAT gateway costs about $33 a month in hours, so it pays off once roughly 1.6 TB a month crosses from that AZ.

Do fix 1 first. It's free and it's usually the biggest chunk.

The full list of leaks I check (NAT, egress, idle EBS, public IPv4 and more) is in this AWS cost optimization checklist, and there's a free cloud cost calculator if you want to plug in your own numbers.

Prices are us-east-1 list prices.