DEV Community: greg Tham

EP05: How Simulcast works and what it's for

greg Tham — Thu, 08 Oct 2026 23:03:03 +0000

Episode 5 of 20 in the **PPCDN Low-Latency Live Streaming Course* — building a self-hosted, low-latency live-streaming CDN from protocols to global delivery. Originally published on the PPCDN docs site; see the full course index for all episodes.*

Recap	EP04 "Transport showdown: RTMP / SRT / WHIP" covered how the three ingest protocols handle loss
Next	EP06 "Publish/play security: signing, tokens and anti-hotlinking"

0. Goals of this episode

By the end, the viewer should be able to:

Explain Simulcast's underlying mechanism: why "one publish, multiple quality layers" needs no server transcode, and who encodes the layers, where, and how.
Explain why Simulcast, not SVC — not an arbitrary choice but a trade-off set by the codec ecosystem today.

1. Opening hook (script notes)

EP02 said "no transcoding on the network side". EP04 covered the ABR degradation logic
"lower bitrate first, then resolution". This episode fills the missing link between them:
how multiple layers are produced at the sender, and how the server switches quality layers
without doing pixel-domain transcoding.

2. What Simulcast is: one publish, multiple independent quality layers at once

Simulcast is not "one encode that the server crops into layers" — the sender produces
multiple independently decodable encodings at the same time. For example, a 1080p input with
3 simulcast layers can produce three independent encodings (1080p, 720p, 360p) sent as separate
RTP streams. A product may implement this with multiple encoder instances, or with a
hardware/software encode pipeline that supports multiple outputs; "you must run several
independent encoder instances" is not a standard requirement.

Compare the traditional way to get multiple layers without Simulcast: server-side transcoding
— the server fully decodes the incoming stream to raw pictures, then re-encodes it once per target
layer. Decode + multiple re-encodes is a heavy compute load and naturally adds processing latency.

Simulcast moves that work from "decode then encode on the server" to "encode several copies at
the publisher", so each layer the server receives is already a complete picture it can forward
directly to viewers — no decode, no re-encode, just byte forwarding. That's the technical
root of EP02's "no transcoding on the network side": the transcoding work didn't vanish; it moved
to the publisher's encoder, trading publisher encode cycles for server transcode cycles.

3. Why Simulcast, not SVC

Someone who knows codecs may ask: isn't Scalable Video Coding (SVC) more bitrate-efficient? Why
not SVC?

SVC's idea: the encoded output contains a base layer plus one or more temporal/spatial/ quality enhancement layers, with dependencies between layers. A forwarder can select a subset to forward according to the dependency structure. It is usually more bitrate-efficient than multiple fully independent encodes, but encoding complexity, actual gains and sender compute cannot be generalized.
The cost is complexity and compatibility: the SFU/Edge must understand and correctly handle layer identifiers, dependencies and switch points, and the receiver must support the matching codec/profile/scalability mode. VP9 and AV1 commonly use SVC modes in WebRTC; whether H264/HEVC is usable depends on browser, system decoders, negotiation and product implementation — a product's current state is not a standard prohibition.
Simulcast's encodings are independently decodable: WebRTC commonly uses RID, SSRC and SDP/RTP mapping to distinguish encodings; RID does not necessarily exist in every protocol and container. The forwarder still has to maintain RTP sequence, timestamps, RTCP feedback and layer-switch state — it's not unconditionally "just byte forwarding".

This project currently centers on H264/HEVC and chooses Simulcast based on target browsers and
existing sender/receiver capabilities. This is PPCDN's current product compatibility trade-off,
not the H264/HEVC standard excluding SVC, nor the only choice for every WebRTC product. For
selection, go by your target device matrix and interoperability tests.

4. How to use it: layering, ABR-safe switching, and multitrack as a separate dimension

Layer cap: PPCDN currently limits H264 to 4 layers and HEVC to 3 ("HEVC simulcast's total layer count maxes out at 3 — it must stay below 4; H264 keeps its existing layer-count cap"); this is not a universal cap defined by Simulcast, RTP or the codec standards, but a product trade-off under this project's current encoder pipeline and device matrix. Each encoding needs a distinguishable RTP identifier and independent state; whether that's RID, SSRC or another mapping depends on negotiation and implementation.
How ABR switches layers: the server can step up/down based on receiver feedback and available bandwidth estimates. Switching to another independent encoding usually waits for the target layer's random-access point; for this project's H264/HEVC streams, the common approach is to request and wait for an IDR/keyframe, then handle RTP timestamp and sequence continuity. A keyframe may need to be requested via PLI/FIR; generating one costs bitrate and has wait time, so a correct switch avoids broken reference chains and artifacts but does not guarantee zero wait, zero stutter or that it is always imperceptible.
Multitrack (H264/HEVC dual streams) is a separate dimension: multitrack answers "give browsers with different decode capability H264 or HEVC respectively", while Simulcast answers "give different network/quality needs different layers". They can run together: H264 and HEVC each maintain their own simulcast layer count, RID sequence and encoder, independently (e.g. H264 with 4 layers while HEVC independently runs 3).

5. Diagrams

5.1 Server transcoding vs Simulcast: where the transcode work went

[Traditional: server transcoding]
Publisher ──single encoded stream──▶ Server
                                       │  decode to raw pictures
                                       ▼
                                  re-encode per target layer (1080p / 720p / 360p)
                                       │  CPU-heavy, adds processing latency
                                       ▼
                                  deliver to different viewers

[Simulcast]
              ┌─ encoder A: 1080p ──┐
Publisher ────┼─ encoder B: 720p  ──┼──▶ Server (byte forwarding; no decode/encode) ──▶ deliver to viewers
              └─ encoder C: 360p  ──┘
        (three encodings run at the publisher, independent and directly decodable)

5.2 Simulcast layer structure

Publisher encoder
   │
   ├── main encoder (e.g. 1080p) ──▶ RTP stream, RID = "high"
   ├── scaled-layer encoder (720p)──▶ RTP stream, RID = "mid"
   └── scaled-layer encoder (360p)──▶ RTP stream, RID = "low"

The three streams are independent and fully decodable; the server identifies them by RID and forwards the chosen layer

5.3 ABR switches safely at keyframe boundaries

Viewer currently plays the "mid" layer (720p)
        │
        ▼
Server sees rising loss, decides to drop to "low" (360p)
        │
        ▼
Request or wait for a usable keyframe (IDR here) on the "low" layer
        │
        ▼
From that IDR on, forward the "low" layer's data
        │
        ▼
Goal: avoid artifacts; whether it's imperceptible depends on keyframe wait, buffering and RTP continuity handling

6. Wrap-up and next episode (script notes)

This episode broke "one publish, multiple layers, no server transcoding" down to the mechanism:
Simulcast has the sender produce multiple independent encodings; the server needn't do pixel-domain
transcoding but still handles RTP/RTCP and switch state. PPCDN chose it over SVC as a product
trade-off for today's device matrix. Switching to an independent layer should start from a usable
random-access point and handle feedback and RTP continuity correctly; a keyframe boundary alone is
not unconditional seamlessness.

Next episode, we move from "how to encode" to "how to defend" — publish/play signing, token
mechanisms and anti-hotlinking.

EP04: Transport showdown — RTMP / SRT / WHIP

greg Tham — Thu, 08 Oct 2026 22:57:52 +0000

Episode 4 of 20 in the **PPCDN Low-Latency Live Streaming Course* — building a self-hosted, low-latency live-streaming CDN from protocols to global delivery. Originally published on the PPCDN docs site; see the full course index for all episodes.*

Recap	EP03 "Content sovereignty: control it yourself, the ultimate moat" wrapped up Module 0 (breaking out)
Next	EP05 "How Simulcast works and what it's for"

0. Goals of this episode

By the end, the viewer should be able to:

Explain the concrete loss/latency trade-off mechanisms of RTMP, SRT and WHIP, rather than the coarse "TCP bad, UDP good" conclusion.
Understand why this project keeps WHIP and SRT side by side instead of picking one and dropping the other — they solve different problems.

1. Opening hook (script notes)

Module 0 covered three dilemmas and the overall architecture. From this episode we enter
Module 1, back to the basics — ingest protocols. EP01 already covered RTMP/HTTP-FLV's
head-of-line blocking on weak networks; this episode digs one level deeper: how do RTMP,
SRT and WHIP actually handle "loss"? And why does this project support both SRT and WHIP
rather than keeping just one?

2. RTMP: a retiring protocol still worth understanding

RTMP (Real-Time Messaging Protocol) usually runs over a persistent TCP connection; after the
handshake, messages are split into chunks multiplexed over several chunk streams. AMF is mainly
used to serialize objects for command and data messages; audio/video messages carry the
respective encoded payload, so it's wrong to say flatly that "media is AMF-encoded". RTMPS
carries RTMP over TLS, giving the link confidentiality and integrity; it does not remove TCP
head-of-line blocking.

This design made sense in its era: TCP's reliable delivery saved the protocol from handling
retransmission itself. But the cost is exactly the head-of-line blocking from EP01 — if any
TCP segment is lost, later data that already arrived must queue until it is retransmitted.
That wasn't a problem when RTMP was designed (typical scenarios were stable wired uplinks), but
on today's mobile/weak networks it's a handicap.

RTMP's place in the ecosystem is thin now: Flash's end removed it from the playback side (EP01),
leaving only "ingest entry" — some old encoders/publishing tools still speak only RTMP. Even
that is being replaced: SRT and WHIP both offer ingest better suited to modern networks, which
is why this project's public ingest protocol list has no RTMP.

3. SRT: a modern choice designed for weak networks

SRT (Secure Reliable Transport) sits on UDP and targets low-latency reliable transport over
unstable networks. Understand it by separating three mechanisms:

ARQ: the receiver detects missing packets by sequence number and sends NAK; the sender retransmits while data is still timely. It solves "how to recover loss".
TSBPD: schedules delivery based on send timestamps and the negotiated latency, absorbing jitter and restoring send timing. It solves "when to deliver" and is not a synonym for the ARQ window.
TLPKTDROP / stale-packet drop: when enabled and its conditions are met, abandons data that can no longer be delivered on schedule, letting both ends skip stale packets. It solves "when to stop remediating", and should not be written as "TSBPD drops packets automatically".

The latency parameter gives retransmission and jitter absorption a time budget: larger usually
improves recovery chances and adds delivery wait; smaller does the opposite. But it is not a
strict mathematical latency bound for the whole SRT session: handshake, network queueing,
application buffering, configuration differences, disconnects and reconnects all lie outside it.
For this project, use end-to-end instrumentation as the source of truth rather than summing
config values to derive total latency.

SRT can be configured with AES-based payload encryption. The exact mode and key length depend on
protocol version and implementation configuration; it protects the transport link and is not
automatically end-to-end encryption across every intermediate processing node.

SRT's core question is "can a weak uplink still deliver the picture intact" — trading latency
for robustness is its design intent, not a defect.

4. WHIP: standardizing WebRTC into an ingest protocol

WHIP (WebRTC-HTTP Ingestion Protocol) is not a brand-new transport but a standardized "publish
signaling procedure" added to WebRTC (using HTTP for the SDP offer/answer exchange); media still
travels the native WebRTC stack:

ICE opens the network path (including NAT-traversal address negotiation).
DTLS-SRTP encrypts the media on the WebRTC peer connection. If media is processed or re-encrypted after SRTP terminates at an SFU/Edge, it's hop-by-hop protection, not E2EE only decryptable by the communicating parties; true E2EE needs additional frame-level media encryption and key distribution.
RTP carries media, with NACK (selective retransmission requests) and optional FEC handling loss, while congestion control (e.g. GCC) adjusts the send bitrate.

WHIP has no SRT-style TSBPD. RTP implementations usually keep a receive/reorder history counted
in packets or bytes to detect reordering, generate NACKs or hold retransmittable data; e.g.
this project's WebRTCInboundRTPBufferSize is a packet capacity. A packet cache is not a time
window: the same 512 packets cover different durations at different bitrates, packet sizes and
frame rates, and it can't be used to derive playback wait directly. Actual waiting is also
determined by the jitter buffer, NACK policy, decode deadline and congestion control.

WHIP was published as RFC 9725 (2025), no longer an IETF draft. It standardizes WebRTC
ingestion's HTTP signaling and resource lifecycle; media security, congestion control and loss
recovery still come from the WebRTC/RTP system.

5. Side-by-side comparison

Dimension	RTMP	SRT	WHIP
Transport	TCP	UDP + ARQ	UDP + RTP (NACK / FEC / congestion control)
Loss handling	TCP must retransmit and deliver in order; possible head-of-line blocking	ARQ recovery; TSBPD timed delivery; configured stale-drop can skip late data	NACK/FEC used per negotiation and implementation; jitter buffer and decode deadline decide waiting
Latency bound	No strict math bound	`latency` is an engineering budget, not an unconditional end-to-end bound	Dynamic buffering and congestion control; no unconditional math bound
Encryption	RTMP itself unencrypted; RTMPS over TLS	Configurable AES payload encryption	Mandatory DTLS-SRTP link encryption; not automatically E2EE
Native browser support	No longer (relies on the discontinued Flash)	No	Yes (standard WebRTC APIs)
Typical role	Legacy, exiting	The modern weak-uplink choice	Mainstream real-time publish/play protocol
Supported here?	No	Yes (ingest)	Yes (ingest + playback all via WHEP)

6. How this project chooses: unify the signal, differentiate the handling

WHIP and SRT are not "who replaces whom" here but provided together for different ingest
scenarios: prefer WHIP when the network is clean and you want the lowest latency; use SRT and
trade a little latency when the uplink is unstable and you need more robustness.

The two receive mechanisms are not equivalent: SRT's TSBPD schedules delivery by timestamp,
while an RTP packet cache usually keeps history by packet/byte count. This project can normalize
both sides into an Unrecoverable Loss Rate (ULR) for product policy, but that is not a
standard metric jointly defined by SRT and WebRTC. The computation must also unify sampling
window, denominator, retransmission de-duplication and counter-reset semantics — you can't
compare directly just because field names look similar.

This unified signal already backs a single degrade state machine shared by WHIP and SRT in this
project: the raise threshold degradeRaisePct defaults to 0.8%, the recovery threshold
degradeLowerPct defaults to 0.3%, the sampling window degradeSampleSec defaults to 6 seconds,
and the cooldown after each state transition, degradeObservationSec, defaults to 60 seconds.
These are configuration defaults, not an SLA guaranteed network-wide — production thresholds
still need calibration against real link measurements.

Degradation is two steps, cost increasing:

Lower the encoding bitrate first: takes effect in real time, no interruption.
Only if bitrate is at the floor and still short, lower resolution (drop simulcast layers): requires restarting the publish, about 1 second of interruption.

The logic is straightforward: bitrate changes are free and can trigger often; layer changes cost
an interruption and are used only when "even the floor isn't enough". The "about 1 second of
interruption" figure today is mainly an SRT-side production observation — mmx carries a dedicated
log marker, SRT publish resumed on path X after <gap> of ingest interruption, used to measure
actual reconnect time — not a value promised across every network environment; the WHIP side still
lacks a production measurement at the same granularity for its restart cost.

7. Diagrams

7.1 Loss handling across the three protocols

[RTMP / RTMPS (TCP)]
loss → TCP must retransmit → in-order delivery → later bytes queue → wait can grow unbounded

[SRT (UDP)]
ARQ: detect a gap → NAK → retransmit while still timely
TSBPD: schedule delivery by timestamp + latency
TLPKTDROP: when enabled and a packet is stale, skip data that can't be delivered in time

[WHIP (WebRTC/RTP, typically UDP)]
sequence numbers and RTCP feedback: detect loss → NACK/FEC can attempt recovery
jitter buffer: handle jitter and playout timing against a dynamic target
decode deadline policy: drop frames and keep playing once recovery is worthless
Note: a packet-count RTP cache is not a millisecond window

7.2 Unify the decision semantics, not the raw counters

SRT native stats                        RTP/WebRTC native stats
loss / retrans / drop / too-late        sequence gap / late / RTX / FEC
         │                                      │
         └──────────────┬───────────────────────┘
                        ▼
  Derive a product metric from window, original-packet identity, recovery result and playout deadline
                        │
          ┌─────────────┴─────────────┐
          ▼                           ▼
  Metrics keep degrading →         Metrics recover steadily →
  lower bitrate / drop a layer     cautiously step back up

Both sides can share the product decision semantics of "when to degrade, when to recover", but
the different protocols' cumulative counters cannot be dropped into one formula. Formal
thresholds also need calibration per protocol, direction, encoding layer and real network samples.

8. Wrap-up and next episode (script notes)

This episode dug one level below EP01's "head-of-line blocking": RTMP/RTMPS inherits TCP's
reliable-ordered byte-stream semantics; SRT combines ARQ, TSBPD and optional stale-drop; WHIP
reuses WebRTC/RTP feedback, congestion control and dynamic buffering. None offers a math
latency bound independent of implementation and network conditions. This project keeps WHIP
and SRT together and normalizes their stats for one product degradation policy.

Next episode: Simulcast — how one publish produces multiple quality layers in one pass, and why
the server needn't transcode.

EP03: Content sovereignty — control it yourself, the ultimate moat

greg Tham — Thu, 08 Oct 2026 22:52:40 +0000

Episode 3 of 20 in the **PPCDN Low-Latency Live Streaming Course* — building a self-hosted, low-latency live-streaming CDN from protocols to global delivery. Originally published on the PPCDN docs site; see the full course index for all episodes.*

Recap	EP02 "The self-hosted CDN answer: PPCDN architecture overview" ended by naming the principle of content sovereignty and deferring the trade-offs to this episode
Next	EP04 "Transport showdown: RTMP / SRT / WHIP"

0. Goals of this episode

By the end, the viewer should be able to:

Break "content sovereignty" from a slogan into three concrete, judgeable risks: takedown risk, throttling risk, and data-sovereignty risk.
State how far self-hosting can respond — not "self-hosting = no censorship" but "decision-making moves from a third party to yourself": two different propositions.
Honestly list what self-hosting makes you responsible for — this episode is about trade-offs, not a one-sided list of benefits.

1. Opening hook (script notes)

Last episode closed with: "under your control" is a harder moat than any performance number.
That sounds like a slogan, so this episode unpacks it — which three risk classes "content
sovereignty" actually means, which ones self-hosting can and cannot block, and what
responsibility you take on to get that control.

2. Content sovereignty split into three concrete risks

EP01 §2.3 said "handing your lifeline to a third party you can't control". This episode splits
that into three separately judgeable risk points.

2.1 Takedown risk: can your business be shut off unilaterally

When you depend on a third-party live/RTC cloud, whether your business keeps running ultimately
depends on their terms of service, risk-control policy and commercial judgment — any of which can
change without prior negotiation. This isn't alarmism; it's an inherent feature of the contractual
structure of any "platform" service: terms that let the platform suspend or terminate an account
unilaterally are standard across nearly all cloud and content platforms, triggered by content
compliance judgments, payment risk, geopolitical shifts, or purely commercial strategy. Teams using
a third party are effectively transferring the decision "can the business keep running" to a party
whose decision process they can't see and over which they have no say.

2.2 Throttling risk: can your bandwidth/quality be degraded unilaterally

More common and harder to notice than a plain takedown is throttling: a third party can adjust
your bandwidth quota, node priority or service quality without terminating the service — your viewers
feel worse, but you may not even be able to tell immediately whether it's a network issue or the
provider's policy change. When self-hosting, node allocation and priority are entirely your own
scheduling logic — PPCDN's node pools are isolated per app and balance toward the node with the most
free capacity, and that logic is in your own code; there's no opaque "they quietly changed a
parameter" zone.

2.3 Data-sovereignty risk: which jurisdiction your data and nodes physically sit in

This is the most easily overlooked, yet consequential: where servers that store, process and forward
media actually are is an important factor in determining applicable law and cross-border transfer
obligations — but not the only one. The operator's location, where the data controller/processor
and users sit, the markets served, contractual arrangements and extraterritorial application of laws
can all create jurisdictional links; "servers in country X" cannot be simplified into "governed only
by X's law". When using a third-party cloud, also verify the specific region, sub-processors,
backup/log locations and cross-region DR paths — don't rely on the region name in the console alone.

3. How far self-hosting can respond: honest boundaries

3.1 Self-hosting isn't "no censorship", it's "the decision is yours"

This must be stated clearly, or the whole episode is marketing: self-hosting does not exempt you
from the laws of your jurisdiction; self-hosted or third-party, your business still follows the
regulatory requirements of where it's deployed and operated. What self-hosting changes isn't
"whether to comply" but "who makes the compliance decision and at what pace" — with a third party,
their risk-control policy decides for you and you accept it passively; self-hosted, you decide which
regions to deploy in, how long to retain data, and how to respond to regulatory requirements. The
decision comes back to you — and so does the responsibility.

3.2 Concrete mechanisms: regional deployment, node-pool isolation, configurable retention

Architecturally, PPCDN already has capabilities that directly serve "you decide where data lands":

Physical deployment location is entirely your own choice, and the isolation boundary is set by the node pool: Origin/Edge nodes can be deployed at whatever physical location you pick — each node carries a free-text Region tag (ppcenter/internal/models/node.go) so you can categorize it by city or continent for your own bookkeeping. One detail worth being honest about, since it's easy to oversell: ppcenter currently has no built-in "auto-lock by continent" mechanism that forcibly isolates traffic for you. The architecture design doc did plan a macroRegion data model (Asia / Europe / Americas), but since 2026-09-20 actual Origin/Edge scheduling has worked as "filter by node pool, then balance within the pool toward whichever node has the most free capacity" — it no longer routes by region, and macroRegion was never actually implemented in the ppcenter code. In other words: "which regions you put nodes in" is your own physical-siting decision; "whether traffic crosses regions" has to be explicitly carved out by the node pool described next — you cannot assume the system will auto-lock traffic by continent for you.
Node pools are what actually create the scheduling boundary: an app can bind explicitly to its own pool; when not bound, the product policy may use the user's pool or the shared default pool. Scheduling filters by pool first, then balances within the pool toward whichever node has the most free capacity — but binding does not automatically equal physical exclusivity; isolation granularity depends on how the pools are divided. If you want "this app's traffic may only land on European nodes", you have to build a pool containing only European nodes and bind it explicitly — not expect the system to auto-group nodes by continent for you.
Recording retention is configurable per app (1–90 days) with automatic expiry — how long data is kept is your product/compliance decision, not a provider default.
A control-plane failure doesn't affect established connections (the decoupling in EP02 §2.2, §4.3): even if your own scheduling center fails, your own team fixes it — there's no extra uncertainty like "waiting for a third-party ticket queue".

3.3 What self-hosting makes you responsible for — it's no free lunch

If this episode covered only benefits, it would betray the theme of trade-offs. Self-hosting moves
all of the following, previously borne by a third party, onto you:

Ongoing security and compliance operations: no third-party risk team judges content compliance, payment risk and regional regulatory changes for you; you need your own processes.
24×7 operations responsibility: node scaling, failure recovery, certificate renewal and version upgrades are your engineering team's job, not "file a ticket and wait". That includes watching your own capacity ceiling — a self-hosted 1 vCPU / 2GB node stayed stable the entire time up to 18 concurrent WHEP viewers (CPU ≤ 53%), but at the 19th viewer, CPU spiked to 88–103% within about 15 seconds and frames started dropping: a steep cliff, not a gentle degradation curve (measured 2026-10-08 at roughly 4ms RTT within the same data center — not representative of real internet paths). You have to measure, watch and set your own scaling thresholds for boundaries like this; there's no third-party cloud SLA backing you up.
Multi-region legal and compliance knowledge: to truly use regional deployment, you must understand each region's specific requirements — an ongoing investment of knowledge and process.
Upfront engineering: on day one, a self-hosted setup is not as complete as a mature third-party cloud; the capabilities in EP01–EP02 are the result of engineering investment, not automatic. The good news is that this bar keeps dropping: the latest iteration of self-hosted node registration (ppcenter v1.0.70) simplified it from "manually distribute and guard each node's own nodeSecret" down to "register with just a license code", so you no longer have to hand-sync secrets between the console and node configs. But this kind of simplification is the result of ongoing iteration — it doesn't mean self-hosting is as hassle-free as a mature managed service from day one.

In one line: self-hosting takes back control of business continuity and compliance decisions, at
the price of taking back the corresponding responsibility too. For teams that depend heavily on
continuous availability and need fine control over their data/content, the trade is usually worth it;
for short-term, small-scale scenarios or ones without sensitive data, a third party's low barrier
may still be the better starting point.

4. Diagrams

4.1 Third-party dependency vs self-hosting: who holds the decision

[Depending on a third-party cloud]
Your business
   │
   ▼
Third party's terms / risk-control policy / commercial judgment   ← the decision lives here; you only accept it
   │
   ▼
Takedown / throttling / data routing — whether and when, you don't decide

[Self-hosted]
Your business
   │
   ▼
Your own deployment policy / compliance process / ops team   ← the decision lives here, and so does the responsibility
   │
   ▼
Takedown / throttling / data-routing risks still exist (regulatory requirements don't vanish),
but when and how to respond, and where nodes sit, is your call

4.2 The three layers of content sovereignty

        Operational sovereignty        Network sovereignty        Data sovereignty
   ─────────────────────────    ─────────────────────────    ─────────────────────────
   Can the business be shut    Can bandwidth/quality be    Which jurisdiction do data
   off unilaterally? Who        degraded unilaterally?       and nodes physically sit in?
   decides?

   Third party: not yours       Third party: opaque          Third party: usually only
                                                             an abstract "region" option

   Self-hosted: control back,   Self-hosted: scheduling      Self-hosted: precise choice
   responsibility too           logic in your own code       of region and retention policy

4.3 Regional deployment: you carve the node pools, there's no built-in set of continents

              [Business picks its own physical deployment locations
                       based on compliance/latency needs]
                                     │
        ┌───────────────────────────┼───────────────────────────┐
        │                           │                           │
  [Node pool A: Europe only]  [Node pool B: APAC only]   [Node pool C: Americas only]
        │                           │                           │
  App must bind this pool    App must bind this pool    App must bind this pool
  explicitly to land on      explicitly to land on      explicitly to land on
  its Origin/Edge nodes      its Origin/Edge nodes      its Origin/Edge nodes

Diagram points (narration cues): these three pools aren't a built-in "continent" option in the
system — you build them yourself based on nodes' physical locations, then bind an app to one
explicitly (§3.2's second point); there is no macroRegion layer in ppcenter's current code that
auto-groups nodes by continent for you. Node pools not being interconnected by default is only part
of media-plane regionalization. To claim data stays in-region, you must also put the control plane,
TURN, relays, telemetry, logs, backups and personnel access into the data-flow diagram and audit;
specific legal obligations should be confirmed by professional advice in the applicable jurisdiction.

5. Wrap-up and next episode (script notes)

This episode split "content sovereignty" into takedown, throttling and data sovereignty, and
explained how far self-hosting responds — not exemption from censorship, but taking back both the
decision and the responsibility. That completes Module 0: the latency paradox, the retiring
protocols, the single-point and sovereignty paradox — all three dilemmas laid out, and PPCDN's
overall architectural response explained.

From the next episode we enter Module 1, starting from the most basic transport protocols — how to
choose among RTMP, SRT and WHIP for ingest.

EP02: The self-hosted CDN answer — PPCDN architecture overview

greg Tham — Thu, 08 Oct 2026 22:45:56 +0000

Episode 2 of 20 in the **PPCDN Low-Latency Live Streaming Course* — building a self-hosted, low-latency live-streaming CDN from protocols to global delivery. Originally published on the PPCDN docs site; see the full course index for all episodes.*

Recap	EP01 "The interactive-video dilemma": the latency paradox, the retiring RTMP/HTTP-FLV, the single-point and sovereignty paradox
Next	EP03 "Content sovereignty: control it yourself, the ultimate moat"

0. Goals of this episode

By the end, the viewer should be able to:

Draw PPCDN's topology on a whiteboard: publisher, Origin, Edge, viewers, and where the control plane handling scheduling/signaling sits.
Explain the engineering response PPCDN gives to each of EP01's three dilemmas — the low-latency path, the real-time protocol choice, and a controllable distributed topology.
Explain the three concrete engineering reasons "self-hosting can beat big-cloud services on cost-effectiveness", rather than the empty claim "self-hosting is cheaper".

1. Opening hook (script notes)

Last episode we broke down three dilemmas: dynamically-accumulating latency across the
pipeline, RTMP/HTTP-FLV exiting the browser playback ecosystem, and single points and
third-party dependencies. This episode stops posing problems — we draw PPCDN's overall
architecture and see how it responds to each.

2. Architecture overview: the whole picture in one diagram

2.1 Three tiers + one control plane

PPCDN's media path has just three tiers, plus a control plane that never touches media data:

Publisher: the in-house ppobs (deeply customized from OBS Studio), or any standard WHIP/SRT publisher.
Origin: the source that receives the publish stream — the sole entry for media into the system.
Edge (multiple): pulls from Origin and serves viewers; multiple viewers of the same stream on one Edge share a single origin-pull link.
ppcenter (control plane): scheduling, auth, node management, P2P signaling, billing metadata and health reports. It carries no media and proxies no media packets.

Keeping the control plane and media plane fully separate is the most fundamental design
principle in the whole architecture; nearly every later "high availability" conclusion rests on it.

2.2 Why separate the control plane from the media plane

If scheduling/auth "decision logic" shared a path with audio/video, a control-plane wobble
would wobble viewers too. PPCDN keeps media packets out of ppcenter: the control plane
mainly participates in connection setup, auth and scheduling; after setup, media takes an
independent path, and a control-plane failure does not interrupt in-progress viewing. This
fundamentally shrinks the failure domain and lets the media plane run independently and
steadily.

Another easily missed premise is clock synchronization. Short-lived credentials'
iat/exp/txTime, cross-device latency instrumentation and fault detection all depend on
time: nodes sync their clocks via NTP and monitor skew, and auth allows only a clear, finite
clock-skew tolerance. The whole network's clocks stay self-consistent with ppcenter as the
common reference — subtracting timestamps across nodes is only meaningful within that frame.

3. How each of the three dilemmas is answered

3.1 Answering Dilemma 1 (latency paradox): codec-aware routing to the low-latency path

EP01 explained that the Edge path adds a server hop, and latency is further shaped by
encoding, network, recovery and buffer policy. PPCDN's architectural answer is to route by
the player's decode capability, opening an additional, usually-shorter P2P direct path for
H.264 clients:

Routing by decode capability at playback time: HEVC-capable clients go straight to the Edge HEVC stream — HEVC is usually more bandwidth-efficient than H.264 at equal quality, and "~30% less" is a commonly-cited interval midpoint in the industry and inside this project, not a number this project has itself measured and validated yet (EP20 digs into the measurement method and its limits), though this path does avoid the first-frame uncertainty of P2P setup. Clients without HEVC first try a P2P direct connect for H.264.
On the P2P route, ppcenter first runs the NAT eligibility check and atomically allocates a signaling slot (at most 3 direct connections per publisher by platform default — defaultMaxP2PSessions=3, adjustable by a superadmin up to 50); the two sides then relay SDP/ICE through ppcenter to establish the direct connection — ppcenter only relays signaling and never parses media.
Once P2P connects, media flows straight from publisher to viewer, bypassing Origin/Edge and consuming zero edge egress — the lowest-latency path.
On any failed check, timeout or setup failure, it cleanly falls back to the Edge H.264 stream, with no error or black screen.
P2P only uses the publisher's measured spare uplink; on detected uplink congestion it is sacrificed immediately, so the main publish's quality and bandwidth budget are never affected.

EP13 covers NAT traversal and eligibility details; for now, remember one line: route by
decode capability first; H.264 connects directly when it can, and cleanly falls back to Edge
when it can't.

3.2 Answering Dilemma 2 (retiring protocols): WHIP/SRT ingest, WHEP playback throughout

EP01 explained that TCP's reliable-ordered byte stream can hit head-of-line blocking on loss.
PPCDN's answer is to prefer a real-time transport stack that can bound recovery by media timeliness:

Ingest supports both WHIP and SRT: typical paths use UDP and bound recovery by media timeliness, so a weak network is not dragged down by TCP head-of-line blocking.
Delivery and playback uniformly use WHEP; Edge offers only low-latency WebRTC playback and no HTTP-FLV endpoint — a constraint locked at the requirements stage.
The weak-network answer isn't "tough out the loss" but proactive degradation: the server steps up/down between simulcast layers based on loss and RTT — first lowering bitrate (without interrupting the publish), and only lowering resolution if bitrate is already at the floor and still short (that step has a brief interruption). EP04 and EP07 cover the transport comparison and weak-network mechanisms in detail.

3.3 Answering Dilemma 3 (single point and sovereignty): shrink the failure domain, keep control in your hands

The Edge tier scales horizontally: Edge deploys across multiple nodes, scheduled by node pool and preferring the node with the most free capacity; multiple viewers of the same stream on one Edge share a single origin-pull link, so one overloaded Edge doesn't drag down everything.
Edge is autonomous: before each origin pull, Edge dynamically resolves the Origin address and short-lived media credentials from ppcenter (POST /internal/mmx/v1/origin/resolve) and caches the result in memory — when the control center is unavailable, it reuses the unexpired cache to keep reconnecting, so the media plane doesn't wobble with the control plane (docs/ppcdn-development-progress.zh-CN.md §3.5, status "completed").
Control plane and media plane are separated: a control-plane failure does not interrupt in-progress viewing; the media path keeps running steadily.
Content sovereignty: the whole system is self-hostable with nodes under your control, so the entire publish-to-delivery path is yours; the sovereignty trade-offs and compliance strategy are covered in EP03.

4. Diagrams

4.1 Overall architecture

                         ppcenter (control plane)
              scheduling · auth · P2P signaling · billing metadata · health reports
                    ▲                              ▲
                    │ control / signaling          │ control / signaling
                    │                              │
  ppobs publisher ──WHIP/SRT──▶ Origin ──forward──▶ Edge 1..N ──WHEP──▶ viewer browser
        │                                                                    ▲
        └──── P2P direct: H.264 first, fall back to Edge (HEVC goes straight to Edge) ────┘

Diagram points (narration cues): note the control plane (ppcenter) is drawn above the
path with dashed arrows — it only takes part in the "establish connection" decision, not on
the media path. Media travels the solid path below, and P2P direct is a shortcut that bypasses
Origin/Edge straight from publisher to viewer.

4.2 Playback routing decision

Viewer requests playback
        │
        ▼
pplayer detects whether the browser can hardware-decode HEVC
        │
        ├─ HEVC supported ──▶ go straight to Edge, append /hevc/whep
        │
        └─ No HEVC
                │
                ▼
        Try a P2P direct connect for H.264 first
                │
                ├─ ppcenter eligibility passes + a free signaling slot ──▶ relay SDP/ICE ──▶ use P2P
                │
                └─ Not eligible / timeout / setup fails / slots full ──▶ cleanly fall back to Edge H.264

4.3 Control/media separation: a control-plane failure does not affect the media plane

                    ppcenter (control plane) fails
                            │
                            ▼
              Established media sessions keep transferring steadily
                            │
                            ▼
        After recovery, all endpoints re-register and rebuild leases

Diagram points (narration cues): control plane and media plane are fully separated, so a
control-center failure does not interrupt in-progress viewing; after recovery, all endpoints
re-register and rebuild leases. This is the key design that lets PPCDN shrink the failure
domain.

5. Cost-effectiveness: why self-hosting can beat big-cloud services

The sections above address "can self-hosting achieve low latency and control risk". This one
answers a more practical question: at equal investment, why can self-hosting beat big-cloud
live/RTC services like Tencent Cloud LEB or Huawei Cloud on cost-effectiveness? Not via
scale discounts, but via three engineering levers that big clouds structurally struggle to copy.

5.1 Custom ppobs: the whole path is optimizable end to end, a lever big clouds don't have

Big-cloud live/RTC services can usually optimize only "their own side" — origin and edge
delivery; the publisher is either a customer's generic SDK or standard OBS, so capture,
encoding, instrumentation and QoS policy are outside the provider's control — the provider and
the client are two teams, two codebases.

PPCDN's ppobs is a deeply customized publisher, co-designed with the server under one
architecture, enabling coordinated optimization a "buy an SDK" model can't:

Absolute timestamps embedded in the bitstream (SEI for H.264/HEVC, metadata OBU for AV1) let the player compute real end-to-end latency — this requires changing encoder/packetizer logic, not a config knob, and a generic SDK can't.
P2P and the main publish share one encoding pass, avoiding per-viewer re-encoding and saving publisher CPU; independent send queues and bandwidth budgets for P2P and the main publish keep "turn on P2P to save money" from hurting the host's quality.
Weak-network degradation (lower bitrate first, without interrupting the publish) is triggered jointly by encoder and server, not by the server's one-sided rate-limit guess.

These coordinated optimizations are essentially the added value of "end-to-end programmability":
only by owning both ends can you tune the whole path as one system instead of bolting two black
boxes together.

5.2 WHIP/SRT simulcast: multiple layers from one client encode pass, no server transcode

Multi-layer quality (e.g. 1080p/720p/360p simulcast) is usually achieved by server-side
transcoding in big clouds — Tencent Cloud LEB bills transcoding per output layer, so more
layers and longer duration cost more. That's industry practice, not an exception.

ppobs uses WHIP/SRT simulcast to output multiple layers from the encoder in one pass; the
server only forwards, no decode, no transcode. For customers who need multiple layers anyway,
eliminating transcoding often cuts the total bill substantially — in many setups, multi-layer
transcoding is the second-largest cost after bandwidth, and removing it materially changes the
bill structure.

5.3 ppobs + pplayer coordinated direct connect: ~70ms, really measured

Whether P2P direct connect works and how low latency can go depends not only on "having a NAT
traversal library" but on whether publisher and player can coordinate eligibility decisions and
timestamp alignment (see §3.1, diagram 4.2). This project's P2P direct connect is already
proven, with end-to-end latency of about 70ms; the ingest path delivered via Edge measured
about 108ms for WHIP and about 386ms for SRT (the gap comes from the ingest protocol's
receive window, not the encoding format).

6. Wrap-up and next episode (script notes)

This episode's architecture diagrams answered EP01's three dilemmas: codec-aware routing adds
a shorter P2P direct path for H.264 clients, the real-time protocol choice escapes TCP
head-of-line backlog risk, and separating the control and media planes while scaling Edge
horizontally shrinks the failure domain and keeps control in your hands. With a custom client,
no server transcoding and end-to-end coordination, PPCDN delivers a self-hosted solution that
is clearly stronger on latency, cost and controllability.

Next episode, we tackle "sovereignty" first — why "under your control" is a harder moat than
any performance number, and how to weigh self-hosting against relying on a third party.

EP01: The interactive-video dilemma

greg Tham — Thu, 08 Oct 2026 22:45:35 +0000

Episode 1 of 20 in the **PPCDN Low-Latency Live Streaming Course* — building a self-hosted, low-latency live-streaming CDN from protocols to global delivery. Originally published on the PPCDN docs site; see the full course index for all episodes.*

0. Goals of this episode

By the end, the viewer should be able to answer two questions:

Is my interactive live-streaming / interactive-game project also stuck in the latency, legacy-protocol and single-point/sovereignty traps?
If my pipeline still uses RTMP or HTTP-FLV, what happens on a weak network, and why is that not an exaggeration?

This episode only lays out problems and verifiable facts — no cure; the cure is
EP02–EP20.

1. Opening hook (script notes)

If you've built interactive live streaming — real-person video, online auctions,
co-streamed games, or games with barrage feedback — you may have hit this: the host
has already moved to the next step, but the viewer's picture is still on the previous one.

That's usually not just "the network happened to hiccup". It's the combined result of
capture, encoding, transport, jitter resistance and decode/render. Today we break these
problems into several real dilemmas and separate protocol mechanics, configuration and
measured data.

2. Three real dilemmas

2.1 Dilemma 1: the latency paradox — mature protocols are the opposite of real-time business needs

HLS/DASH is the most mature way to deliver internet video. Based on "segment then
download", it inherently carries seconds of latency. That barely matters for one-way
VOD, but it is a structural conflict for highly interactive scenarios:

Scenario	Why latency-sensitive	Consequence of traditional segmentation
Live video / real-time trading	The interaction window is measured in seconds; the picture must match the business state	Users see a stale picture and can't participate
Online auctions / flash sales	Bid order decides fairness; the picture can't lag the business state	Ordering confusion, disputes
Interactive classrooms	Teacher-student Q&A and exercises need instant feedback	Interaction degrades into one-way playback
Social co-streaming	Conversation depends on natural turn-taking	Talking over each other, latency stacking — unusable

Do real-time protocols, usually UDP-based (WebRTC/SRT), fix it automatically? No.
End-to-end latency is the combined result of capture, encoding, network queueing and
propagation, loss recovery, receiver jitter buffer, decode and render. Some waits are
bounded by parameters, some vary dynamically with network and implementation; you can't
just add a few defaults together and call it a measurement, and you certainly can't treat
one buffer parameter as a strict mathematical upper bound for an entire protocol.

The project's confirmed, same-caliber measurements are below. They are PPCDN case data,
not a guarantee of WebRTC, SRT, or any codec in all environments:

Scenario	Measured end-to-end latency	Notes
P2P direct	~70ms	Real link, media bypasses Origin/Edge forwarding
WHIP ingest (via Edge)	~108ms	Measured on the current actual pull path
SRT ingest (via Edge)	~386ms	Measured on the current actual pull path

The WHIP-vs-SRT gap mainly comes from the ingest protocol itself — SRT's receive window
(TSBPD) inherently queues longer than WebRTC ingest does. That's a protocol difference,
not a codec difference: HEVC currently doesn't even participate in P2P direct connect, it
only goes through Edge delivery, so the encoding format isn't the variable driving these
numbers. P2P skipping a server hop usually helps latency, but the result still depends on
the network path, NAT traversal outcome, encoder and player. A protocol name only tells
you which mechanisms are available; whether it meets a real-time goal must be measured with
unified instrumentation, endpoints and clock caliber.

2.2 Dilemma 2: a protocol being retired — RTMP/HTTP-FLV on weak networks

2.2.1 An industry migration that already happened

Adobe formally ended Flash Player support on 2020-12-31, and mainstream browsers then removed the Flash runtime. RTMP (Real-Time Messaging Protocol) was originally designed for Flash Player playback in the browser — that "native browser playback" path no longer exists.
What people call "RTMP live" in a browser today doesn't actually use RTMP for playback: it's re-wrapped as HTTP-FLV (same TCP transport, but the browser still needs a JavaScript library like flv.js to de-mux client-side), or transcoded to HLS/DASH (segment download buys playback compatibility at the cost of real-time — see Dilemma 1).
WHIP (WebRTC-HTTP ingestion) and WHEP (WebRTC-HTTP egress) matured during IETF standardization and have been adopted by mainstream publishers including OBS Studio and several real-time services as the next-generation ingest/playback protocols — precisely to replace RTMP at the "ingest" position.
This isn't one vendor's preference but the joint direction of browser capability, codec standards and protocol standardization over recent years: the playback-side RTMP/Flash combo is gone, and ingest-side RTMP is being replaced by WHIP/SRT.

2.2.2 Root cause: why it's a disaster on weak networks, not hyperbole

RTMP and HTTP-FLV are both built on TCP. TCP delivers an "ordered, complete" byte stream:
if any packet is lost, later data that already arrived must wait for its retransmission
before reaching the application — this is Head-of-Line Blocking, a design property of
TCP itself, not a defect of some implementation.

On a weak network (with some loss rate), this cascades:

A packet is lost → retransmit, waiting at least one more round-trip (RTT);
During retransmission, all later-arrived audio/video is blocked in the buffer and can't play;
Sustained loss or repeated retransmits → blocking can accumulate from tens of ms to seconds or more, and has no theoretical upper bound;
The player buffer fills or drains → stutter, catch-up, and in the worst case a reconnect.

This differs from common WebRTC/SRT trade-offs: an application can bound how long it waits
for retransmission based on a packet's timeliness and drop data once it's stale, avoiding
endless accumulation for the sake of completeness. SRT does this with ARQ, TSBPD and
stale-packet drop; WebRTC does it with RTP/RTCP feedback, congestion control, jitter buffer
and decoder frame-drop policy. Neither offers a strict mathematical latency bound
independent of network, configuration and implementation: congestion, queueing, outages,
reconnects or implementation policy can still push latency well up.

In one sentence: TCP byte streams prioritize reliable, ordered delivery, so loss causes
head-of-line blocking; a real-time media stack can drop stale data by timeliness, which
usually makes latency easier to control — but it is not an unconditional upper bound.

2.2.3 Protocol comparison

Dimension	RTMP / HTTP-FLV	SRT	WebRTC (WHIP / WHEP)
Transport	TCP	UDP + ARQ	Usually UDP + RTP/RTCP; can fall back to TURN/TCP/TLS
Loss handling	TCP must deliver reliably in order; possible head-of-line blocking	ARQ requests retransmit; TSBPD delivers by time; configurable stale-drop can discard late data	Whether NACK/FEC is used depends on negotiation/implementation; the receiver can drop stale frames
Weak-network latency	Queueing/retransmit can accumulate to seconds or kill the stream	Engineering bounds via `latency` etc., but no unconditional math bound	Jitter buffer and congestion control adjust dynamically; no unconditional math bound
Native browser playback	No longer supported (relies on the discontinued Flash Player)	Not supported (usually needs protocol conversion server-side)	Native, standard Web APIs
Current role	Legacy protocol exiting the playback path	The modern choice for weak uplinks	The mainstream real-time playback protocol

2.2.4 A concrete architecture choice

This isn't abstract — the project's public technical notes say it plainly: ingest supports
both WHIP and SRT (the SRT ingest measurements in §2.1 are exactly that path), playback is
uniformly WHEP, and HTTP-FLV playback endpoints are deliberately not provided. In other
words, "no HTTP-FLV" isn't an omission but a constraint locked in at protocol-selection
time; SRT is not contradictory — it addresses weak-uplink robustness, while playback still
goes only through WHEP. Such constraints are increasingly common in mature real-time
systems and are the concrete form of the "industry migration" discussed here.

2.3 Dilemma 3: the single-point and sovereignty paradox — the easier it is, the more you hand over your lifeline

Beyond protocols, real projects carry two equally important structural risks that don't show
up in latency numbers:

Single point of failure: an Origin is usually a single copy — the sole entry for all publishing. When it fails, the result is not "degradation" but a whole-path outage: every viewer loses the picture at once. This isn't theory — the architecture design doc lists "Origin publish disconnect" as a known failure mode, and explicitly not an automatic, seamless failover: the publisher must re-publish to a new Origin before any viewer can recover. This is an objective risk point in many real interactive-live architectures today, and something to weigh carefully later: whether a single point is worth eliminating is an engineering calculation, not the dogma "a single point must be eliminated".
Content sovereignty: to cut engineering effort, some teams outsource distribution to a single third party, even treating it as a "backup fallback". It looks like double insurance, but actually hands business continuity to a third party you can't control — their rate limiting, service changes or cross-border compliance shifts can all directly affect your availability, and you have no say in those decisions.

These two are still blank spots in most discussions about interactive-live projects —
few systematically assess whether a single point is worth eliminating, and few discuss how to
weigh content sovereignty (EP03 expands on it). This episode just points them out.

3. Diagrams

3.1 End-to-end latency composition: defaults can't replace measurement

Capture → Encode → Send queue → Network propagation/queueing/loss recovery
        → Receiver reorder & jitter buffer → Decode → Render

These stages overlap or vary dynamically; configuration values can't be mechanically added.

PPCDN same-caliber measurements:
P2P direct ~70ms
WHIP ingest (via Edge) ~108ms, SRT ingest (via Edge) ~386ms
(the gap comes from the ingest protocol's receive window, not the codec —
HEVC currently doesn't participate in P2P direct connect)

Diagram points (narration cues): unify the measurement endpoints and clock first, then
look at per-stage instrumentation. Network, encoder and buffer policy can each become the
bottleneck; you can't infer end-to-end latency from a single RTP packet count, an SRT
latency parameter, or a player's target buffer value.

3.2 The protocol triangle: latency, weak-network robustness, ecosystem compatibility

                    Lowest latency
                    ╱        ╲
                   ╱ WebRTC    ╲
                  ╱ (WHIP/WHEP) ╲
                 ╱________________╲
   Best weak-net robustness ──── Best ecosystem compatibility
     (SRT, trading retransmit     (RTMP once unified the world
      window for weak-net           via the Flash Player;
      availability)                 after Flash ended, this
                                    corner collapsed)

Diagram points (narration cues): RTMP's historical edge was exactly the "ecosystem
compatibility" corner — almost every browser could play it via Flash. After Flash ended,
that corner no longer holds for RTMP, and it never had an upper bound on the
"weak-network robustness" corner either (see the head-of-line analysis in §2.2.2). That's
the graphical explanation of "RTMP/HTTP-FLV is being retired": it wasn't beaten by a
competitor; its one advantage collapsed on its own.

3.3 Weak-network loss handling: head-of-line blocking vs dropping by timeliness

[RTMP / HTTP-FLV (TCP)]
frame1  frame2  frame3(lost)  frame4  frame5 ...
                   │
                   ▼
     TCP demands "ordered + complete" delivery
                   │
                   ▼
     frame4, frame5 must queue until frame3 is retransmitted
                   │
                   ▼
     Retransmit waits ≥ 1 RTT; on a weak network, repeats stack to seconds, no theoretical bound
                   │
                   ▼
     Player buffer fills → stutter / catch-up / reconnect

[SRT / WebRTC (real-time media policy, typically UDP)]
frame1  frame2  frame3(lost)  frame4  frame5 ...
                   │
                   ▼
     Try retransmit, FEC, or wait for reorder per protocol/implementation policy
                   │
                   ▼
     Once stale, recovery can be abandoned to avoid further backlog
                   │
                   ▼
     Viewer experience: usually trades localized quality loss for more controllable latency

3.4 A real project's architecture today: single point and third-party dependency

                 [Viewers / Internet]
                          │  egress
        ┌─────────────────┼─────────────────┐
        │                 │                 │
   [Edge 1]          [Edge 2]   ...    [Edge N]
        └─────────────────┼─────────────────┘
                internal forwarding
                          │
                   [Origin]   ← sole entry, single point, failure = whole-path outage
                          │
                    ingest
                          │
              [publish site · long online]

     [Origin] ──backup route──> [third-party live service]  ← continuity depends on a third party's decisions

Diagram points (narration cues): this is the distribution architecture of one real
interactive-live project today, not a hypothetical. Both risks of Dilemma 3 map onto it —
the Origin is the sole entry (single-point risk), and the backup route hangs off a
third-party cloud (sovereignty risk). This is why the course starts from "dilemmas" rather
than jumping straight to solutions.

4. Wrap-up and next episode (script notes)

The dilemmas seen here reduce to one line: a traditional live path optimized for one-way
viewing cannot be assumed to meet real-time interaction goals. Dynamic or configured
buffering, TCP head-of-line blocking, architecture single points and third-party
dependencies all need scenario-specific measurement and trade-offs — not conclusions from a
protocol name alone.

Next episode, we start taking apart how PPCDN addresses these architecturally — trying P2P
first and falling back to Edge in sequence on failure, controlling weak-network backlog by
media timeliness, and keeping the critical path in your own hands.

ppmmx Standalone Stress-Test Report — The Concurrency Where CPU Hits 80% on a 1 vCPU Node

greg Tham — Thu, 08 Oct 2026 09:40:00 +0000

ppmmx Standalone Stress-Test Report

1 vCPU / 2GB self-hosted node: the concurrency at which CPU usage hits 80%


Node under test	Self-hosted ppmmx standalone node, LightNode, 1 vCPU / 2GB, Ubuntu 24.04
Deployment	Official public `deploy.sh` (`ppmmxDocker`), `docker compose up -d --build`, default bridge network + port mapping
Software under test	ppmmx `v1.19.1rc064`
Load generator	`whep-loadgen` (pion/webrtc, RTP read-and-discard)
Ingest	Local `ffmpeg` (`testsrc2` + `libx264`, 1280×720@30, ~2.5 Mbps, video-only) pushed over the public internet via SRT
Date	2026-10-08

Summary

This 1 vCPU / 2GB node stayed stable through 18 concurrent viewers (CPU ≤ 53%). The moment the 19th viewer joined, CPU climbed from ~40% to 88-103% within about 15 seconds and stayed there, and the server started dropping frames for multiple sessions (reader is too slow). The concurrency at which CPU crosses 80% is 19.

This isn't a smooth curve — it's a cliff. There's almost no transition zone between 18 and 19 viewers. The reason is straightforward: this is a single-core machine with no second core to absorb GC pauses, network polling, and bursty processing. Once the one core gets close to saturated, SRT/WebRTC processing can't keep up in real time, buffers start backing up (the SRT ingest's RTT jumped from ~4ms to 100-500ms during the 19-viewer run), which drives CPU even higher — a small congestion-collapse feedback loop rather than linear degradation.

Along the way, testing also surfaced and fixed two real bugs affecting every customer who self-hosts via licenseCode-only deployment (see "Findings along the way" below); both fixes are already live in production.

1. Method

Structurally identical to the first round of an earlier 2 vCPU stress test, just on a 1 vCPU box with a finer-grained ladder:

10 -> 15 -> 17 -> 18 -> 19 -> 20

Each step ramps up to the target number of WHEP readers and holds for 45-50 seconds (covering the full ramp plus a steady-state window), sampling the container's CPU% via docker stats --no-stream every 3 seconds (single-core box, so 100% = one full core saturated) along with memory, while watching server logs for reader is too slow warnings or RTT anomalies. Test stream: ffmpeg testsrc2 + libx264, 1280×720@30, ~2.5 Mbps over SRT — matching the first round of the earlier report (no audio, no multi-track).

Measured RTT between the load generator and the node under test was ~4ms, close to same-datacenter quality, so this measures the server's own capacity rather than real-world internet-path behavior.

2. Results

Concurrency	Verdict	Connected	CPU (steady-state mean / peak)	Server-side anomalies
10	Stable	10/10	26% / 30%	None
15	Mostly stable	15/15	~45% (with two ~95-98% transient spikes, likely a periodic background task rather than sustained load)	None
17	Stable	17/17	47% / 64%	None
18	Stable (last clean step)	18/18	45% / 53%	None
19	Crosses 80%, overload begins	19/19	Climbs from ~40% to 88-103% within ~15s of full ramp, then stays high	`reader is too slow, discarding N frames` (multiple sessions)
20	Saturated	20/20	94-104% (high from the start of steady state)	Same as above, plus SRT ingest RTT rising from ~4ms to 100-500ms

CPU is the container-level percentage from docker stats; 100% = one full core saturated.

3. The Key Finding: A Cliff, Not a Gradient

From 10 to 18 viewers, CPU climbs roughly linearly with concurrency (~26% → ~45-53%, about 2-3% per viewer) — consistent in direction with the linear model seen on multi-core machines. But the step from 18 to 19 blows right past that trend:

Extrapolating the 10-18 slope would predict ~48-55% at 19 viewers.
The actual measurement was 88-103%, and it didn't happen instantly — with 19 viewers already connected and CPU still sitting at ~40%, it took roughly 15 seconds before it started climbing, eventually settling at a high plateau, with frame-drop warnings logged for multiple readers along the way.

This echoes the "CPU rises roughly linearly with concurrency until one step suddenly runs away" pattern seen on a 2 vCPU machine in the earlier report, but the cliff on a single-core machine arrives much earlier and is much narrower — with no second core to fall back on, once the main processing path can't keep up in real time, SRT/WebRTC buffer backlog and retransmit/frame-drop handling consume even more CPU, creating a self-reinforcing congestion spiral rather than a graceful degradation.

4. Conclusions and Recommendations

This 1 vCPU / 2GB node's practical concurrency ceiling is 18 viewers at 2.5 Mbps WHEP (peak CPU 53%, leaving headroom); 19 is the point where CPU crosses 80% and frame drops begin.
Don't estimate a 1 vCPU node's capacity by simply halving a 2 vCPU model like "CPU% ≈ 15 + 0.97% × N" (that would optimistically suggest ~35 viewers) — a single core's behavior near saturation is a non-linear cliff, not half of a linear curve.
Memory stayed healthy throughout (peak 161.8 MiB, well under the 2GB limit) — in this test, CPU hit its ceiling first, not memory, unlike the earlier report's conclusion that small standalone nodes hit memory first. The difference is simply that the CPU budget dropped from 2 cores to 1, so CPU became the binding constraint instead.
If this node needs to serve more viewers, upgrade to 2 vCPU rather than adding RAM.

5. Findings Along the Way (fixed / logged)

Testing itself got blocked twice by real bugs, both of which got handled:

A licenseCode-only self-hosted node's publish whitelist never syncs. In the node-registration protocol, after ppcenter resolves a node's identity from its license code, it never echoed the resolved node secret back to the node — yet the whitelist-sync endpoint requires that same secret as a credential. The node could never obtain it, so its whitelist stayed permanently empty and no appId could ever publish. Fixed: the registration acknowledgment now includes that resolved value (only for licenseCode-only registrations), and the node caches it lazily after a successful registration. Already deployed to production and verified on the test node (publishing was accepted normally).
WHEP playback can't connect under the default bridge-network docker-compose.yml deployment. The container defaults to gathering ICE candidates from its network interfaces, but under bridge networking the container only sees its internal Docker bridge IP, not the public one — so WebRTC's ICE candidates point at an address nothing outside the host can ever reach. SRT ingest is unaffected (it's a plain port forward), but playback just times out waiting to connect. This test worked around it by manually adding a public-IP override to the container config, and confirmed that fixes it; the underlying code hasn't been patched yet and is tracked as a separate follow-up.

6. Limitations and Caveats

Single-track SRT H.264 only, no audio, no multi-track. Matches the earlier report's first round for comparability, but doesn't represent the cost of a real multi-track WHIP ingest path.
Short hold times. Each step ran 45-50 seconds; long-term stability (thermal throttling, multi-hour GC behavior) wasn't tested.
Load generator and node under test were near the same datacenter (~4ms RTT). This measures server capacity; real internet-path loss/jitter would further reduce effective capacity.
Default limits were raised for testing, then restored. The node's default per-path reader cap was temporarily set to unlimited to see the real CPU ceiling; the ICE workaround from section 5 was likewise temporary. Both were restored to factory defaults and verified by restart before this report was finalized.
Single test run, not repeated. The 18/19 boundary could shift by ±1 viewer on a repeat run (the 15-viewer step already showed one hard-to-explain transient spike). "18 is safe, 19 crosses the line" is a solid conclusion, but treat the exact boundary as "somewhere between 18 and 19" rather than an absolute number.

Live End-to-End Latency: P2P 70ms vs WHIP 108ms vs SRT 386ms

greg Tham — Wed, 07 Oct 2026 15:42:55 +0000

Live Latency Performance Test Report

Attribute	Value
Document type	Test report (method + result analysis)
Subject	PPCDN end-to-end live latency: ppobs publish ??ingest ??delivery ??pplayer playback
Key metric	End-to-end latency (the player panel's "P2P Delay")
Result summary	P2P direct 70ms, WHIP ingest 108ms, SRT ingest 386ms
Updated	2026-09-29

0. Summary

After tuning, the measured end-to-end latency of three access/playback modes:

Scenario	Path	Measured	Screenshot
P2P direct	Player ??publisher direct (no Edge)	70ms	?4.1
WHIP ingest	WHIP ingest ??Edge delivery ??playback	108ms	?4.2
SRT ingest	SRT ingest ??Edge delivery ??playback	386ms	?4.3

Conclusion: latency P2P < WHIP < SRT. SRT is about 278ms higher than WHIP,
mainly from the ingest-side SRT receive window (TSBPD), a fixed delivery delay; WHIP
ingest has no such window and is clearly lower; P2P direct has the shortest path and
the lowest latency.

1. Goal & scope

Quantify and compare P2P direct, WHIP ingest and SRT ingest end-to-end latency.
Verify the latency gains after tuning and document a reproducible, publishable method.

Not tested: throughput ceiling, concurrency scale, billing correctness, recording path.

2. Metric definition and measurement principle

2.1 End-to-end latency

The one-way delay from the ppobs capture instant (in-bitstream mark) to pplayer
rendering, covering capture, encode, ingest, delivery (Origin??dge or P2P direct),
jitter buffer and decode. The player panel's "P2P Delay" is this value (the label is
shared by the Edge / P2P paths; the actual path is shown as Connection in the panel).

2.2 Measurement path

ppobs writes an absolute UTC timestamp into the bitstream (H.264/H.265 SEI); Origin/Edge pass it through hop by hop without decoding or transcoding.
ppplayer subtracts it from a clock calibrated against ppcenter:

  corrected_now   = local_clock + offset
  one_way_delay   = corrected_now - embedded_timestamp

Readings add a fixed +50ms compensation for the encode/decode/render overhead between the mark and rendering that cannot be measured directly.
Cross-browser measurable: to avoid non-Chromium browsers (e.g. iOS Safari, which lacks WebCodecs Insertable Streams and cannot read the in-bitstream SEI) seeing only an estimate, Edge nodes parse the ingest stream's SEI and push OBS_TIMESTAMP to the player over the ABR control WebSocket, so the measured value is readable on iOS Safari too.

2.3 Boundaries

Readings come from the ppplayer panel; full percentiles are aggregated per minute/day by ppcenter and visible in the console.
Single machine, single stream, limited window ??it does not represent a network-wide SLA.

3. Test configuration

Item	Value
Publisher	ppobs
Ingest / connection	P2P direct / WHIP / SRT (three-way comparison)
Codec	H.264 (HEVC included in multitrack scenarios)
Player	pplayer
Playback buffer	ppplayer default

4. Measured results

4.1 P2P direct ??70ms

The player is P2P-direct to the publisher (no Edge); end-to-end latency 70ms.

4.2 WHIP ingest ??108ms

WHIP ingest, Edge delivery; end-to-end latency 108ms.

4.3 SRT ingest ??386ms

SRT ingest, Edge delivery; end-to-end latency 386ms.

5. Analysis

Scenario	Measured	vs WHIP	Main difference
P2P direct	70ms	??8ms	Shortest path: player directly to publisher, no ingest/cascade
WHIP ingest	108ms	baseline	No TSBPD receive window, small ingest overhead
SRT ingest	386ms	+278ms	Ingest SRT receive window (TSBPD) adds a fixed delivery delay, plus weak-network retransmit buffering

P2P direct is lowest: it removes the Origin??dge cascade and ingest queue, leaving only capture/encode + end-to-end network + decode/render.
WHIP beats SRT: WHIP is WebRTC-based and has no fixed TSBPD receive window on ingest ??consistent with the design intent (SRT's receive window is deliberately enlarged for weak-network retransmission: a loss-robustness ??latency trade-off).
SRT is highest: the ~278ms overhead closely matches the SRT receive window (hundreds of ms at its floor); on a lossy uplink, retransmission/receive buffering pushes it higher. On a clean uplink SRT falls back near its receive-window floor.
The gaps are stable and reproducible, indicating the difference comes from the protocol/link structure, not occasional noise.

6. SLA note

Metric	Target	Ceiling	Path
P80	??300ms	??600ms	P2P direct
P99	??2s	??	P2P direct

This test's P2P direct 70ms is well within the P80 target (??00ms). Note the SRT
ingest path's latency is governed by the ingest receive window and is not the same
metric as the P2P direct SLA.

7. Conclusion & limitations

Conclusion

End-to-end: P2P direct (70ms) < WHIP ingest (108ms) < SRT ingest (386ms).
For the lowest latency, prefer P2P direct, then WHIP ingest; SRT ingest trades several times WHIP's latency for weak-uplink retransmission robustness ??the choice depends on your tolerance for weak networks.

Known limitations

Single machine, single stream, limited window ??not a network-wide SLA.
SRT readings depend on uplink loss/retransmission and receive-window adaptation, and vary over time.
Mobile devices are strongly affected by carrier network and signal strength.

8. Revision history

Date	Revision	Notes
2026-09-26	Draft	Test plan finalized
2026-09-29	Results	Merged end-to-end latency analysis; added 5-scenario measurements
2026-09-29	Tuned results	Updated to a P2P/WHIP/SRT comparison (70/108/386ms) with screenshots

ppmmx Standalone Stress-Test Report: Concurrent WHEP Viewer Capacity on 2 vCPU Nodes

greg Tham — Wed, 07 Oct 2026 14:45:36 +0000

ppmmx Standalone Stress-Test Report

Concurrent WHEP viewer capacity on 2 vCPU nodes


Report	ppmmx standalone load-capacity and VPS-egress test
Version	v4 (final)
Date	2026-10-07
Software under test	ppmmx `v1.19.1rc064`, `NODE_ROLE_STANDALONE`
Load generator	`whep-loadgen` (pion/webrtc, RTP read-and-discard)
Providers compared	LightNode (Manila / Taipei / Singapore / Tokyo); AWS Lightsail (Singapore)

Abstract

ppmmx in standalone mode was stress-tested for concurrent WHEP viewer
capacity on small cloud VPS instances, over two ingest paths (single-track SRT,
then multi-track WHIP with simultaneous HEVC + H.264), and across two providers.
The central findings are:

The first provider tested (LightNode) never delivered its advertised 1 Gbps. Across three separately provisioned batches and four cities, measured egress was ~50??06 Mbps (per-instance, per-direction, both TCP and UDP, and to a third-party endpoint). A node on that network saturates the link at roughly 39 concurrent 2.5 Mbps viewers ??long before its CPU.
On a provider that delivered real bandwidth (AWS Lightsail: ~2.4 Gbps public, ~5 Gbps private), a smaller 2 vCPU / 2 GB node completed 250 concurrent WHEP viewers with zero disconnects. The 300-viewer step ran the node out of memory, not bandwidth and not CPU.
Multi-track WHIP ingest costs materially more than single-track SRT ??an ingest-only baseline of ~16% of a core and ~108 MB, and roughly double the per-viewer CPU at the top stable step.
Memory, not CPU and not the network, is what a small standalone node hits first. Budget from measured egress and measured per-reader memory, not from the plan label.

Section 6 gives a side-by-side provider comparison.

1. Background

ppmmx is a low-latency live-video CDN node. The standalone role is the
self-contained deployment: a single instance both ingests a publisher
(WHIP/SRT/RTMP) and serves viewers directly over WebRTC (WHEP), with no
forwarding to other mmx nodes and no control-plane dependency.

The operational question is: how many concurrent viewers does one box hold,
and what limits it? This report answers that for the smallest sensible
instance sizes and two real ingest paths.

2. Method

2.1 Procedure

A stepped concurrency ladder is run against the node:

12 (10-min soak, Round 1)  ->  25  ->  50  ->  75  ->  100  ->  150  ->  200  ->  250  ->  300

At each step, the target number of WHEP readers is opened, held for 2 minutes,
then closed together. A step is recorded PASS only if every session
establishes and closes with no disconnects; the ladder stops at the first step
that fails. Node CPU, RSS, available memory and the node's /metrics are
sampled throughout.

2.2 Tooling and measurement

Readers: whep-loadgen, a purpose-built Go client that opens N real pion/webrtc PeerConnections against the WHEP endpoint. Each session completes full SDP / ICE / DTLS / SRTP and then reads RTP and discards it (no decoding), so the generator is not measuring its own video pipeline. Sessions are spread evenly across two load generators.
Server sampling: process CPU and RSS every 2 s from /proc/<pid>/{stat,status} (CPU as a delta, i.e. instantaneous, not a lifetime average); available memory and load average from /proc.
Node /metrics: session count, outbound bytes/packets, RTP loss, discarded frames.
Per-session: connect time, first-frame time, disconnects, bytes, packets.
Ramp: 500 ms between session starts.

2.3 Test stream

Round 1: one H.264 1280?720@30, ~2.5 Mbps stream, generated with ffmpeg (testsrc2 + libx264) and published over SRT. Video only (no audio).
Round 2: the real product ingest path ??ppobs publishing over WHIP with simultaneous HEVC + H.264 multi-track (three H.264 simulcast layers + three HEVC layers per stream, each session also carrying Opus), pushed from a genuine consumer broadband uplink.

3. Round 1 ??SRT ingest, single H.264 track

Environment: 2 vCPU / 4 GB node and two 2 vCPU / 4 GB load generators, all on
LightNode (Manila), same datacenter.

Step	Sessions	Verdict	Connected	Disconnects	mmx CPU avg / peak	mmx RSS avg / peak	Egress	Loss
baseline	0	??	??	??	0.5% / 2.5%	46 / 48 MB	??	0
12 (10 min)	12	PASS	12/12	0	26.4% / 30%	114 / 116 MB	~31 Mbps	0
25 (2 min)	25	PASS	25/25	0	39.5% / 45%	167 / 175 MB	~64 Mbps	0
50 (2 min)	50	PASS	50/50	0	63.1% / 79.5%	274 / 294 MB	~128 Mbps	0
75 (2 min)	75	FAIL	71/75	5	134% / 177%	618 / 1068 MB	(overloaded)	0*

CPU is per-process where 100% = one full core. *RTP loss stays 0, but at
75 the server logs reader is too slow, discarding 27 frames ??the degradation
appears as dropped frames, not on-wire loss.

From 12 ??50 viewers, CPU rises almost linearly:

mmx CPU% ??15%  +  0.97%  ?  concurrent_viewers     (1 core = 100%)

about 1% of a core per 2.5 Mbps viewer plus ~15% fixed overhead. The last
stable step (50) used 0.63 of a core. At 75 the model breaks: measured CPU
reached 134% average / 177% peak (a two-core box at ~88%, run-queue load
~2.0); the B2 shard could not establish its last sessions (WHEP POST timeouts
and ICE timeouts), first-frame latency for that shard rose from ~0.3 s to
3.1 s average / 9.6 s worst, and the server began discarding frames for slow
readers.

Measured egress is ~2.55 Mbps per viewer. At the last stable step that is
~128 Mbps of a 1 Gbps port (~13%). (Caveat: this is the server's own send
count and may include bytes the provider later dropped; see ?5.)

RSS grows roughly linearly with readers (~14??6 MB/reader) and is reclaimed
after sessions close: the Go runtime returned memory over ~2?? minutes back
toward a ~170 MB baseline (from a 46 MB cold start), mem_alloc fell from
~486 MB to ~77 MB, and goroutines stayed flat at 18 ??no leak.

4. Round 2 ??ppobs WHIP ingest, HEVC + H.264 multi-track

Same node and ladder, but the real ingest path: ppobs over WHIP, three H.264 +
three HEVC simulcast layers and Opus per session, from a consumer uplink.

The uplink is a real internet path, not the LAN-like SRT link of Round 1. The
base layer arrived at 0% loss; the upper simulcast layers lost ~20% ??normal
behaviour for a home uplink once the stream exceeds the available upstream. Loss
on that link is a normal network condition, not a test artefact, but it means
the node's CPU below includes retransmission/discard work.

Step	Verdict	Connected	Disconnects	mmx CPU avg / peak	mmx RSS avg / peak
ingest only (0 viewers)	??	??	??	15.8%	108 MB
12 (3 min)	PASS	12/12	0	32% / 39.5%	118 / 127 MB
25 (2 min)	PASS	25/25	0	44.9% / 64%	173 / 193 MB
50 (2 min)	PASS	50/50	0	130.7% / 157%	315 / 351 MB
75 (2 min)	FAIL	73/75	0	109% / 164%	443 / 653 MB

Compared with Round 1:

The ceiling on this provider is unchanged (~50 stable / 75 fails).
Multi-track WHIP ingest is not free. With zero viewers, ingesting the two publishers (H.264 + HEVC, three layers each, plus two Opus tracks) already costs ~16% of a core and ~108 MB RSS; the idle Round 1 SRT node was ~0% and 46 MB.
Per-viewer CPU roughly doubled: 0.63 core (Round 1, 50 viewers) ??1.31 core (Round 2).

Caveat. The 50-viewer step shows server-side frame discards and the uplink
reconnected mid-run, so these CPU figures bundle multi-track cost with
real-WAN loss handling. The direction is solid; treat the exact multiplier as
approximate.

5. VPS egress measurements

Before scaling to a larger instance, the actual egress of the provider's
instances was measured ??it did not match the plan label. Representative
measurements on a 4 vCPU / 8 GB LightNode instance (Manila):

Test	Result
A??1 TCP, 16 streams	103 Mbps
A??2 TCP, 16 streams	106 Mbps
A??1 UDP (received)	99.5 Mbps (97% loss)
A??1 and A??2 simultaneously	~106 Mbps total
Upload to a third-party endpoint, 1 flow	~104 Mbps
Upload to a third-party endpoint, 8 concurrent flows	~103 Mbps aggregate

Three separately provisioned batches, same-region and cross-region, four cities
(Manila, Taipei, Singapore, Tokyo), never exceeded ~106 Mbps; cross-region paths
fell to ~50 Mbps. There was no tc shaping inside the guest (only the default
mq/fq_codel) and ethtool reported the virtual NIC speed as unknown ??the
cap is on the provider's side, not in the VM.

Consequence: on that network a node tops out at ~39 concurrent 2.5 Mbps
viewers, i.e. network-bound well before its CPU ceiling. On budget VPS,
"1 Gbps" often denotes port speed, a shared uplink, or a burst credit, not
sustained per-instance egress.

6. Provider comparison ??LightNode vs AWS Lightsail

Dimension	LightNode (as tested)	AWS Lightsail (as tested)
Region	Manila (also Taipei / Singapore / Tokyo)	Singapore
Node sizes used	2 vCPU / 4 GB (Rounds 1??); 4 vCPU / 8 GB (egress probe)	2 vCPU / 2 GB
Advertised egress	1 Gbps	1 Gbps (plan)
Measured private-network egress	not measured	~4.6??.0 Gbps
Measured public egress (third-party endpoint)	~50??06 Mbps	~2.4 Gbps
Max completed concurrent WHEP viewers	50 (2 vCPU / 4 GB; 75 failed ??network-induced)	250 (2 vCPU / 2 GB; 300 = out-of-memory)
First binding constraint	Provider network (~100 Mbps cap)	Node memory (2 GB)
Fitness as a high-fan-out live edge	Poor	Suitable (scale RAM)

7. AWS Lightsail capacity run

Same method, re-run on AWS Lightsail (Singapore) on delivering nodes. The node
was 2 vCPU / 2 GB and also runs the production ppmmx, so the benchmark
instance shared the box on a separate port set (webrtc 18888, ice 18188/udp,
srt 17890/udp, api 19996, metrics 19998) ??production was never stopped.

Step	Verdict	Connected	Disconnects	CPU avg / peak	RSS avg / peak	MemAvailable min
12, 25, 50, 75, 100	PASS	all	0	??	??	??
150 (2 min)	PASS	150/150	0	101% / 138%	640 / 713 MB	723 MB
200 (2 min)	PASS	200/200	0	128% / 174%	828 / 940 MB	511 MB
250 (2 min)	PASS	250/250	0	154% / 202%	1014 / 1163 MB	289 MB
300 (2 min)	host lost	(clients connected)	n/a	163% / 204%	1224 / 1514 MB	33 MB

12 ??250 completed cleanly; every client connected and closed with zero disconnects. Nothing failed at 75, unlike the LightNode run.
At 250 the box is at its edge: CPU peak 202% (2-core saturated), RSS ~1.16 GB, 289 MB RAM remaining on the 2 GB box.
At 300 the host ran out of memory. Clients did connect (150 per generator), but available memory fell to ~33 MB, CPU peaked at ~204%, and SSH stopped responding; the node had to be rebooted. 300 is not a passing capacity point ??it is an out-of-memory failure.
The first binding resource is memory, then CPU ??not the network (~2.4 Gbps, barely used).

8. Findings and capacity guidance

A 2 vCPU / 2 GB ppmmx standalone node completed 250 concurrent 2.5 Mbps
WHEP viewers with zero disconnects on a network that delivered ~2.4 Gbps, and
failed at 300 on memory. The earlier "50 stable / 75 fails" result was caused
mainly by the first provider's ~100 Mbps egress cap, not by the node's CPU.

Guidance:

On a network that genuinely delivers ?? Gbps, budget around 250 concurrent 2.5 Mbps viewers for a 2 vCPU / 2 GB node, and treat memory as the binding resource.
Budget roughly 4?? MB of RSS per reader and keep headroom.
Measure the VPS egress and per-reader memory before sizing from a plan label. "1 Gbps" is not a reliable predictor of sustained per-instance egress.
Multi-track WHIP (HEVC + H.264) ingest costs materially more than single-track SRT at the same viewer count; plan ingest capacity accordingly.

(Rounds 1?? remain a valid relative comparison of SRT vs. multi-track WHIP
ingest cost; only their absolute viewer ceiling was distorted by the first
provider's egress cap.)

9. Limitations and caveats

Ingest realism differs by round. Round 1 was a single, video-only H.264 track over SRT. Round 2 used the real multi-track WHIP path but from a consumer uplink with real loss, so its CPU figures mix multi-track cost with loss handling. Neither round used a loss-free multi-track source.
Same-region clients. Load generators were co-located with the node, so this measures server capacity; real internet paths add loss/jitter that reduce effective capacity.
Small instances only. 2 vCPU / 4 GB (Rounds 1??) and 2 vCPU / 2 GB (Lightsail). No dedicated 4 vCPU / 8 GB run was completed.
The Lightsail node was shared with production. The benchmark instance ran alongside the live production ppmmx on the same 2 GB box (separate ports, production untouched). This makes the memory ceiling worse than a dedicated node, so the 250-viewer result is a conservative floor.
Short holds. Only the 12-viewer Round 1 step was a 10-minute soak; all other steps were 2-minute holds. Long-term stability (thermals, multi-hour GC, memory creep) is not established.
Isolated mode. Publish whitelist and playback auth were disabled (no control plane) so the media plane could be tested in isolation; production runs with them enabled.
Load-generator headroom was not instrumented. Each Lightsail generator backed up to 150 sessions with no observed bottleneck, but its own CPU was not sampled.

10. Reproducibility

All tooling is scripted:

# server (A)
sudo ./mmx-host-setup.sh --binary ./mmx-linux-amd64 --config ./standalone.test.yml

# each load generator (B1, B2)
./loadgen-host-setup.sh --binary ./whep-loadgen-linux-amd64

# drive the ladder from B1
MMX_HOST=<A_IP> MMX_SSH=root@<A_IP> ./loadgen-fleet.sh \
  --hosts "local,root@<B2_IP>" \
  --loadgen ~/whep-loadgen-linux-amd64 \
  --bitrate 2500k --out ./results

The runner splits each step across hosts, samples the server's CPU/RSS and
/metrics throughout, writes a CSV per step, and stops at the first step that
cannot establish every session. standalone.test.yml disables the admin
listener and opens /metrics + the control API to the load generators only; the
Lightsail run used an equivalent config on a separate port set so it could
coexist with the production node.

Per-step CSVs and logs for every run in this report are archived alongside it.