Live Streaming at Scale: Why Latency Is the Hard Part

The hard problem in live streaming is not bandwidth or picture quality. It is that a goal scored at 20:14:03 should reach a million screens at 20:14:05, and the standard way of delivering internet video was designed to make that impossible. Everything interesting in live streaming over the last decade has been an attempt to claw back seconds that were deliberately given away.

The glass-to-glass budget

Glass-to-glass latency is the delay between the camera lens and the viewer’s display. It is a sum, and it helps to see where each part goes:

1. Capture and encode. A hardware encoder needs a group of pictures to work with — tens to hundreds of milliseconds, more if it uses B-frames for efficiency.
2. Contribution. Getting the feed from the venue to the origin.
3. Transcode and package. Producing the ABR ladder and cutting it into segments. A segment cannot be published until it is complete, so its full duration lands here whether you like it or not.
4. Distribution. The CDN edge fetching, caching and serving those segments.
5. Player buffer. Traditionally the largest single term, and the most deliberate.
6. Decode and render. Small, though on a television not always as small as you would hope.

Only the buffer is a choice. It exists because a stall is worse than a delay: hold ten seconds of playback in hand and a ten-second hiccup is invisible; shrink it to one second and the same hiccup is a visible freeze. That trade-off drives everything below.

Contribution is not distribution

Contribution is one stream from the venue to the encoder or origin: one connection, high bitrate, and if it breaks nothing works. Distribution is that content going out to a very large audience, where any individual connection matters far less than aggregate cost. Opposite requirements, so different protocols.

RTMP and SRT

RTMP, the Real-Time Messaging Protocol, was created by Macromedia for Flash Player in the early 2000s. Adobe ended Flash Player support at the end of 2020 and RTMP is long dead as a playback protocol — yet it remains the most widely supported contribution format in the world, because every encoder and streaming appliance speaks it and the major platforms still accept it. Mux’s protocol guide puts its glass-to-glass latency at two to five seconds. It runs over TCP, which is both its virtue (ordered, reliable) and its weakness: one lost packet stalls everything behind it while it is retransmitted.

SRT, Secure Reliable Transport, was designed for exactly the case RTMP handles badly: a long, lossy, unmanaged internet path. The Internet-Draft published by Haivision engineers in January 2024 describes a user-level protocol over UDP that recovers loss with selective retransmission, optionally adds forward error correction, and encrypts with AES at 128, 192 or 256 bits. Crucially it lets the operator set a latency budget: within that window it retransmits, and beyond it it drops rather than stalling. That is the right behaviour for live video and the wrong behaviour for a file transfer, which is why it is a separate protocol.

Why HLS and DASH added ten to forty-five seconds

Segmented HTTP streaming won distribution because it turned video into cacheable static files, which meant any CDN could carry it at any scale. The cost was latency, and it compounds.

The packager cannot publish a six-second segment until six seconds have elapsed, and the manifest cannot advertise it until it is published. On top of that, clients have historically started playback several segments behind the live edge to guarantee a buffer — Amazon’s media engineering blog notes that beginning three segments behind the last available one produces an 18-second forward buffer with six-second segments, and describes traditional HLS latency running between 40 and 60 seconds in the field. The SRT draft makes the same point more gently, noting that HTTP-based streaming introduces end-to-end delay measured in “a few tens of seconds”.

Shortening segments helps and then stops helping: two-second segments cut the buffer term but triple the number of HTTP requests and manifest reloads, and hurt encoding efficiency because each segment needs its own keyframe.

Low-latency HLS and DASH

The fix in both ecosystems is the same idea: publish pieces of a segment before the segment is finished.

LL-HLS was introduced by Apple in June 2019 and folded into the HLS specification itself from revision 7 of the “HTTP Live Streaming 2nd Edition” Internet-Draft. Apple’s documentation describes five mechanisms working together, and each removes a specific delay:

• Partial segments (EXT-X-PART). A six-second segment is also published as thirty parts of 200 milliseconds each, available almost immediately.
• Blocking playlist reload. The client asks for a future segment by number; the server holds the request open until that segment exists. This eliminates polling, and with it the average half-poll-interval of wasted time.
• Preload hints (EXT-X-PRELOAD-HINT). The client requests the next part before it exists, and the server answers the moment it does — one fewer round trip.
• Playlist delta updates (EXT-X-SKIP). Only the changed tail of a playlist is sent, because the client now reloads it constantly.
• Rendition reports. Each playlist carries the current position of the other rungs, so switching quality does not cost extra round trips.

Apple’s stated goal is to bring latency “into the range of standard television broadcasts”. The example playlist in the documentation uses parts of about 0.33 seconds and a PART-HOLD-BACK of one second.

LL-DASH reaches the same place through CMAF chunks and HTTP chunked transfer encoding: the packager writes chunks into a segment file that is still open, and the CDN forwards bytes as they arrive rather than waiting for a complete object. The joint DASH-IF and DVB report on low-latency live services, dated 28 July 2017, sets the targets: end-to-end latency of a few seconds, and a live-edge start-up delay of roughly 0.6 to 2 seconds to match conventional television.

WebRTC for sub-second, and what it costs

Below about two seconds, HTTP segmentation runs out. WebRTC — a W3C Recommendation since 26 January 2021, alongside a set of IETF standards — was built for two-way conversation and delivers media over UDP with essentially no buffer.

The trade is structural. WebRTC does not produce cacheable files, so a CDN cannot fan it out for free; scale requires selective forwarding servers holding a session per viewer, at a cost that rises roughly with audience size. And with no buffer, a network hiccup is not smoothed over — it is visible at once, as a frozen frame or a quality drop. That is why WebRTC tends to appear where low latency is worth real money.

Approach	Typical role	Latency character	Main cost
RTMP over TCP	Contribution	Seconds; two to five glass-to-glass per Mux	Head-of-line blocking on lossy paths; obsolete for playback
SRT over UDP	Contribution	Configurable latency window, sub-second to a few seconds	Newer, so less universal encoder support than RTMP
Standard HLS / DASH	Distribution	Tens of seconds	None — this is the cheap, resilient, infinitely cacheable option
LL-HLS / LL-DASH	Distribution	Broadcast-like, a few seconds	Smaller buffers, far more requests, stricter CDN requirements
WebRTC	Distribution at small or costly scale	Sub-second	No HTTP caching; per-viewer server capacity; no buffer to hide loss

Why sports and betting forced the issue

For most live content, a thirty-second delay is invisible — nobody knows when the concert actually happened. Sport broke that, because sport is watched socially and alongside other feeds. If your stream is thirty seconds behind, a neighbour’s cheer, a phone notification or a television in the next room tells you the result before your screen does, and viewers of the same match on different services end up many seconds apart.

In-play betting turned that annoyance into a commercial problem. Odds are suspended and re-priced around events on the pitch, so a bettor whose video lags the market is at a disadvantage — while a bettor with a feed faster than the operator’s own is a genuine risk to that operator. Once latency affects the fairness of a transaction, it stops being a quality-of-experience metric and becomes a requirement, which is why the sports and wagering sector pushed low-latency adoption harder than entertainment ever did.

What to do about it

• Decide what latency is actually worth to you. If nobody is betting, comparing or reacting in real time, standard HLS is cheaper, more resilient and perfectly adequate.
• Fix contribution before distribution. A jittery venue uplink on RTMP will ruin a low-latency delivery chain; SRT with a sensible latency window is usually the higher-value change.
• Budget the whole chain, not just the player. Encoder GOP length, segment duration and buffer target each contribute, and cutting only one of them buys less than you expect.
• Expect more rebuffering as you cut latency, and measure both. A stream three seconds behind that stalls twice an hour is worse than one fifteen seconds behind that never does.
• Check that your CDN supports what you enable. Blocking playlist reloads and chunked transfer encoding both require specific edge behaviour, and LL-HLS falls back to ordinary HLS quietly when it is not met.