End-to-End Streaming: DMA-BUF to H.265 Hardware Encode to RTP/UDP

The Portal streaming daemon captures compositor output from a Wayland display, transforms GPU-compressed DMA-BUF frames into linear pixel data, encodes them as H.265 via V4L2 hardware acceleration, packetizes into RTP, and broadcasts over UDP to AR glasses. This page traces every stage of that pipeline — from the Wayland protocol handshake through the EGL UBWC linearizer, the GStreamer encoding chain, and the optional HMAC-SHA256 stream authentication layer — with precise code-level evidence for each architectural decision.

Pipeline Overview

The streaming pipeline is a single-process C daemon (portal_stream) that runs as a systemd service alongside the Wayfire compositor. It operates as a Wayland client, consuming frames via the zwlr_export_dmabuf_manager_v1 protocol, then feeding them through a GStreamer pipeline for encoding and transmission.

flowchart LR
    subgraph Compositor
        WF[Wayfire Compositor] -->|renders GLASSES-1 output| DMABUF[DMA-BUF Frame\nUBWC modifier]
    end
    subgraph portal_stream binary
        WL["Wayland Client\nzwlr_export_dmabuf_v1"] -->|frame_object callback| EGL["EGL Linearizer\neglCreateImageKHR → glReadPixels"]
        EGL -->|linear BGRx\nCPU buffer| APPSRC["appsrc\nGStreamer"]
        APPSRC --> VC[videoconvert\nBGRx → NV12]
        VC --> ENC["v4l2h265enc\n8 Mbps CBR\nSPS/PPS at IDR"]
        ENC --> PAY[rtph265pay\nPT=96]
        PAY --> HMAC["HMAC Pad Probe\nRFC 5285 ext\noptional"]
    end
    HMAC -->|UDP broadcast| UDP["udpsink\n192.168.50.255:5000"]
    UDP -.->|5GHz WiFi ~1ms| GLASSES[AR Glasses\nH.265 decode → VR render]

The daemon maintains a continuous fire-and-forget stream: packets are silently dropped if the glasses disconnect, and the pipeline resumes without reinitialization when they reconnect. This design eliminates the need for connection-state tracking and guarantees that mid-stream reconnects can decode immediately — provided parameter sets are present at every IDR frame.

Sources: portal/streaming/portal_stream.c#L1-L35, portal/systemd/portal-stream.service#L1-L34

Stage 1: Wayland DMA-BUF Capture

The daemon begins by connecting to the Wayland display socket (wayland-1) and performing two registry roundtrips to discover available outputs and bind the zwlr_export_dmabuf_manager_v1 interface. The registry listener registers the dmabuf manager on first sight and binds up to 8 wl_output objects into a struct output_entry array, capturing each output’s name and current resolution via the output listener callbacks.

Sources: portal/streaming/portal_stream.c#L191-L211, portal/streaming/portal_stream.c#L162-L189

Output Selection

The daemon selects a target output by substring-matching the output_name parameter (defaulting to HEADLESS-1) against discovered output names. When multiple outputs match, it picks the one with the highest pixel area — this handles headless virtual outputs that may report identical names but different resolutions. If no match is found, the first available output is used as a fallback.

Sources: portal/streaming/portal_stream.c#L738-L757

Frame Capture Lifecycle

The capture loop is driven by four Wayland protocol callbacks registered against each zwlr_export_dmabuf_frame_v1 object:

Callback	Trigger	Responsibility
frame_frame	Frame metadata arrives	Records fourcc, modifier (UBWC), dimensions on first frame
frame_object	DMA-BUF file descriptor arrives	EGL linearization → GStreamer buffer push
frame_ready	Frame fully delivered	Destroys frame object, requests next capture
frame_cancel	Compositor cancels frame	16ms backoff retry, then re-captures

The capture is strictly sequential: frame_ready issues the next capture_output call, creating a self-sustaining loop that dispatches via wl_display_dispatch. The frame_cancel path introduces a 16ms usleep backoff to prevent spin-loops on persistent compositor errors — a deliberate choice over immediate retry, validated by the semgrep rule frame-cancel-no-backoff.

Sources: portal/streaming/portal_stream.c#L392-L437, portal/streaming/portal_stream.c#L771-L779

Stage 2: EGL UBWC Linearization

This is the most architecturally significant stage in the pipeline. On the Snapdragon X Elite platform, wlroots’s gles2 renderer allocates compositor output DMA-BUFs with DRM_FORMAT_MOD_QCOM_COMPRESSED (0x0500000000000001) — Qualcomm Universal Bandwidth Compression (UBWC). UBWC is a tiled/compressed memory layout where the raw dmabuf bytes do not represent a linear pixel array. GStreamer’s videoconvert reads raw dmabuf bytes via mmap, which produces garbage when the modifier is UBWC.

The Problem

When portal_stream naively passed the dmabuf fd into GStreamer, videoconvert interpreted the compressed bytes as linear BGRx, producing garbage into v4l2h265enc and ultimately garbage on the AR glasses. Debug flags (TU_DEBUG=noubwc, FD_DEBUG=noubwc) had no effect — wlroots explicitly requests modifiers via gbm_bo_create_with_modifiers, and the driver returns UBWC as valid regardless.

Sources: portal/streaming/STREAMING_UBWC_FIX.md#L21-L42

The Solution: EGL Import + glReadPixels

The linearizer creates a one-time EGL/GBM context during daemon initialization (portal_egl_init), then reuses a single FBO and texture across all frames. Per-frame linearization (portal_egl_linearize) follows a five-step sequence:

flowchart TD
    A["eglCreateImageKHR\nmodifier-aware attrs"] --> B["glEGLImageTargetTexture2DOES\nimport as GL texture"]
    B --> C["glFramebufferTexture2D\nattach to scratch FBO"]
    C --> D["glReadPixels\nGPU → CPU linear BGRx"]
    D --> E["eglDestroyImageKHR\nrelease EGL resource"]
    E --> F["close dmabuf fd\nkernel can release"]

The EGL import is modifier-aware: when the modifier is not DRM_FORMAT_MOD_LINEAR, the function adds EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT and EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT attributes so the GPU’s EGL implementation understands the UBWC layout and produces correct linear pixels.

The output buffer (width * height * 4 bytes) is allocated via malloc, wrapped into a GstBuffer using gst_buffer_new_wrapped_full with GST_MEMORY_FLAG_NO_SHARE and a free destroy notify. The buffer’s PTS and duration are calculated from frame_count and configured fps, ensuring correct timestamps for the encoder’s rate control.

Sources: portal/streaming/portal_stream.c#L213-L336, portal/streaming/portal_stream.c#L342-L390

Initialization Details

The EGL context initialization opens /dev/dri/card0, creates a GBM device, obtains an EGL display via eglGetPlatformDisplayEXT with EGL_PLATFORM_GBM_MESA, and establishes a GLES2 context. Three EGL extension function pointers are cached for per-frame use:

Function Pointer	EGL Extension	Purpose
eglCreateImageKHR	EGL_KHR_image	Import dmabuf as EGLImage
eglDestroyImageKHR	EGL_KHR_image	Release EGLImage after readback
glEGLImageTargetTexture2DOES	GL_OES_EGL_image	Bind EGLImage to GL texture

Sources: portal/streaming/portal_stream.c#L219-L274

Performance Characteristics

The glReadPixels call is a synchronous GPU→CPU stall moving approximately 480 MB/s for BGRx at 1080p60 (2560×1080×4×60 ≈ 633 MB/s). On the 12-thread Snapdragon X Elite, this consumes ~87% of one core with ~147 MB RSS — sustainable but acknowledged as a bottleneck. The long-term optimization path is GPU-side color conversion via gst_glcolorconvert, which would move BGRx→NV12 onto the GPU and only pass NV12 to CPU.

Sources: portal/streaming/STREAMING_UBWC_FIX.md#L82-L87, portal/streaming/STREAMING_UBWC_FIX.md#L200-L209

Stage 3: GStreamer Encoding Pipeline

The encoding pipeline is constructed as a single GStreamer launch string via gst_parse_launch, containing five linked elements plus an optional debug tee. The pipeline operates in live streaming mode with no synchronization on the sink.

Pipeline String Anatomy

Element	Caps / Config	Role
appsrc	video/x-raw,format=BGRx	Receives linearized frames from C code
queue	max-size-buffers=2, leaky=downstream	Decouples capture from encode; drops oldest
videoconvert	video/x-raw,format=NV12	BGRx → NV12 color space conversion
v4l2h265enc	video_bitrate=8000000, video_bitrate_mode=1, prepend_sps_and_pps_to_idr=1	Hardware H.265 encode via V4L2 M2M
rtph265pay	pt=96	RTP packetization per RFC 7798
udpsink	bind-address, multicast-iface=uap0	UDP broadcast to glasses

The appsrc is configured with format=time, is-live=true, and block=false — it never blocks the Wayland event loop waiting for the pipeline to consume buffers. The queue’s leaky=downstream policy means if the encoder falls behind, the oldest pending frame is dropped rather than stalling capture.

Sources: portal/streaming/portal_stream.c#L566-L598

V4L2 H.265 Encoder Configuration

The encoder is configured with three critical V4L2 controls:

video_bitrate=8000000 (8 Mbps) — targets high-quality AR streaming at 60fps for a 2560×1080 virtual display.

video_bitrate_mode=1 (CBR) — constant bitrate mode ensures predictable network utilization on the WiFi AP interface.

prepend_sps_and_pps_to_idr=1 — this is the fix for the mid-stream reconnect bug. Without it, v4l2h265enc emits VPS/SPS/PPS only at stream start. Glasses that disconnect and reconnect mid-stream would receive IDR slices without parameter sets and fail to decode. With this control, every IDR frame is prefixed with VPS/SPS/PPS NAL units, enabling decode regardless of connection timing.

An earlier attempt to achieve the same effect via rtph265pay config-interval=-1 was a regression — it produced a near-black decoded image due to an apparent bug in rtph265pay’s VPS/SPS/PPS reinsertion on GStreamer 1.28. The encoder-side approach avoids this entirely.

Sources: portal/streaming/portal_stream.c#L582-L598, portal/streaming/STREAMING_UBWC_FIX.md#L90-L108

Output Caps Negotiation

The encoder output is constrained to video/x-h265,stream-format=byte-stream,alignment=au, which tells rtph265pay to expect Annex B byte-stream format with access-unit alignment. This is essential for correct RTP packetization — the payloader must know where NAL unit boundaries are to apply FU (Fragmentation Unit) splitting for NALs exceeding the MTU.

Sources: portal/streaming/portal_stream.c#L589-L591

Debug Snapshot Tee

When the PORTAL_STREAM_DEBUG_SNAP environment variable is set to a file path, the pipeline inserts a tee after videoconvert that branches to a PNG snapshot. This isolates whether corruption originates in the dmabuf capture/EGL linearization (before the tee) or in the encoder (after the tee). Production deployments leave this unset.

Sources: portal/streaming/portal_stream.c#L567-L580

Stage 4: HMAC-SHA256 RTP Authentication

When the PORTAL_STREAM_HMAC_KEY environment variable is set (64 hex characters = 32 bytes), the daemon attaches a GStreamer pad probe to udpsink’s sink pad. This probe intercepts every outbound RTP packet, computes an HMAC-SHA256 tag over the full packet bytes (header + payload), and appends it as an RFC 5285 two-byte header extension.

Why Two-Byte Extension Form

The 32-byte HMAC-SHA256 tag exceeds the one-byte RFC 5285 extension form’s 16-byte per-element limit (the one-byte form can address at most 4 four-byte words). The two-byte form (profile 0x100, app id 1) accommodates the full tag and matches what the glasses receiver parses. Receivers that don’t validate the extension simply ignore it per RFC 3550 §5.3.1, preserving backward compatibility with unauthenticated glasses.

Sources: portal/streaming/portal_stream.c#L96-L103, portal/streaming/portal_stream.c#L452-L534

Two-Stage Buffer Mapping

The probe uses two complementary GStreamer APIs in sequence:

Stage 1 — GstBuffer map (READ): Maps the buffer as raw bytes to compute HMAC-SHA256 over the complete on-wire RTP packet. This gives access to the actual bytes that will be transmitted.

Stage 2 — GstRTPBuffer map (READWRITE): Maps the same buffer with RTP-aware semantics to add the extension via gst_rtp_buffer_add_extension_twobyte_header, which automatically sets the X bit and the 0x100 extension profile in the RTP header.

Sources: portal/streaming/portal_stream.c#L493-L533

Fail-Closed Policy

The authentication probe follows a strict fail-closed policy:

Condition	Action	Rationale
Buffer missing or wrong probe type	Pass through unchanged	Non-RTP event; no auth needed
Buffer not writable	DROP + log	Cannot modify; never send unauthenticated
GstBuffer map fails	DROP + log	Cannot read for HMAC; fail-closed
HMAC computation fails	DROP + log	Never send a packet that should have been tagged but wasn’t
GstRTPBuffer map fails	DROP + log	Cannot add extension; fail-closed

Sources: portal/streaming/portal_stream.c#L460-L472, portal/streaming/portal_stream.c#L487-L533

HMAC Utility Implementation

The HMAC computation uses OpenSSL’s HMAC() one-shot API with EVP_sha256(). The key is loaded from hex-encoded environment input via portal_hex_decode_key, which validates exact length (64 hex chars), rejects non-hex characters, and decodes using a branch-free lookup table. Although OpenSSL 3.0+ deprecates HMAC() in favor of EVP_MAC, the function remains exported and is the simplest vetted API for this use case.

Sources: portal/streaming/hmac_util.c#L1-L68, portal/streaming/hmac_util.h#L1-L45

Stage 5: UDP Transport Configuration

The udpsink element is configured for broadcast delivery over the uap0 access point interface — the dedicated WiFi AP that the glasses connect to. Three properties control the transport:

bind-address — sourced from PORTAL_STREAM_BIND (default 192.168.50.1), the uap0 interface address. This ensures packets originate from the correct interface even when multiple network interfaces exist.

multicast-iface=uap0 — explicitly selects the WiFi AP interface for multicast/broadcast delivery, preventing the kernel from choosing a different default route.

sync=false async=false — disables GStreamer’s clock synchronization and async state changes. The pipeline should emit frames as fast as the encoder produces them, with no buffering or timestamp-based pacing.

Sources: portal/streaming/portal_stream.c#L543-L598, portal/systemd/portal-stream.env.example#L1-L22

Pipeline Injection Prevention

All string values destined for the GStreamer pipeline string (host, port, bind address, multicast interface) pass through validate_safe_token(), which rejects shell metacharacters (;, |, $, `) that could inject pipeline syntax via gst_parse_launch. This is enforced by a dedicated semgrep rule (gstreamer-pipeline-injection) and validated by cmocka unit tests.

Sources: portal/streaming/portal_stream.c#L441-L450, portal/streaming/tests/test_portal_stream_extended.c#L1-L60

Rust Streaming Library

Alongside the C daemon, the portal-stream Rust crate provides protocol-level utilities for RTP packet construction, H.265 NAL unit parsing, and UDP streaming. This crate serves the receiver side and other consumers that need to construct or parse RTP/H.265 data programmatically.

RTP Packet Handling

Type	Purpose
RtpHeader	Parse/serialize the 12-byte fixed RTP header (RFC 3550)
RtpPacket	Header + payload container with full serialization
RtpBuilder	Stateful builder with auto-incrementing sequence/timestamp

The RtpBuilder advances the sequence number by 1 and timestamp by a caller-specified increment per build_frame call. Both fields use wrapping arithmetic to handle u16/u32 overflow correctly — verified by property tests covering u16::MAX sequence wrapping and u32::MAX timestamp wrapping.

Sources: portal/stream/src/rtp.rs#L1-L217, portal/stream/src/error.rs#L1-L41

H.265 NAL Unit Utilities

The nal module classifies H.265 NAL unit types by extracting bits 1–6 of the header byte (the nal_type field). Key types include VPS (32), SPS (33), PPS (34), IDR (19/20), and trailing pictures (0–9). The is_keyframe() and is_parameter_set() predicates support receiver-side logic for detecting decoder-critical NALs. The find_start_codes function locates 3-byte Annex B start codes (0x00 0x00 0x01) in a byte buffer.

Sources: portal/stream/src/nal.rs#L1-L92

Optional UDP HMAC Feature

The udp-hmac Cargo feature gates an alternative HMAC authentication scheme for UDP transport: each datagram is wrapped as seq (BE 4 bytes) || payload || tag (16 bytes) with HMAC-SHA256 truncated to 16 bytes. This mirrors the voice pipeline’s packet authentication format and provides a Rust-native path for authenticated streaming when the C daemon’s RFC 5285 RTP extension approach is not suitable.

Sources: portal/stream/src/udp.rs#L1-L184, portal/stream/Cargo.toml#L1-L25

Systemd Service Integration

The daemon runs as portal-stream.service, configured to start after portal.service (the Wayfire compositor) and as part of that service’s dependency tree. The service unit implements a startup guard: ExecStartPre polls for the wayland-1 socket existence up to 30 times (15 seconds) before launching, ensuring the compositor is ready.

The ExecStart line constructs the daemon invocation with the output name as HEADLESS-1, the RTP destination from $PORTAL_STREAM_TARGET, and the port from $STREAM_PORT (defaulting to 5000). The Restart=always with RestartSec=3 ensures rapid recovery from crashes.

Environment Variable	Source	Purpose
PORTAL_STREAM_TARGET	Required — no fallback	RTP destination IP (glasses unicast or broadcast)
PORTAL_STREAM_BIND	Optional (default 192.168.50.1)	GStreamer bind address
PORTAL_MULTICAST_IFACE	Optional (default uap0)	Outbound interface for broadcast
STREAM_PORT	Optional (default 5000)	RTP destination port
PORTAL_STREAM_HMAC_KEY	Optional	64-char hex HMAC-SHA256 key
PORTAL_STREAM_DEBUG_SNAP	Optional	Debug PNG snapshot path

Sources: portal/systemd/portal-stream.service#L1-L34, portal/systemd/portal-stream.env.example#L1-L22

Critical Regression Guards

Three documented regressions have explicit source-code comments warning against reintroduction. These represent the pipeline’s most fragile interaction points:

Regression A: GstVideoMeta with stride/offset

Adding gst_buffer_add_video_meta to frame_object() with the wlroots-reported stride breaks streaming because the dmabuf size (12 MB) doesn’t match the expected size (10.8 MB for 2560×1080×4). The videoconvert element misinterprets the mismatched stride/offset, producing corrupted encoder input. The source carries a CRITICAL comment: do not add GstVideoMeta here.

Sources: portal/streaming/portal_stream.c#L375-L377, portal/streaming/STREAMING_UBWC_FIX.md#L112-L116

Regression B: rtph265pay config-interval=-1

This produced a near-black decoded image. The mechanism is unclear but likely involves rtph265pay mishandling VPS/SPS/PPS reinsertion on GStreamer 1.28. The encoder-side prepend_sps_and_pps_to_idr=1 achieves the same effect without payloader involvement.

Sources: portal/streaming/STREAMING_UBWC_FIX.md#L118-L120

Regression C: Direct videoconvert on UBWC dmabuf

Removing the EGL linearizer silently regresses to garbage output. The diagnostic signature is a first-frame log line showing mod=0x0500000000000001 (UBWC) without the EGL code path active.

Sources: portal/streaming/STREAMING_UBWC_FIX.md#L122-L124

Build and Hardening

The daemon is compiled with full security hardening via build.sh, which delegates to gcc with flags pinned by the Makefile:

Category	Flag	Effect
Warnings	-Werror -Wall -Wextra	All warnings are errors
Stack protection	-fstack-protector-strong	Canaries on functions with local char arrays > 8 bytes
Buffer checks	-D_FORTIFY_SOURCE=2	libc buffer-overflow detection (requires -O2+)
PIE/ASLR	-fPIE -pie	Position-independent executable
RELRO	-Wl,-z,relro -Wl,-z,now	Full RELRO — read-only GOT after load

Wayland protocol bindings are regenerated at build time from wlr-protocols pinned to commit bf4fc79abc359eea5a0edec0ac6d4a2b2955f82a for reproducibility. The build links against EGL, GLESv2, GBM, libdrm, OpenSSL, and four GStreamer libraries (core, app, rtp, allocators).

Sources: portal/streaming/build.sh#L1-L91, portal/streaming/Makefile#L1-L95

Latency Budget

The end-to-end frame-to-glasses latency has been measured against a 20ms budget:

Stage	Measured	Budget
EGL linearize + glReadPixels	~480 MB/s GPU→CPU stall	—
V4L2 H.265 encode	~5 ms	10 ms
RTP packetize (rtph265pay)	~120 ns/frame	—
RTP build + serialize (64KB NAL)	1.5 µs	5 µs
WiFi 5GHz UDP transit	~1 ms	5 ms
Total frame-to-glasses	~6 ms	20 ms

Sources: docs/LATENCY_BUDGET.md#L43-L61

Next Steps

This page covered the full capture-encode-transmit pipeline. The following pages dive deeper into specific aspects:

• RTP Packetization and HMAC-SHA256 Stream Authentication — detailed protocol analysis of the RTP extension format and the Rust receiver-side HMAC validation
• Latency Budget: Per-Stage Budgets and Benchmark Methodology — complete benchmark methodology across all subsystems including the streaming pipeline’s criterion benchmarks