Skip to content
Demain Docs
Waitlist →

Overview: Spatial Computing Platform for AR Glasses

Portal is a Wayland compositor stack that transforms a laptop — the HP EliteBook Ultra G1q (“Spaceboard”, Snapdragon X Elite) — into a spatial computing host that renders and streams a full desktop environment to INMO AIR3 AR glasses over a dedicated 5 GHz Wi-Fi link. Unlike traditional laptops, the Spaceboard has no conventional monitor interaction paradigm: its tactile surface serves as both keyboard and trackpad, while all visual output flows to the glasses through hardware-encoded H.265 video streamed over RTP/UDP. The platform adds 3DoF head tracking, on-device voice AI powered by the Hexagon DSP, an emulation layer for x86/Windows applications, and a spatial domain model that lets windows be organized across virtual zones — center, left peripheral, right peripheral, and overlay — projected into the wearer’s field of view.

Sources: README.md#L1-L4, portal/docs/architecture.md#L1-L22

What Portal Does

Section titled “What Portal Does”

At its core, Portal solves a deceptively simple problem: run a complete Linux desktop on ARM64 hardware and make it usable through AR glasses instead of a monitor. This requires rethinking every layer of the traditional stack — from how frames are rendered and captured, to how input flows when there is no physical keyboard or trackpad in the conventional sense, to how AI workloads share a single NPU between voice processing and language model inference.

The result is a system where the Spaceboard’s Snapdragon X Elite runs Wayfire (a Wayland compositor) with two logical outputs: the eDP-1 tactile surface (2240×1400) used exclusively for the on-screen keyboard, and a virtual GLASSES-1 output (2560×1080) whose composited frames are captured via DMA-BUF, hardware-encoded to H.265 by the Qualcomm IRIS encoder, packetized as RTP, and broadcast over UDP to the glasses. The glasses run an Android/Unity receiver that decodes the stream, renders it to a virtual quad in 3D space, and sends 3DoF head orientation data back over UDP — closing the loop with homography-based spatial warp correction on the compositor side.

Sources: README.md#L10-L33, portal/docs/architecture.md#L19-L47, docs/x-elite-deployment-state.md#L67-L103

System Architecture at a Glance

Section titled “System Architecture at a Glance”

The following diagram shows the major data flows across the platform’s subsystems:

flowchart TB
    subgraph Spaceboard["Spaceboard (Snapdragon X Elite)"]
        WF["Wayfire Compositor<br/>DRM backend · eDP-1 + GLASSES-1"]
        SP["Spatial Plugins<br/>Zone assignment · Dimension switching"]
        WP["Spatial Warp<br/>Homography head-tracking correction"]
        WM["Window Manager Daemon<br/>Toplevel tracking · Zone policy"]
        
        KB["Keyboard Daemon<br/>DRM/KMS scanout on eDP-1<br/>Fusion engine · evdev touch"]
        
        STR["Streaming Pipeline<br/>DMA-BUF → IRIS H.265 → RTP/UDP"]
        
        VAD["Voice Pipeline<br/>VAD → STT → NLU → TTS"]
        NPU["Hexagon DSP<br/>QNN HTP v73 · 45 TOPS"]
        LLM["LLM Daemon<br/>Qwen3-0.6B on DSP"]
        
        NET["Network AP<br/>5 GHz hostapd · dnsmasq DHCP"]
        PCP["PCP Daemon<br/>App registry · Capability routing"]
        
        WF --> SP
        SP --> WP
        WM <-->|"Unix socket IPC"| SP
        KB -->|"uinput → virtual devices"| WF
        WF -->|"DMA-BUF capture"| STR
        VAD --> NPU
        LLM --> NPU
    end
    
    STR -->|"RTP/UDP H.265"| GLASSES
    NET -.->|"5 GHz Wi-Fi AP"| GLASSES
    
    subgraph GLASSES["INMO AIR3 Glasses"]
        RX["Android/Unity Receiver<br/>H.265 decode · 3DoF tracking"]
    end
    
    GLASSES -->|"3DoF head orientation UDP"| WP
    GLASSES -->|"Mic audio UDP"| VAD

Sources: portal/docs/architecture.md#L1-L47, docs/x-elite-deployment-state.md#L67-L103, README.md#L10-L33

Major Subsystems

Section titled “Major Subsystems”

Portal is organized as a Rust workspace of 30+ crates plus a C streaming daemon and C++ Wayfire plugin shims. Each subsystem is independently buildable, testable, and deployable as a systemd service. The table below maps each subsystem to its role:

Subsystem Crate(s) Role Service
Spatial Domain Model portal-spatial Zone types, dimension configs, assignment policies, IPC protocol, homography math (library)
Spatial Plugin portal-spatial-plugin Wayfire cdylib: Wayland global, zone registry, dimension state portal.service
Spatial Warp portal-warp Homography-based GLSL shaders for head-tracking correction portal.service
Window Manager portal-wm Foreign toplevel tracking, zone assignment policy, IPC client portal-wm.service
Keyboard/Input portal-input DRM/KMS direct scanout, fusion engine, gesture recognition, evdev portal-input.service
Streaming (C) portal/streaming/ DMA-BUF capture → IRIS H.265 encode → RTP/UDP broadcast portal-stream.service
Streaming (Rust) portal-stream RTP packetization, HMAC-SHA256 authentication, protocol types (library/daemon)
Voice portal-voice VAD (QNN DSP), STT (sherpa-onnx), NLU (MiniLM), TTS (Elara VITS) portal-voice.service
LLM portal-llm Qwen3-0.6B inference via GenieX llama.cpp on Hexagon DSP portal-llm.service
NPU Runtime portal-npu-runtime Safe Rust wrappers for Qualcomm Hexagon FastRPC (library)
PCP portal-pcp/* (15 crates) Portal Capability Protocol: app registry, AT-SPI2, IPC, daemon portal-pcp.service
Context Engine portal-context Event ingestion, decision making, template synthesis (library)
Launcher portal-launcher Action registry, fuzzy search, plugin system portal-launcher.service
Shell portal-shell Status bar, notifications, window switcher (library)
Style portal-style Design token system: colors, typography, spacing, themes (library)
Common portal-common Shared error types, IPC auth, test harness, mock hardware (library)

Sources: Cargo.toml#L1-L45, README.md#L94-L127, portal/docs/architecture.md#L24-L136

The Spatial Domain Model

Section titled “The Spatial Domain Model”

Portal’s defining innovation is the spatial domain model, which extends the flat 2D desktop metaphor into a structured spatial layout that maps naturally to how AR glasses render content. The model has three core abstractions:

Zones divide the 2560×1080 glasses canvas into logical regions: Center (primary focal area), LeftPeripheral and RightPeripheral (side areas for secondary windows), and Overlay (floating UI elements). The zone assignment engine uses view position to automatically route windows — for example, a code editor might land in Center while a chat window drifts to RightPeripheral.

Dimensions represent complete spatial configurations — Desktop, VR, AR, and Tablet modes — each defining which zones are active and how they are arranged. Users switch dimensions with keyboard shortcuts (Ctrl+Alt+Right / Ctrl+Alt+Left), instantly reconfiguring the workspace layout.

Homography-based warp correction applies a perspective transform to the compositor output based on real-time 3DoF head tracking data from the glasses. As the wearer turns their head, the spatial warp plugin recomputes a homography matrix and adjusts the GLSL shader uniforms so the rendered content appears stable and anchored in physical space.

Sources: portal/spatial/src/lib.rs#L1-L92, portal/spatial/src/zone.rs#L12-L32, portal/spatial/src/dimension.rs#L13-L80, portal/protocols/portal-spatial-v1.xml#L8-L55, portal/compositor/wayfire.ini#L1-L40

The Streaming Pipeline

Section titled “The Streaming Pipeline”

The streaming pipeline is the critical path that determines perceived system responsiveness. It operates as a zero-copy DMA-BUF pipeline: Wayfire composites the GLASSES-1 output to a GPU buffer, the streaming daemon imports that buffer via EGL (handling Qualcomm’s UBWC tiled layout by rendering through a scratch FBO to linearize it), feeds the linearized frames into GStreamer’s appsrc, which passes them to the IRIS V4L2 H.265 hardware encoder (v4l2h265enc), then through rtph265pay for RTP packetization, and finally out via udpsink to the glasses.

The end-to-end frame-to-glasses latency is measured at approximately 6 ms (5 ms IRIS encode + 1 ms Wi-Fi transit), well within the 20 ms budget. An optional HMAC-SHA256 stream authentication layer (RFC 5285 two-byte header extension) can sign each RTP packet to prevent tampering or injection on the wireless link.

Sources: portal/streaming/portal_stream.c#L1-L63, portal/docs/architecture.md#L34-L47, docs/LATENCY_BUDGET.md#L43-L61

On-Device AI: Voice and LLM

Section titled “On-Device AI: Voice and LLM”

Portal runs two AI workloads entirely on-device, sharing the Hexagon v73 DSP (45 TOPS):

The voice pipeline flows from the glasses’ microphone (UDP audio at ~2 ms transit) through RNNoise denoising (~1 ms), QNN-based Voice Activity Detection on the DSP (~50 ms), sherpa-onnx Speech-to-Text on CPU (~200 ms), MiniLM-based Natural Language Understanding (~15 ms), and finally Elara VITS Text-to-Speech with encoder on DSP (~800 ms) and decoder via onnxruntime-qnn plugin (~1.6 s). The full voice round-trip completes in approximately 2.7 seconds, under the 5-second budget.

The LLM daemon runs Qwen3-0.6B (Q4_0 GGUF quantization) through GenieX llama.cpp on the Hexagon DSP, achieving 0.3–0.5 second response times with thinking disabled. An NPU scheduler arbitrates DSP access between VAD, TTS, and LLM workloads to prevent resource contention.

Sources: portal/voice/src/lib.rs#L18-L56, portal/voice/README.md#L1-L11, docs/LATENCY_BUDGET.md#L63-L80, docs/x-elite-deployment-state.md#L40-L42, portal/npu-runtime/README.md#L1-L6

Hardware Platform

Section titled “Hardware Platform”

The Spaceboard (HP EliteBook Ultra G1q) is a purpose-built spatial computing host built around the Snapdragon X Elite SoC:

Component Specification Portal Role
SoC Snapdragon X Elite (X1E80100) Primary compute
GPU Adreno X1-85 (Mesa 26.1.2, Freedreno/Turnip) wgpu rendering, Wayland EGL, GLSL shaders
NPU Hexagon v73 (45 TOPS, QNN SDK v2.44) VAD, TTS, LLM inference
Video Encoder Qualcomm IRIS (/dev/video1) H.265 hardware encode for glasses stream
Wi-Fi WCN6855 (ath11k, 5 GHz AP) Dedicated streaming link to glasses
Touch ELAN 04F3:4328 (hid-over-i2c) Tactile surface input (keyboard + trackpad)
Display eDP-1, 2240×1400 @ 60 Hz Tactile keyboard surface (owned by keyboard daemon)
Glasses Output Virtual GLASSES-1, 2560×1080 @ 60 Hz Composited desktop → streamed to INMO AIR3
OS Debian testing (trixie), kernel 7.0.13-iris Custom kernel with IRIS encoder support

The INMO AIR3 glasses serve as the display endpoint: they receive the H.265 RTP stream on port 5600, decode via Android MediaCodec hardware decoder, render to a virtual quad in Unity with 3DoF head tracking, and feed orientation data back to the compositor for warp correction.

Sources: portal/docs/architecture.md#L10-L18, docs/x-elite-deployment-state.md#L9-L24, docs/RENDERING_ARCHITECTURE.md#L57-L82

Service Topology

Section titled “Service Topology”

Portal runs as a constellation of 13 systemd services (plus 2 watchdogs), all communicating through Unix sockets and Wayland protocols under the shared runtime directory /run/portal:

flowchart LR
    subgraph Core["Core Services"]
        PORTAL["portal.service<br/>(Wayfire)"]
        INPUT["portal-input.service<br/>(Keyboard + Fusion)"]
        STREAM["portal-stream.service<br/>(H.265 → RTP)"]
    end
    
    subgraph Management["Window & App Management"]
        WM["portal-wm.service"]
        PCP["portal-pcp.service"]
        LAUNCHER["portal-launcher.service"]
    end
    
    subgraph AI["AI Services"]
        VOICE["portal-voice.service"]
        LLM["portal-llm.service"]
    end
    
    subgraph Infra["Infrastructure"]
        NET["portal-network.service"]
        DBUS["portal-dbus.service"]
        HEALTH["portal-health-monitor"]
        WLO1["wlo1-watchdog"]
    end
    
    PORTAL -->|"PartOf"| STREAM
    PORTAL -->|"BindsTo"| INPUT
    WM -->|"Wayland"| PORTAL
    WM -->|"Unix socket"| PCP
    LAUNCHER --> PCP
    VOICE --> LLM

Sources: docs/x-elite-deployment-state.md#L29-L50, portal/systemd/portal.service#L1-L29, portal/systemd/portal-stream.service#L1-L34

Repository Structure

Section titled “Repository Structure”

The repository is a single Cargo workspace rooted at the project top level, with all Portal crates under portal/, Unity receiver projects under unity/, cross-component documentation under docs/, and deployment automation under scripts/:

portal/
├── spatial/              # Spatial domain model (zones, dimensions, IPC, homography)
├── spatial-plugin/       # Wayfire spatial plugin (Rust cdylib → C++ shim)
├── warp/                 # Spatial warp rendering (homography GLSL shaders)
├── wm/                   # Window manager daemon (toplevel tracking, zone policy)
├── input/                # Keyboard/input (DRM/KMS, fusion engine, gestures)
├── stream/               # RTP protocol types (Rust library)
├── streaming/            # Main streaming pipeline (C/GStreamer/IRIS)
├── voice/                # Voice pipeline (VAD, STT, NLU, TTS, DSP arbitration)
├── llm/                  # On-device LLM daemon (GenieX llama.cpp)
├── npu-runtime/          # Hexagon FastRPC safe Rust wrappers
├── pcp/                  # Portal Capability Protocol (15 sub-crates)
├── context/              # Context engine (event ingestion, template synthesis)
├── launcher/             # Universal launcher (fuzzy search, action registry)
├── shell/                # Shell layer (status bar, notifications)
├── style/                # Design token system
├── common/               # Shared utilities (errors, IPC auth, test harness)
├── compositor/           # Wayfire configs (wayfire.ini, spatial-warp.toml)
├── protocols/            # Wayland protocol XML definitions
├── wayfire-plugins/      # C++ shim plugins for Wayfire (CMake)
├── streaming/            # C streaming daemon source
├── systemd/              # Service unit files
├── network/              # hostapd, dnsmasq configs
└── android/              # Android receiver apps


unity/
├── SpatialPortal_3DoF/   # Unity 3DoF head tracking receiver

Sources: README.md#L94-L127, Cargo.toml#L1-L45

Emulation Layer

Section titled “Emulation Layer”

Portal includes transparent x86/Windows application support via Box64 (v0.4.3, x86/x64 dynamic recompiler on ARM64) and Wine 10.0. The portal-binfmt-fix service ensures Box64 — not QEMU — handles x86 ELF binaries through binfmt_misc, which is critical for Wine’s internal architecture. File watchers (portal-file-watcherd) and a Wine monitor (portal-wine-monitor) automatically detect and register new .exe binaries. The PCP AT-SPI2 integration provides accessibility capability derivation for both native and emulated applications.

Sources: README.md#L31-L32, docs/x-elite-deployment-state.md#L56-L63, portal/docs/architecture.md#L24-L33

Where to Go Next

Section titled “Where to Go Next”

This overview establishes the mental model for the entire Portal platform. The documentation catalog is organized as a progressive deep dive — each section builds on the concepts introduced here:

Getting Started:

Deep Dives — Explore any subsystem that sparks your curiosity, from the Dual-Display Wayland Architecture through the streaming pipeline’s DMA-BUF to H.265 encode path, the voice AI pipeline, and the PCP application framework.