Voice Pipeline: VAD, STT (sherpa-onnx), NLU (MiniLM), and TTS (Elara VITS)

The Portal voice pipeline is a real-time, bidirectional speech system that runs as the portal-voiced daemon on the Snapdragon X Elite (Spaceboard) platform. Audio arrives over RTP/UDP from AR glasses, flows through RNNoise denoising, Silero VAD on the Hexagon DSP, and Sherpa-onnx Zipformer transcription on CPU. Transcripts are classified into intents via a MiniLM sentence-similarity model (or rule-based fallback), routed through a response pipeline that may execute system commands, launch applications, or query an on-device LLM daemon, and finally spoken back to the glasses as synthesized speech via the Elara VITS engine. This page traces that end-to-end data path, examines the architecture of each stage, and documents the trait abstractions and scheduling contracts that hold the pipeline together.

Pipeline Architecture at a Glance

The voice subsystem is organized as a stage-sequential pipeline with shared state coordination. Audio frames flow synchronously through preprocessing → VAD → STT in a single-threaded main loop, while TTS generation and audio transmission run on background threads coordinated by the TTS arbitrator. The Voice facade struct owns all subsystem components and exposes a unified API that other Portal services interact with over a Unix-domain IPC socket.

flowchart LR
    subgraph Glasses["AR Glasses (Remote)"]
        MIC["Microphone"]
        SPK["Speaker"]
    end

    subgraph Daemon["portal-voiced (Single-Threaded Main Loop)"]
        UDP["UDP Capture\nRTP/UDP Port 5601"]
        FB["Frame Buffer\n480 samples/frame"]
        DENOISE["RNNoise\nDenoiser"]
        VAD["Silero VAD v5\nQNN NPU"]
        STT["Sherpa-onnx\nZipformer STT\nCPU 4 threads"]
    end

    subgraph Response["Response Path"]
        NLU["NLU Pipeline\nMiniLM Classifier"]
        ACT["ResponseAction\nSpeak / Execute / Launch"]
        TTS["Elara VITS TTS\nQNN Encoder + Decoder"]
    end

    subgraph Back["Return Path"]
        ARB["TTS Arbitrator\nPriority Queue"]
        RTP["RTP Audio Sender\nUDP Port 5002"]
    end

    MIC -->|"audio"| UDP
    UDP --> FB --> DENOISE --> VAD --> STT
    STT -->|"UtteranceComplete"| NLU --> ACT
    ACT --> TTS --> ARB --> RTP
    RTP -->|"audio"| SPK

The main loop reads 16 kHz mono PCM frames from the network, accumulates them into 480-sample (30 ms) units, denoises each frame through RNNoise, runs VAD to gate whether audio is forwarded to STT, and feeds qualifying frames to the streaming Zipformer recognizer. When the recognizer’s endpoint detector fires, the complete transcript is sent through the NLU response pipeline and the resulting speech audio is streamed back to the glasses.

Sources: portal/voice/src/bin/portal-voiced/main.rs#L53-L91, portal/voice/src/voice.rs#L40-L52

Voice State Machine

At the top of the hierarchy sits a validated finite state machine with eight states and an explicit transition table. The machine guards every state change against a VALID_TRANSITIONS array of (from, to) pairs, rejecting invalid transitions with a structured error. The state machine provides two notification channels: a broadcast::Sender<VoiceEvent> for fire-and-forget event delivery, and a watch::Sender<VoiceState> for subscribers that only need the latest value.

stateDiagram-v2
    [*] --> Initializing
    Initializing --> Listening
    Initializing --> Error
    Initializing --> Muted
    Listening --> Processing: speech segment detected
    Listening --> Muted: user mute
    Listening --> AudioExclusive: TTS owns audio
    Processing --> Speaking: response ready
    Processing --> Listening: ambient discarded
    Processing --> AmbientDiscarded: low confidence
    Speaking --> Listening: TTS finished
    Speaking --> Processing: new intent
    Muted --> Listening
    Error --> Initializing: recovery

State	Meaning	Transitions Out
Initializing	Models loading progressively (L0–L5)	Listening, Error, Muted
Listening	Capturing audio, VAD active	Processing, Muted, Error, AudioExclusive
Processing	STT + NLU in progress	Speaking, Listening, AmbientDiscarded, Muted, Error
Speaking	TTS synthesis streaming to glasses	Listening, Processing, Muted, Error
AmbientDiscarded	Utterance was not addressed to Portal	Listening, Muted
Muted	Microphone paused via GPIO	Listening, Error
AudioExclusive	TTS holds exclusive audio access	Listening, Error
Error	Unrecoverable subsystem failure	Initializing, Muted

A force_reset method exists for error recovery scenarios that bypasses transition validation, intended for administrative overrides only.

Sources: portal/voice/src/voice_state/machine.rs#L23-L56, portal/voice/src/types.rs#L13-L22

Preprocessing: RNNoise Denoising, AGC, and Echo Cancellation

Before audio reaches VAD, three preprocessing stages clean the signal. The denoiser wraps RNNoise via a native FFI binding (rnnoise-sys), processing exactly 480-sample frames through a trained recurrent neural network. The NoiseDenoiser trait exposes process(&self, input, output) with a configurable enable flag — when disabled, it degrades to a zero-cost memory copy. The automatic gain control stage maintains consistent audio levels by tracking peak amplitude in dBFS and applying a smooth gain envelope clamped to [0.1, 10.0]. The acoustic echo canceler uses spectral subtraction, fed by the TTS playback audio as a reference signal; when no TTS is active, process() is a zero-overhead memcpy.

The daemon’s main loop invokes only the denoiser on each frame before passing it to VAD. The AGC and echo canceler traits are available for integration but are not wired into the current portal-voiced main loop, which relies on RNNoise alone for noise removal.

Sources: portal/voice/src/preprocessing/mod.rs#L21-L87, portal/voice/src/bin/portal-voiced/main.rs#L322-L326

Voice Activity Detection (VAD)

VAD is the gatekeeper that prevents wasted STT cycles on silence. The pipeline supports two detector backends behind a common VadDetector trait: a CPU-based energy detector (SileroVadDetector) and an NPU-accelerated RKNN/QNN detector (RknnVadDetector). Both share identical segment-tracking logic through the VadSegmentTracker struct, which consolidates the core constants and the update_segment_state algorithm.

VAD Constants and Segment Tracking

Constant	Value	Purpose
FRAME_SIZE	480 samples	Frame size at 16 kHz = 30 ms
SAMPLE_RATE	16,000 Hz	Fixed capture rate
MIN_SEGMENT_FRAMES	9 frames (~270 ms)	Minimum segment to emit; shorter = discarded
MIN_SILENCE_FRAMES	10 frames (~300 ms)	Consecutive silence to close a segment
PADDING_SAMPLES	2,400 samples (150 ms)	Lead-in padding before segment start

The segment tracker maintains a state machine: when speech is detected, silence_frames resets to zero and if not already in a segment, the start is recorded with padding. When silence exceeds MIN_SILENCE_FRAMES, the segment closes; if it contains at least MIN_SEGMENT_FRAMES frames of speech, it becomes a completed VadSegment; otherwise it increments the discarded_segments counter.

CPU Energy-Based Detector (SileroVadDetector)

When no model file is present, the CPU detector falls back to an RMS-energy heuristic. It computes the root-mean-square of the 480-sample frame, then maps it through a sigmoid function with midpoint 0.05 and temperature 0.02 to produce a probability in [0, 1]. The binary decision uses probability > threshold (default 0.5). This exercises the identical trait API and segment-tracking logic as the neural model path, allowing the pipeline to function without a model artifact at the cost of accuracy.

NPU Detector (RknnVadDetector)

The NPU detector loads a Silero VAD v4 .rknn/.so model with a stateful LSTM architecture: 3 inputs (audio x[1,512], hidden h[2,1,64], cell c[2,1,64]) and 3 outputs (probability, next hidden, next cell). Externally the detector maintains the 480-sample frame contract; internally it zero-pads to 512 for the model input. LSTM hidden and cell states persist across frames and reset to zero on reset(). When an NPU scheduler is available, the detector optionally coordinates through NpuScheduler for workload tracking.

Sources: portal/voice/src/vad/mod.rs#L65-L148, portal/voice/src/vad/detector.rs#L32-L123, portal/voice/src/vad/npu_vad.rs#L1-L39, portal/voice/src/vad/adaptive.rs#L9-L74

Adaptive Threshold

A PortalAdaptiveVadThreshold implementation drifts the speech/silence cutoff based on running noise floor estimates. It uses an exponential moving average (α = 0.1) over non-speech frames’ probabilities, recalibrating every 100 frames. In noisy environments (noise estimate > 0.3), the threshold moves toward 0.8 (requiring stronger evidence for speech); in quiet environments it drifts back toward the base. Adjustments are rate-limited to 30% of the total range [0.3, 0.8] per step, preventing oscillation.

Sources: portal/voice/src/vad/adaptive.rs#L9-L74

Speech-to-Text: Sherpa-onnx Zipformer

The STT engine wraps the sherpa-onnx streaming Zipformer model through libloading dynamic FFI. Rather than linking against sherpa-onnx at compile time, the engine dlopens libsherpa_wrapper.so at runtime, resolves eleven C function symbols, and caches them as typed function pointers. This design allows the model and runtime library to be deployed independently of the Rust binary.

SherpaZipformerEngine Architecture

flowchart TB
    subgraph Load["load()"]
        DL["dlopen libsherpa_wrapper.so"]
        SYM["Resolve 11 C symbols\nsherpa_create_recognizer\nsherpa_create_stream\nsherpa_accept_waveform\n..."]
        REC["Create recognizer\ntokens + encoder + decoder + joiner"]
        STREAM["Create stream\nfrom recognizer"]
    end

    subgraph Feed["feed_audio(samples)"]
        ACCEPT["sherpa_accept_waveform\n16kHz, f32 samples"]
        DECODE["while is_ready:\n  sherpa_decode(recognizer, stream)"]
        RESULT["sherpa_get_result\n→ partial text"]
        ENDPOINT["sherpa_is_endpoint?\n→ UtteranceComplete event"]
    end

    DL --> SYM --> REC --> STREAM
    ACCEPT --> DECODE --> RESULT --> ENDPOINT

The feed_audio method accepts raw 16 kHz f32 samples, passes them to sherpa_accept_waveform, then runs the greedy decoder loop (while is_ready: decode). After decoding, it retrieves the current best transcription. If the text has changed since the last frame, it emits a StreamingSttEvent::Partial event (for visual display). When the endpoint detector fires — indicating the user has paused — the engine emits StreamingSttEvent::UtteranceComplete with the final text and resets the stream for the next utterance.

The SherpaConfig struct specifies paths to the Zipformer’s four model components:

Component	Purpose	Default Path
tokens_path	BPE token vocabulary	tokens.txt
encoder_path	Streaming encoder ONNX	encoder.onnx
decoder_path	Decoder ONNX	decoder.onnx
joiner_path	Joiner ONNX	joiner.onnx

The engine implements Send but intentionally withholds Sync, because the underlying sherpa-onnx C++ library does not document thread-safety for concurrent calls on the same recognizer handle. The daemon owns the engine as &mut self in a single tokio task, satisfying this constraint.

Mel Filterbank Feature Extraction

A companion FeatureExtractor module converts raw PCM to 80-dimensional log-mel features matching sherpa-onnx/kaldi-native-fbank parameters: frame length 400 samples (25 ms), frame shift 160 samples (10 ms), FFT size 512, 80 mel bins, frequency range 20 Hz–7600 Hz, pre-emphasis 0.97, Povey window. The module implements a recursive radix-2 FFT and a triangular mel filterbank, producing features for the streaming encoder in fixed-size chunks.

Sources: portal/voice/src/stt/sherpa_engine.rs#L14-L211, portal/voice/src/stt/sherpa_engine.rs#L217-L278, portal/voice/src/stt/feature_extractor.rs#L1-L22

Natural Language Understanding: Intent Classification Pipeline

NLU transforms a transcript string into a ResponseAction — either speaking a response, executing a system command, or launching an application. The ResponsePipeline orchestrates a four-step decision cascade, designed so that the system never echoes the user’s input and always produces a meaningful response even when all ML backends are unavailable.

Response Pipeline Decision Cascade

flowchart TD
    IN["Transcript text"]
    CLS["Classify intent\n(NluClassifier trait)"]
    CHECK{"High-confidence\nsystem intent?\n(≥ 0.6)"}
    SYSCMD["SystemCommand::from_intent\n→ ExecuteAndSpeak"]
    BLOCK{"System domain but\nunhandled action?"}
    REFUSE["Refuse: 'I can't do that yet.'"]
    APPTGT{"App launch target\ndetected?"}
    LAUNCH["LaunchApp\n→ gtk-launch"]
    LLM{"LLM daemon\navailable?"}
    LLMRESP["Speak: LLM response\n(stripped of think tags)"]
    CANNED["Speak: canned fallback\n(never echoes input)"]

    IN --> CLS --> CHECK
    CHECK -->|"yes"| SYSCMD
    SYSCMD -->|"recognized"| OUT["ResponseAction emitted"]
    SYSCMD -->|"not recognized"| BLOCK
    BLOCK -->|"yes"| REFUSE
    CHECK -->|"no"| APPTGT
    APPTGT -->|"yes"| LAUNCH
    APPTGT -->|"no"| LLM
    LLM -->|"yes, non-empty"| LLMRESP
    LLM -->|"no / failed"| CANNED
    REFUSE --> OUT
    LAUNCH --> OUT
    LLMRESP --> OUT
    CANNED --> OUT

Rule-Based Classifier (Default)

The RuleBasedClassifier is a deterministic, taxonomy-driven classifier that pattern-matches transcripts against a JSON-structured IntentTaxonomy. The taxonomy (version 1.0.0) defines 28 intent domains including Application, Window, System, Navigation, Media, Communication, Time, Weather, and Settings. Each domain contains typed intents with required_slots, optional_slots, and examples. The classifier extracts slots (target, direction, amount) from the transcript text using keyword patterns.

MiniLM Sentence-Similarity Classifier (Feature-Gated)

When the nlu-minilm feature is enabled and model files are present, the pipeline transparently replaces the rule-based classifier with a MiniLM sentence-embedding classifier. This classifier loads a 384-dimensional model.onnx (a BERT-family model), a HuggingFace tokenizer.json, and pre-computed intent_embeddings.npy containing the centroid embedding for each intent class.

Classification proceeds in three steps: (1) tokenize the transcript with truncation to 64 tokens and fixed padding, (2) run ONNX inference to produce last_hidden_state, then mean-pool over non-padding tokens with L2 normalization, and (3) compute cosine similarity between the query embedding and all intent centroids, returning the top-3 candidates.

Parameter	Value	Purpose
MAX_SEQ_LEN	64	Tokenizer truncation length
EMBEDDING_DIM	384	MiniLM hidden dimension
Optimization	GraphOptimizationLevel::Level3	Maximum ONNX graph optimization
Embedding file	intent_embeddings.npy	NumPy v1/v2 float32, C-contiguous

The MiniLM classifier applies a verb-based override: if the transcript starts with “close”, “quit”, “exit”, or “kill” and the top candidate is an App/open intent, it overrides to App/close. Slot extraction (extract_slots_from_text) runs on the final classified intent to populate target, direction, and amount fields.

Sources: portal/voice/src/nlu/pipeline.rs#L43-L176, portal/voice/src/nlu/classifier/mod.rs#L41-L58, portal/voice/src/nlu/minilm_classifier/classifier.rs#L22-L272, portal/voice/src/nlu/intents.rs#L12-L69

System Command Execution

High-confidence system intents (≥ 0.6) are routed to SystemCommand::from_intent, which maps domain/action/direction slots to typed commands. The SystemCommand enum supports VolumeUp(u8), VolumeDown(u8), MuteToggle, BrightnessUp(u8), and BrightnessDown(u8). Each command executes via Command::new(binary).args([...]) — never shell interpolation — using amixer for volume and brightnessctl for brightness. Amounts default to 5 and are clamped to [1, 50].

A security-critical branch handles the case where the classifier detects a System-domain intent at high confidence but SystemCommand::from_intent returns None (e.g., shutdown, restart, lock, screenshot). These intents are blocked from the LLM to prevent arbitrary text generation, returning a canned refusal (“I can’t do that yet.”) until PCP routing is implemented.

Sources: portal/voice/src/nlu/system_commands.rs#L16-L168, portal/voice/src/nlu/pipeline.rs#L118-L134

Text-to-Speech: Elara VITS

The Elara VITS engine is a split-architecture neural TTS that runs the encoder and decoder on the Hexagon DSP via QNN contexts, while the stochastic duration predictor and length regulation execute on CPU. The engine implements the TtsEngine trait, which abstracts away the underlying runtime and supports voice cloning, emotion conditioning, and streaming chunk delivery.

Elara VITS Synthesis Pipeline

flowchart LR
    TEXT["Input Text"] --> PHON["Phonemization\npiper-plus-g2p"]
    PHON --> IDS["Phoneme IDs\n[BOS, 0, ...tokens..., EOS, 0, 0, ...]\nMax 256 tokens"]
    IDS --> ENC["Encoder (NPU/QNN)\nEmbedding + FFT attention + projection"]
    ENC --> MP["m_p, logs_p, x_mask"]
    MP --> MID["Middle Process (CPU)\nDuration prediction + length regulation\n+ noise injection"]
    MID --> ZP["z_p [INTER_CHANNELS × actual_mel]\ny_mask [actual_mel]"]
    ZP --> DEC["Decoder (NPU/QNN)\nflow.reverse + HiFi-GAN Generator"]
    DEC --> WAVE["Waveform f32 samples\n22.05 kHz"]

The generate method orchestrates four sequential stages:

Step 1 — Phonemization: Text is converted to phoneme IDs using Piper’s grapheme-to-phoneme engine (piper-plus-g2p). The phoneme sequence is wrapped with BOS (1) and EOS (2) tokens, then zero-padded to MAX_SEQ (256) elements. If phonemization fails, a character-level fallback maps each lowercase ASCII character to its codepoint.

Step 2 — Encoder (NPU): Phoneme IDs (as INT32) and lengths tensor are fed to the QNN encoder context, which returns x, m_p, logs_p, and x_mask — the prior distribution parameters and attention mask.

Step 3 — Middle Process (CPU): The stochastic duration predictor runs in a subprocess (DpServer) because it requires operations unsupported by NPU (RandomNormalLike, ScatterND, CumSum). It predicts per-token durations scaled by length_scale, clamped to [1, 20] frames. The duration predictor uses Box-Muller transform for Gaussian noise sampling. If the DP server is unavailable, an energy heuristic computes durations from m_p magnitudes. Length regulation then expands the prior distribution along the time axis, injecting Gaussian noise scaled by noise_scale * exp(logs_p).

Step 4 — Decoder (NPU): The z_p latent and y_mask are fed to the QNN decoder context, which runs flow.reverse (4 residual coupling layers) followed by the HiFi-GAN generator, producing waveform samples at 22.05 kHz. The output is trimmed to actual_mel × UPSAMPLE_RATIO samples.

Elara VITS Constants

Constant	Value	Meaning
ELARA_SAMPLE_RATE	22,050 Hz	Output audio sample rate
MAX_SEQ	256	Maximum phoneme sequence length
MAX_MEL	2,048	Maximum mel frames
INTER_CHANNELS	192	Hidden dimension
UPSAMPLE_RATIO	256	Mel-to-audio upsampling factor
N_VOCAB	256	Phoneme vocabulary size
noise_scale	0.667	Default posterior noise injection
length_scale	1.0	Duration scaling factor
noise_scale_w	0.8	Duration predictor noise

Dynamic-Shape Optimization

The decoder uses a dynamic-shape strategy: rather than always processing the full MAX_MEL (2,048) mel frames, the middle process computes actual_mel from the predicted durations (sum clamped to 2,048), and the decoder only processes that many frames. This reduces both NPU compute time and output trimming, significantly lowering latency for short utterances.

Sources: portal/voice/src/tts/elara_engine/mod.rs#L1-L96, portal/voice/src/tts/elara_engine/tts_impl.rs#L13-L123, portal/voice/src/tts/elara_engine/encoder.rs#L14-L102, portal/voice/src/tts/elara_engine/middle.rs#L22-L117, portal/voice/src/tts/elara_engine/decoder.rs#L15-L158, portal/voice/src/tts/elara_engine/phonemize.rs#L7-L62

TTS Arbitration and Priority Scheduling

The TTS arbitrator manages concurrent speech requests using a priority-preemptive queue with RAII-based slot management. When a component requests speech, the arbitrator grants a SpeechToken that holds a closure callback; dropping the token (or calling release()) automatically frees the speech slot, promoting any queued request.

Priority	Value	Use Case
Ambient	0	Weather notifications, background info
ProactiveNotification	1	Context crate proactive suggestions
UserIntent	2	Direct response to a voice command
SystemCritical	3	Battery low, danger alerts

The SpeechToken uses an AtomicBool guard to prevent double-release, ensuring the callback fires exactly once regardless of whether release is manual or via Drop. The default queue capacity is 5 slots, with a maximum of 8 arbitrator slots configured in the Voice facade.

The Voice::speak_streaming method integrates the arbitrator with the TTS engine: it requests a speech slot, then spawns a background thread that acquires the engine mutex, calls generate_streaming, and delivers AudioChunks through a bounded crossbeam channel. The caller receives the channel’s Receiver end and can stream audio chunks to the network without blocking.

Sources: portal/voice/src/tts_arbitration/mod.rs#L14-L58, portal/voice/src/tts_arbitration/speech_token.rs#L41-L103, portal/voice/src/voice.rs#L275-L314

Daemon: End-to-End Data Flow

The portal-voiced binary orchestrates the entire pipeline in a single-threaded main loop with background threads for IPC, TTS generation, and RTP audio transmission. The daemon’s lifecycle is controlled by a static RUNNING: AtomicBool flag set by the signal handler (SIGINT/SIGTERM).

Initialization Sequence

flowchart TD
    CFG["Load VoiceConfig::default()\nOverride from env vars"]
    VALID["config.validate()"]
    UDP["Open UDP capture\n0.0.0.0:5601"]
    DENOISE["RnnoiseNoiseDenoiser::new()"]
    SCHED["PortalNpuScheduler::new()\n(3 cores detected)"]
    VAD["RknnVadDetector::load_with_scheduler()"]
    STT["SherpaZipformerEngine::load()\n4 threads, 16kHz"]
    VOICE["Voice::new(config)\n→ Elara VITS engine"]
    IPC["Spawn IPC server thread\nUnix socket"]
    NLU["Build ResponsePipeline\n+ MiniLM if available"]
    TTS_S["Create TtsSender\n(glasses_ip, port 5002)"]
    READY["Daemon ready"]

    CFG --> VALID --> UDP --> DENOISE --> SCHED --> VAD --> STT --> VOICE --> IPC --> NLU --> TTS_S --> READY

Main Loop

Each iteration of the main loop executes the following sequence:

1. Read audio frame from the UDP capture backend (100 ms timeout)
2. Buffer into 480-sample frames via PortalVadFrameBuffer::feed
3. For each complete frame: denoise through RNNoise
4. VAD process: if tts_active is true (TTS is speaking back), skip STT feed to prevent echo
5. STT feed: pass denoised samples to sherpa_engine.feed_audio()
6. Handle events: for UtteranceComplete, trim the transcript and route to the TTS sender

The tts_active flag is an Arc<AtomicBool> shared between the main loop and the RTP sender thread. When TTS audio is being streamed to the glasses, STT is suppressed to prevent the TTS output from being captured and re-transcribed — a software echo gate complementing the acoustic echo canceler trait.

TtsSender: NLU → TTS → RTP Response Path

When a transcript arrives, the TtsSender executes the response pipeline: pipeline.generate(text) returns a ResponseAction, which may Speak (LLM or canned response), ExecuteAndSpeak (system command like amixer set Master 5%+), or LaunchApp (via gtk-launch). The resulting text is synthesized via voice.speak_streaming() with UserIntent priority and Reactive source, producing audio chunks delivered through a crossbeam channel. An RtpAudioSender thread then packetizes and transmits the audio over UDP to the glasses’ IP address on port 5002, setting tts_active = true during transmission.

Sources: portal/voice/src/bin/portal-voiced/main.rs#L53-L91, portal/voice/src/bin/portal-voiced/main.rs#L287-L374, portal/voice/src/bin/portal-voiced/tts_sender.rs#L44-L118

IPC Server: Cross-Service Speech Requests

Other Portal services (context engine, launcher, etc.) can request speech through a Unix-domain socket at /run/portal/voice.sock. The IPC server accepts up to 4 concurrent connections, each handling a single request-response cycle using PostCard serialization (not JSON — PostCard’s compact binary format and rejection of internally-tagged enums informed the message type design).

Request Variant	Response	Description
Speak { text, priority, source, tone }	Ok / Error	Request TTS speech; spawns RTP sender thread
QueryAvailability	Availability { busy, queue_depth }	Check if voice subsystem is speaking
CancelSpeech { text }	Ok	Cancel all ongoing speech
Ping	Pong	Connection health check

The VoiceIpcHeader is a 20-byte #[repr(C)] struct containing message type, payload size, sequence ID, and Unix timestamp, enabling a lightweight framing protocol compatible with both Rust and future C consumers.

Sources: portal/voice/src/bin/portal-voiced/ipc_server.rs#L20-L167, portal/voice/src/ipc/types.rs#L47-L134

Progressive Model Loading (L0–L5)

Models load progressively through six levels, allowing the system to become partially operational before all models are resident. Each level emits a VoiceEvent::LoadingLevelReached broadcast, enabling UI components to display loading progress.

Level	Models	Operational Capability
L0	VAD model	Audio gating only
L1	+ STT model	Transcription available
L2	+ Address detection	Listening-capable (knows when addressed)
L3	+ NLU classifier	Intent classification
L4	+ TTS model (background)	Full speech output
L5	All models verified	Fully operational

The ProgressiveLoader tracks individual model load status in a HashMap<String, bool> and exposes report_level_loaded / report_level_failed methods. Failures do not roll back to a lower level — the system continues operating at the highest level achieved.

Sources: portal/voice/src/model_manager/progressive.rs#L18-L99

Feature Flags and Build Configuration

The voice crate uses an extensive feature-gate system to support multiple hardware targets. The production Spaceboard build (Snapdragon X Elite) uses QNN features, while legacy Orange Pi builds use RKNN.

Feature	Dependencies	Purpose
qnn-vad	qnn-sys	VAD on Hexagon DSP via QNN
qnn-stt	qnn-sys, libloading	STT with QNN encoder
qnn-tts	qnn-sys, piper-plus-g2p, rand	Elara VITS via QNN
qnn-scheduler	qnn-sys	NPU core arbitration
cpu-vad	ort	CPU-based VAD via ONNX Runtime
nlu-minilm	ort, tokenizers	MiniLM intent classifier
network-audio	opus, portal-stream	Bidirectional UDP audio + Opus codec
udp-hmac (default)	rand	ECDH key exchange + HMAC-SHA256 audio auth

The HP production build command is: cargo build --features 'qnn-tts,qnn-vad,npu-stt,network-audio,nlu-minilm'

Sources: portal/voice/Cargo.toml#L72-L103

NPU Workload Assignment

The NPU scheduler assigns workloads to Hexagon DSP cores based on a static partitioning scheme. This is detailed fully in NPU Scheduling: Hexagon DSP Arbitration for VAD, TTS, and LLM Workloads, but the voice pipeline’s core assignments are:

Core	Workloads	Reservation Type
Core 0	STT Encoder, Decoder, Joiner	Permanent
Core 1	TTS Encoder, Decoder	Permanent (shared queue)
Core 2	LLM Inference	Permanent

VAD runs on CPU (energy heuristic) or as a lightweight NPU call that does not require permanent core allocation. The scheduler tracks per-workload inference counts and timing, reporting through NpuCoreUtilization structs with inference_count and total_inference_us fields.

Sources: portal/voice/src/npu_scheduler/types.rs#L9-L24, portal/voice/src/npu_scheduler/types.rs#L94-L147

What’s Next

The voice pipeline does not operate in isolation — its NPU workload scheduling, LLM integration, and NPU runtime wrappers form a tightly coupled on-device AI stack:

• NPU Scheduling: Hexagon DSP Arbitration for VAD, TTS, and LLM Workloads — Deep dive into the 3-core Hexagon allocation scheme, Core 1 shared queue, and starvation prevention between STT and TTS workloads.
• On-Device LLM Daemon: GenieX llama.cpp on the Hexagon DSP — How the LLM daemon at /run/portal/llm.sock generates conversational responses that the voice NLU pipeline consumes as ResponseAction::Speak.
• NPU Runtime: Safe Rust Wrappers for Qualcomm Hexagon FastRPC — The qnn-sys crate and QnnContext abstraction that underlies the Elara VITS encoder and decoder.
• Context Engine: Event Ingestion, Decision Making, and Template Synthesis — How the context engine consumes VoiceEvent broadcasts to drive proactive speech and spatial gaze targeting.