Portal Voice — Speech Interaction Pipeline

vad

Voice Activity Detection — Silero VAD model on QNN DSP (Hexagon). Detects speech segments in audio stream.

stt

Speech-to-Text — Sherpa-onnx Zipformer model on CPU (A76/Oryon cores). Streaming transcription of detected speech.

nlu

Natural Language Understanding — MiniLM sentence similarity classifier. 28 intents, cosine similarity matching, slot extraction.

tts

Text-to-Speech — Elara VITS engine. Encoder on QNN DSP, decoder via onnxruntime-qnn. 22050Hz output, resampled to 48kHz for glasses.

wake_word

Wake word detection — listens for activation phrase before starting full pipeline.

preprocessing

Audio preprocessing — RNNoise denoising, format conversion, sample rate adjustment.

audio_io

Audio I/O — UDP audio reception from glasses (:5601), RTP audio transmission to glasses (:5002).

mic_controller

Microphone controller — manages audio capture, noise gate, automatic gain control.

npu_backend

NPU backend abstraction — QNN model loading, context management, tensor I/O.

npu_scheduler

NPU scheduler — arbitrates DSP access between VAD, TTS encoder, and LLM to prevent context conflicts.

voice_state

Voice state machine — Idle/Listening/Processing/Speaking transitions, mic control coordination.

tts_arbitration

TTS arbitration — prevents overlapping speech, queues responses, handles interruption.

clarification

Clarification handler — asks follow-up questions when NLU confidence is low or slots are missing.

address_detection

Address detection — determines if user is addressing Portal ('Hey Elara') vs. talking to someone else.

error_recovery

Error recovery — handles NPU failures, model reload, graceful degradation.

subprocess_integrity

Subprocess integrity — validates GenieX Python subprocess health, restarts on crash.

telemetry

Telemetry — latency tracking, classification accuracy, RTF (Real-Time Factor) measurement.