NPU Scheduling: Hexagon DSP Arbitration for VAD, TTS, and LLM Workloads

The Portal voice stack runs three concurrent neural inference workloads—voice activity detection (VAD), text-to-speech (TTS), and large language model (LLM) generation—on a single Qualcomm Hexagon DSP. This page documents the scheduling architecture that arbitrates DSP time across these workloads, the priority and preemption policies that prevent starvation, the health-monitoring state machine that drives graceful degradation, and the FastRPC transport layer that delivers tensor data to the DSP cores. Understanding this system is essential for anyone debugging voice latency, optimizing inference throughput, or extending the NPU workload set.

Core Partitioning Model: Workload-to-Core Affinity

The scheduler operates on a static partitioning principle: each NPU-bound workload is assigned to a specific Hexagon core index at allocation time, and that assignment is permanent for the process lifetime. The NpuWorkload enum encodes seven distinct workload types, but VAD is deliberately excluded from the NPU—it runs on the CPU via ONNX Runtime or a dedicated RNNN backend. The remaining six workloads map to three logical cores.

graph TB
    subgraph "Portal NPU Scheduler — Core Affinity Map"
        subgraph "Core 0 — STT Permanent"
            STT_E[STT Encoder<br/>Zipformer]
            STT_D[STT Decoder<br/>Zipformer]
            STT_J[STT Joiner<br/>Zipformer]
        end
        subgraph "Core 1 — TTS Permanent"
            TTS_E[TTS Encoder<br/>Elara VITS]
            TTS_D[TTS Decoder<br/>Elara VITS]
        end
        subgraph "Core 2 — LLM Permanent"
            LLM[LLM<br/>GenieX / llama.cpp]
        end
        subgraph "CPU — No NPU"
            VAD[VAD<br/>Silero v4 — ONNX/RKNN]
        end
    end
    VAD -.->|"Rejected by scheduler"| CORE0_REF["allocate_core() → Error"]
    style VAD fill:#e8f5e9,stroke:#2e7d32
    style CORE0_REF fill:#ffebee,stroke:#c62828

The affinity table below shows how each workload maps to its core, the priority level assigned by convention, and the allocation semantics:

Workload	Core	Priority	Allocation	Notes
VAD	CPU	—	Rejected	`NpuCoreAllocationFailed` error; “VAD runs on CPU, not NPU”
STT Encoder	Core 0	Medium	Permanent	User-initiated speech recognition
STT Decoder	Core 0	Medium	Permanent	Shares Core 0 with encoder/joiner
STT Joiner	Core 0	Medium	Permanent	Joins encoder + decoder outputs
TTS Encoder	Core 1	Low	Permanent	System-initiated response synthesis
TTS Decoder	Core 1	Low	Permanent	Shares Core 1 with encoder
LLM	Core 2	Low	Permanent	GenieX subprocess on Hexagon DSP

Sources: portal/voice/src/npu_scheduler/types.rs#L7-L37, portal/voice/src/npu_scheduler/scheduler/allocation.rs#L12-L34

Scheduler Trait and Job Abstraction

The NpuScheduler trait defines the scheduling contract. Every implementation must provide eight methods covering core lifecycle management, inference dispatch, utilization reporting, and model registry operations. The trait is Send + Sync and uses oneshot::Receiver for asynchronous shared submissions, allowing callers to await inference results without blocking the scheduling thread.

pub trait NpuScheduler: Send + Sync + std::fmt::Debug {
    fn is_available(&self) -> bool;
    fn allocate_core(&self, core: NpuCore, workload: NpuWorkload, priority: NpuPriority)
        -> Result<NpuCoreReservation>;
    fn release_core(&self, reservation: NpuCoreReservation);
    fn submit_inference(&self, core: NpuCore, job: Box<dyn NpuJob + Send>) -> Result<NpuJobResult>;
    fn shared_submit(&self, workload: NpuWorkload, priority: NpuPriority,
        job: Box<dyn NpuJob + Send>) -> oneshot::Receiver<Result<NpuJobResult>>;
    fn utilization(&self) -> Vec<NpuCoreUtilization>;
    fn queue_depth(&self) -> usize;
    fn register_model(&self, core: NpuCore, workload: NpuWorkload, model_path: String) -> Result<()>;
    fn unregister_model(&self, core: NpuCore, workload: NpuWorkload);
    fn record_inference_for(&self, workload: NpuWorkload, inference_us: u64, success: bool);
    fn registered_models(&self) -> Vec<RegisteredModel>;
}

The NpuJob trait is the unit of inference work. Each job carries its workload() identity and an execute() method that returns timing and output tensors. Jobs are boxed and submitted through the scheduler, which instruments every execution with microsecond-precision timing and feeds results back into utilization statistics and the health monitor.

Sources: portal/voice/src/npu_scheduler/scheduler/scheduler_trait.rs#L11-L61

PortalNpuScheduler: Multi-Core Implementation

PortalNpuScheduler is the production implementation. Its internal state comprises four fields: an atomic boolean vector tracking per-core reservation state, an availability flag set by hardware detection, a RwLock<Vec<NpuCoreUtilization>> for statistics, a Mutex<NpuMonitor> for health tracking, and a RwLock<HashMap<(NpuCore, NpuWorkload), RegisteredModel>> for the model registry.

NPU Detection

At construction, detect_npu() probes the hardware on aarch64 targets. When PORTAL_NPU_CORE_COUNT=1 (single-DSP Hexagon mode), it checks for /dev/fastrpc-adsp or /dev/dma_heap/fastrpc_mem. In multi-core mode, it probes /dev/dri/renderD128 or the PORTAL_NPU_CLASS_PATH sysfs directory (default: /sys/class/npu). On non-aarch64 targets, detection always returns false, forcing CPU fallback paths.

flowchart TD
    A[PortalNpuScheduler::new] --> B{target_arch = aarch64?}
    B -->|No| C[available = false<br/>Monitor = Fallback]
    B -->|Yes| D{PORTAL_NPU_CORE_COUNT = 1?}
    D -->|Yes| E{/dev/fastrpc-adsp<br/>or /dev/dma_heap/fastrpc_mem<br/>exists?}
    D -->|No| F{/dev/dri/renderD128<br/>or /sys/class/npu exists?}
    E -->|Yes| G[available = true<br/>Monitor = Healthy]
    E -->|No| H[available = false<br/>Monitor = Fallback]
    F -->|Yes| G
    F -->|No| H
    C --> I[Initialize core_reserved<br/>Vec AtomicBool]
    G --> I
    H --> I
    I --> J[Allocate NpuCoreUtilization<br/>per core index]

Allocation Flow

The allocate_core method branches on single-core versus multi-core mode. In single-core mode (PORTAL_NPU_CORE_COUNT=1), all workloads collapse to Core 0 with a permanent reservation—no affinity check occurs. In multi-core mode, the allocate_multi_core function enforces strict affinity: each workload variant has a predetermined expected core, and any mismatch returns a descriptive NpuCoreAllocationFailed error. The reserve_permanent helper performs an atomic swap(true, SeqCst) on the core’s reservation flag—if the flag was already set (indicating a prior permanent reservation), the allocation fails.

The release_core method is intentionally a no-op. The comment in the NpuCoreReservation documentation is explicit: dropping the reservation handle does not release the core, because STT, TTS, and LLM reservations are permanent for the process lifetime. This design eliminates the risk of core thrashing or accidental deallocation under concurrent access.

Sources: portal/voice/src/npu_scheduler/scheduler/portal_scheduler.rs#L43-L97, portal/voice/src/npu_scheduler/scheduler/allocation.rs#L36-L68

Inference Submission

Two submission paths exist: direct submit_inference and shared shared_submit. The direct path executes the job synchronously, measures elapsed time with Instant, and routes the result through the health monitor—record_success() on success, record_failure() on error. The shared path wraps the result in a oneshot channel, routing to Core 0 in single-core mode or Core 1 in multi-core mode, providing an async-compatible interface for callers that need to await results without blocking the scheduler thread.

Both paths guard against unavailability: if self.available is false, they return NpuNotAvailable immediately without touching the job. This ensures that CPU fallback paths operate independently of the NPU scheduler—VAD on ONNX Runtime never requires the scheduler to be present.

Sources: portal/voice/src/npu_scheduler/scheduler/portal_scheduler.rs#L184-L226

Model Registry and Memory Budget

The scheduler maintains a model registry keyed by (NpuCore, NpuWorkload) tuples. Registration deduplicates by key—attempting to register the same workload on the same core twice returns an error. Each RegisteredModel entry captures the filesystem path, the file size (queried via std::fs::metadata at registration time), load timestamp, cumulative inference count, total inference microseconds, and failed inference count.

The record_inference_for method updates per-model statistics and triggers a 100 MB memory budget warning when the total registered model sizes across all cores exceed that threshold. This acts as an early-warning system: on the Hexagon DSP, model context binaries (QNN .so archives) consume DMA-BUF memory that competes with tensor I/O buffers. The largest legitimate model in the Portal stack is the QNN context binary at approximately 222 MB, but cumulative registration across STT + TTS + LLM can exceed 100 MB of combined model memory, prompting the warning.

Sources: portal/voice/src/npu_scheduler/scheduler/portal_scheduler.rs#L236-L292, portal/voice/src/npu_scheduler/types.rs#L149-L182

Health Monitoring and Graceful Degradation

The NpuMonitor implements a three-state degradation automaton driven by consecutive failure counts. This is separate from the core allocation logic—it operates on the inference results flowing through submit_inference, tracking whether the DSP hardware is responsive.

stateDiagram-v2
    [*] --> Healthy: new() / detect_npu = true
    Healthy --> Degraded: 3 consecutive failures
    Degraded --> Fallback: 10 consecutive failures<br/>(cumulative from Healthy)
    Degraded --> Healthy: record_success()<br/>+ 60s since last failure
    Fallback --> Healthy: set_status(Healthy)<br/>(manual or restart)
    Healthy --> Healthy: record_success()

Threshold	Value	Effect
`DEGRADE_THRESHOLD`	3 consecutive failures	Status transitions to `Degraded`
`FALLBACK_THRESHOLD`	10 consecutive failures	Status transitions to `Fallback`
`RECOVERY_WINDOW_SECS`	60 seconds	Auto-recovery from `Degraded` to `Healthy` if no new failures

The recovery logic requires both a successful inference and 60 seconds of elapsed time since the last failure. This dual-condition prevents flapping: a single success after a burst of failures does not immediately clear the degraded state. The monitor exposes consecutive_failures() for introspection, and set_status() allows startup-time force-set when NPU detection itself fails (e.g., setting Fallback when detect_npu() returns false).

Sources: portal/voice/src/npu_scheduler/monitor.rs#L18-L94

Core-1 Shared Queue: Priority Preemption and Starvation Protection

Although STT and TTS now have dedicated cores (Core 0 and Core 1 respectively), the Core1Scheduler retains a sophisticated priority-based job queue reserved for future use. This module documents the arbitration policy that would apply if STT and TTS shared a single Hexagon core—a scenario that remains relevant for single-DSP deployments where PORTAL_NPU_CORE_COUNT=1.

Preemption Policy

When a new job arrives and a TTS job is currently running, the scheduler evaluates whether to preempt. The rule: if the incoming priority is High (above Medium), or if the incoming priority is Medium and the running job is a TTS workload, the running job’s preempt flag is set. This flag is an Arc<AtomicBool> that the running job’s execute() method can poll cooperatively, allowing it to yield early rather than being forcibly interrupted.

Starvation Protection

Continuous STT preemption would starve TTS indefinitely. The Core1Scheduler tracks STT dominance duration: when STT workloads enqueue consecutively, a timer starts. If that timer exceeds STARVATION_THRESHOLD_SECS (30 seconds), the next dequeue operation force-prioritizes a TTS decoder job, promoting it to High priority and clearing the dominance timer. This guarantees that TTS response synthesis cannot be blocked for more than 30 seconds by continuous speech recognition activity.

The constant CORE1_QUEUE_CAPACITY (16 jobs) bounds the pending queue. When full, the NpuQueueFull error prevents unbounded memory growth under burst load.

Sources: portal/voice/src/npu_scheduler/core1_queue.rs#L87-L131, portal/voice/src/npu_scheduler/core1_queue.rs#L188-L236, portal/voice/src/npu_scheduler/types.rs#L143-L147

TTS Arbitration Layer: Speech Priority Scheduling

Separate from NPU core scheduling, the PortalTtsArbitrator governs which speech request gets to use the TTS engine at any given time. This is a request-level priority queue with preemption semantics, addressing the scenario where multiple system components (context engine notifications, user intent responses, critical alerts) compete for speech output simultaneously.

Priority Hierarchy

The SpeechPriority enum defines four ordered levels:

Priority	Value	Typical Source	Preemption Behavior
`Ambient`	0	Weather, ambient notifications	Lowest; never preempts
`ProactiveNotification`	1	Context engine suggestions	Preempts Ambient only
`UserIntent`	2	Reactive responses to voice commands	Preempts Ambient + Proactive
`SystemCritical`	3	Battery low, danger alerts	Always preempts (unconditional)

The preemption logic in should_preempt encodes three rules: SystemCritical always wins, a ProactiveNotification can be preempted by a Reactive UserIntent, and in all other cases the numerically higher priority wins. When the active slot is preempted, its AtomicBool occupancy flag is set to false, signaling the TTS engine to abort generation. The token’s Drop implementation prevents double-release through an atomic guard.

The arbitrator also implements queue pressure relief: when the pending queue reaches capacity (default 5 slots), the lowest-priority entry is evicted, and the incoming request is dropped with SpeechQueueDropped. This bounds memory while preserving the most important pending speech.

Barge-In Support

The barge_in method provides a one-shot interrupt: if the current speech slot has priority below SystemCritical, the slot is vacated and the method returns true. This is wired to VAD detection—when the user starts speaking, barge-in immediately halts TTS output to allow the new utterance to be processed.

Sources: portal/voice/src/tts_arbitration/arbitrator.rs#L35-L129, portal/voice/src/tts_arbitration/speech_token.rs#L12-L25, portal/voice/src/tts_arbitration/mod.rs#L14-L55

FastRPC Transport: DMA-BUF to Hexagon DSP

Beneath the scheduler abstraction lies the portal-npu-runtime crate, which provides safe Rust wrappers around the Qualcomm FastRPC kernel interface. The DSP is accessed through the Compute DSP Secure Domain (CDSP_DOMAIN_ID = 3), opened at /dev/fastrpc-cdsp-secure. The FastRpcSession struct owns two file descriptors (a metadata fd and an RPC fd) and serializes all ioctl operations through a per-session Mutex to avoid kernel-level race conditions.

sequenceDiagram
    participant Rust as PortalNpuScheduler
    participant Runtime as FastRpcSession
    participant Kernel as FastRPC Kernel Driver
    participant DSP as Hexagon DSP

    Rust->>Runtime: submit_inference(job)
    Runtime->>Kernel: ALLOC_DMA_BUFF ioctl
    Kernel-->>Runtime: DMA-BUF fd + host mmap ptr
    Runtime->>Kernel: MMAP ioctl (map to DSP)
    Kernel-->>Runtime: DSP virtual address
    Note over Runtime,DSP: Tensor data written to DMA-BUF<br/>via host mmap
    Runtime->>Kernel: INVOKE ioctl (scalars)
    Kernel->>DSP: FastRPC dispatch<br/>portal_qnn_execute()
    DSP-->>Kernel: Inference result in DMA-BUF
    Kernel-->>Runtime: ioctl returns
    Runtime->>Runtime: SeqCst fence + read DMA-BUF
    Runtime-->>Rust: NpuJobResult

DMA-BUF Allocation and Mapping

DMA-BUF allocation follows a three-step pattern: ALLOC_DMA_BUFF ioctl returns a kernel file descriptor, mmap with PROT_READ|PROT_WRITE and MAP_SHARED maps it into the host process, and MMAP ioctl maps it into the DSP’s address space. All sizes are page-rounded to 4096 bytes and bounded at MAX_DMA_ALLOC (256 MB) to prevent overflow during rounding.

The DmaBuffer wrapper is intentionally !Sync—its UnsafeCell<*mut [u8]> field marks interior mutability because the DSP may write concurrently via DMA. Read access uses with_data, which issues a SeqCst fence before reading to ensure the CPU observes any in-flight DSP writes. Write access uses with_data_mut, which fences before writing to order prior DSP reads.

Shell Binary Verification

Before any model can be loaded, the FastRPC shell binary (/usr/lib/dsp/cdsp/fastrpc_shell_unsigned_3) undergoes SHA-256 + Ed25519 cryptographic verification via verify_shell_binary. If verification fails, the session open fails closed—no shell is loaded, and no model inference is possible. This is a security hardening measure (labeled P0 #4) to prevent DSP code injection via a tampered shell.

Sources: portal/npu-runtime/src/lib.rs#L71-L115, portal/npu-runtime/src/lib.rs#L283-L352, portal/npu-runtime/src/device.rs#L14-L183, portal/npu-runtime/src/fastrpc_ffi.rs#L13-L23

QNN Interface: Model Loading and Graph Execution on DSP

The portal_qnn.idl interface definition specifies five RPC methods that the DSP skeleton (portal_qnn_skel.c) implements against Qualcomm’s QNN SDK. The skeleton manages a fixed-size model table (MAX_MODELS = 4, MAX_GRAPHS_PER_MODEL = 8) and loads model .so archives via dlopen, calling their composeGraphs entry point to register computation graphs with the QNN backend.

IDL Method	Purpose	Key Parameters
`model_load`	Load QNN model onto DSP	`model_so_dsp_addr`, `model_so_size` → `model_handle`, `n_graphs`
`set_input`	Set input tensor on a graph	`model_handle`, `graph_idx`, `tensor_idx`, `data_dsp_addr`
`execute`	Run inference on a graph	`model_handle`, `graph_idx`
`get_output`	Retrieve output tensor	`model_handle`, `graph_idx`, `tensor_idx`, `data_dsp_addr`
`model_unload`	Free model resources	`model_handle`

The QNN context lifecycle mirrors the scheduler’s model registry: a model is loaded once (acquiring a handle), inputs are set per-inference, the graph executes, outputs are read, and the model is unloaded at shutdown. The scheduler wraps this lifecycle through register_model / unregister_model for lifecycle tracking, while submit_inference drives the per-frame set_input → execute → get_output cycle.

Sources: portal/npu-runtime/idl/portal_qnn.idl#L1-L41, portal/npu-runtime/skel/portal_qnn_skel.c#L14-L61

LLM on Hexagon: GenieX Subprocess Bridge

The LLM workload follows a different execution model than STT/TTS. Rather than loading a QNN model directly, PortalLlmHexagon spawns a Python subprocess running GenieX (Qualcomm’s LLM runtime wrapper over llama.cpp’s Hexagon backend). The bridge communicates via stdin/stdout using a JSON-RPC-style protocol: the parent writes a request object to the child’s stdin, and reads a JSON response from stdout.

The subprocess is configured with GGML_HEXAGON_HOSTBUF=0 (disabling host-buffered inference), GENIEX_N_CTX=512 (512-token context window), and resolves the Python interpreter via an absolute path (PORTAL_PYTHON_BIN, default /usr/bin/python3) to prevent PATH hijacking. The venv directory (PORTAL_VENV_DIR) is required—there is no hardcoded fallback, and a missing env var produces an immediate error.

On the scheduling side, the LLM workload is registered on Core 2 with Low priority. The GenieX bridge uses Mutex<Option<GenieXBridge>> for thread-safe access, and the generating AtomicBool flag allows non-blocking status queries. Context resets invoke bridge.reset() on the subprocess, which drops the KV cache.

Sources: portal/llm/src/hexagon_engine.rs#L15-L111, portal/llm/src/hexagon_engine.rs#L113-L188, portal/voice/qnn-sys/src/genie.rs#L1-L48

Feature Flag Architecture

NPU scheduling is gated behind a layered feature flag system. The npu-scheduler and qnn-scheduler features are empty marker features that control conditional compilation of scheduler integration code. When neither is enabled, the Voice struct omits the npu_scheduler field entirely via #[cfg], and all scheduler-dependent code paths compile to no-ops.

Feature Flag	Dependency	Effect
`npu-scheduler`	(none)	Enables `PortalNpuScheduler` field in `Voice`
`qnn-scheduler`	`qnn-sys`	Enables scheduler + QNN backend types
`qnn-vad`	`qnn-sys`	Enables `RknnVadDetector` with scheduler hooks
`qnn-tts`	`qnn-sys`, `piper-plus-g2p`, `rand`	Enables `ElaraVitsEngine` with QNN models
`qnn-stt`	`qnn-sys`, `libloading`	Enables STT via QNN shared libraries

The production HP build uses --features 'qnn-tts,qnn-vad,npu-stt,network-audio,nlu-minilm', which activates QNN backends for TTS and VAD while keeping STT on a dynamically-loaded library path. The scheduler itself is always available in these builds through the qnn-scheduler transitive enablement from qnn-tts.

Sources: portal/voice/Cargo.toml#L72-L89, portal/voice/src/npu_backend.rs#L11-L51, portal/voice/src/voice.rs#L24-L27

Integration in the Voice Facade

The Voice struct is the top-level facade that wires the scheduler into the voice pipeline. During Voice::new(), the scheduler is conditionally created when config.npu_enabled is true, wrapped in Arc<PortalNpuScheduler>. This Arc is then passed to create_tts_engine_with_scheduler, which injects the scheduler into the Elara VITS engine for per-inference timing and health tracking. VAD detectors that support scheduler integration (via load_with_scheduler) receive the same Arc clone.

The telemetry snapshot from Voice::telemetry() includes an npu_telemetry field (defaulted to empty when telemetry is unavailable), providing per-core utilization and health status to monitoring systems.

Sources: portal/voice/src/voice.rs#L40-L52, portal/voice/src/voice.rs#L155-L203, portal/voice/src/vad/npu_vad.rs#L82-L92