Skip to content
Demain Docs
Waitlist →

On-Device LLM Daemon: GenieX llama.cpp on the Hexagon DSP

The portal-llmd daemon brings conversational AI directly onto the Snapdragon X Elite’s Neural Processing Unit, serving Qwen3-0.6B completions through a Unix-socket RPC. Rather than routing to cloud endpoints, every inference request flows from the voice pipeline through postcard-framed IPC, across a JSON-over-stdio bridge to a GenieX Python subprocess, and ultimately into llama.cpp’s ggml-hexagon backend running on the Hexagon DSP via FastRPC. This page traces that full request path — from client connection to DSP offload — and documents the security, concurrency, and text-safety layers that make unattended on-device LLM inference production-safe.

Sources: portal/llm/src/lib.rs#L1-L51, portal/npu-runtime/P10_LLM_STATUS.md#L1-L58

Architecture: Three-Tier Inference Stack

Section titled “Architecture: Three-Tier Inference Stack”

The LLM daemon decomposes into three distinct tiers, each with its own process boundary and serialization format. This separation is deliberate: the Rust daemon handles security and concurrency, the Python bridge manages model lifecycle and tokenizer chat templates, and the DSP backend executes tensor math. Understanding where each responsibility lives is key to tracing a request end-to-end.

flowchart TB
    subgraph Clients["Portal Subsystems"]
        VP["Voice Pipeline<br/>(ResponsePipeline)"]
        CTX["Context Engine"]
        PCP["PCP Daemon"]
    end

    subgraph Daemon["portal-llmd (Rust, User: portal)"]
        LST["UnixListener<br/>/run/portal/llm.sock"]
        AUTH["SO_PEERCRED<br/>UID Verification"]
        POOL["Semaphore<br/>Max 4 Concurrent"]
        RL["Rate Limiter<br/>100 req/sec"]
        HNDL["Request Handler<br/>(Postcard IPC)"]
    end

    subgraph Bridge["GenieXBridge (Python Subprocess)"]
        STDIO["JSON over stdio<br/>Newline-delimited"]
        GENX["geniex.AutoModelForCausalLM"]
        TPL["Tokenizer<br/>apply_chat_template"]
    end

    subgraph DSP["Hexagon CDSP"]
        GGUF["Qwen3-0.6B Q4_0 GGUF<br/>(365 MB resident)"]
        FASTRPC["FastRPC / CDSP"]
    end

    VP -->|"LlmRequest::Generate"| LST
    CTX -.->|"LlmRequest::Generate"| LST
    PCP -.->|"LlmRequest::HealthCheck"| LST
    LST --> AUTH --> POOL --> RL --> HNDL
    HNDL -->|"JSON request"| STDIO
    STDIO --> GENX --> TPL --> GGUF
    GGUF --> FASTRPC
    FASTRPC -.->|"JSON response"| STDIO
    STDIO -.->|"Postcard response"| HNDL

The daemon binary (portal-llmd) is feature-gated behind the hexagon Cargo feature — it compiles only when the GenieX DSP bridge is available. The library crate (portal-llm) exposes a platform-agnostic core: the IPC protocol, client, configuration, and text utilities are always compiled, while the hexagon_engine module activates conditionally. This split lets unit and integration tests run on any platform without DSP hardware.

Sources: portal/llm/Cargo.toml#L24-L34, portal/llm/src/lib.rs#L31-L50

The GenieX Bridge: Rust-to-Python-to-DSP

Section titled “The GenieX Bridge: Rust-to-Python-to-DSP”

The PortalLlmHexagon struct is the engine abstraction the daemon holds. Internally, it wraps a GenieXBridge — a managed Python Child process communicating over stdin/stdout pipes using newline-delimited JSON. This bridge pattern isolates the GenieX/llama.cpp dependency boundary inside a subprocess, protecting the Rust daemon from Python GIL contention or C-extension crashes.

Startup sequence: When load() is called, the bridge spawns the Python interpreter with an embedded script (passed via -c), sets environment variables for the model path, device map, tokenizer path, and context window size, then blocks reading a single line from stdout — the {"ready": true} handshake. The model loads once at daemon startup and stays resident in DSP memory for the daemon’s lifetime.

Sources: portal/llm/src/hexagon_engine.rs#L26-L47, portal/llm/src/hexagon_engine.rs#L118-L188

Embedded Python Script

Section titled “Embedded Python Script”

The bridge script is embedded directly in the Rust source as a raw string constant. This eliminates the need for a separate Python file on the filesystem — the daemon is self-contained. The script performs three operations: model initialization via geniex.AutoModelForCausalLM, a readiness handshake, and a read-eval-print loop that processes JSON requests.

Python Operation Trigger Input Output
AutoModelForCausalLM.from_pretrained() Daemon load() GGUF path + device_map + n_ctx {"ready": true}
tokenizer.apply_chat_template() Per generate() call Messages array with system + user roles Prompt string
model.generate() Per generate() call Prompt + max_new_tokens Response text
model.reset() Per generate() or explicit reset — KV cache cleared
model.close() Process exit (Drop) — DSP context released

Each generate call begins with model.reset() to clear the KV cache, ensuring every response is independent with no conversation history bleeding between requests. The tokenizer’s apply_chat_template is called with enable_thinking=False, disabling Qwen3’s chain-of-thought reasoning mode at the template level — the model never generates <think> blocks.

Sources: portal/llm/src/hexagon_engine.rs#L283-L332, portal/llm/src/hexagon_engine.rs#L53-L80

Environment Hardening

Section titled “Environment Hardening”

The bridge refuses to operate without explicit environment configuration — no hardcoded fallback paths exist in production code. This is a v0.2.1 hardening decision documented across two tracked issues (T42, P2-22):

Variable Purpose Default Hardening Rationale
PORTAL_VENV_DIR GenieX Python virtualenv root None — fails loud Prevents PATH hijacking via planted directories
PORTAL_PYTHON_BIN Absolute Python interpreter path /usr/bin/python3 Bare python3 searches PATH, exploitable via PATH injection
GENIEX_MODEL_PATH GGUF model file Set from CLI --model-path Model location is deployment-specific
GENIEX_DEVICE_MAP Device placement strategy "auto" Controls CPU/NPU split
GENIEX_N_CTX Context window size 512 Bounds DSP memory allocation
GGML_HEXAGON_HOSTBUF Hexagon host buffer flag 0 Disables host-side buffering for latency

Sources: portal/llm/src/hexagon_engine.rs#L135-L179, portal/systemd/portal-voice.env.example#L1-L23

IPC Protocol: Postcard over Unix Sockets

Section titled “IPC Protocol: Postcard over Unix Sockets”

The daemon listens on /run/portal/llm.sock and speaks a compact binary protocol. The wire format is a 4-byte big-endian length prefix followed by a postcard-serialized payload. Postcard — a no-std, varint-encoded Serde format — was chosen over JSON or bincode for minimal wire size and deterministic cross-platform deserialization. The protocol defines three request variants and three response variants, all as plain Rust enums with #[derive(Serialize, Deserialize)].

sequenceDiagram
    participant C as Client (Voice/Context/PCP)
    participant D as portal-llmd Daemon
    participant B as GenieX Bridge (Python)
    participant NPU as Hexagon DSP

    C->>D: Connect to /run/portal/llm.sock
    D->>D: SO_PEERCRED check (UID == portal?)
    D->>D: Semaphore try_acquire (max 4)
    D->>D: RateLimiter check (100 req/s)
    C->>D: LlmRequest::Generate { prompt, max_tokens }
    D->>B: {"messages": [...], "max_tokens": N}
    B->>B: tokenizer.apply_chat_template()
    B->>B: model.reset()
    B->>NPU: model.generate(prompt, max_new_tokens)
    NPU-->>B: Response text
    B->>B: Strip <think> tags (regex)
    B-->>D: {"ok": true, "text": "..."}
    D->>D: strip_think_tags() (defense-in-depth)
    D-->>C: LlmResponse::Generated { text, tokens }

Request and Response Types

Section titled “Request and Response Types”
LlmRequest Variant Fields Access Level Description
Generate prompt: String, max_tokens: Option<u32> Portal user Text completion from prompt
HealthCheck — Portal user Query model status without inference
Shutdown — Root only (UID 0) Graceful daemon termination
LlmResponse Variant Fields Description
Generated text: String, tokens_generated: u32 Successful completion
Error code: LlmErrorCode, message: String Structured failure
Health model_loaded: bool, npu_core: u8, kv_cache_usage: f32 Health probe result

The LlmErrorCode enum provides six distinct error codes — ModelNotLoaded, GenerationFailed, InvalidPrompt, Timeout, InternalError, PermissionDenied — enabling clients to choose appropriate retry strategies without parsing error strings.

Sources: portal/llm/src/ipc.rs#L1-L72, portal/llm/src/ipc.rs#L87-L155

Payload Safety

Section titled “Payload Safety”

The protocol enforces a 64 KiB maximum payload size on every received frame. If a peer sends a length prefix exceeding MAX_PAYLOAD_SIZE, the daemon returns an InvalidData I/O error before allocating any buffer. This prevents a hostile or buggy client from triggering unbounded heap allocations via a crafted 4-byte length prefix.

Sources: portal/llm/src/ipc.rs#L21-L22, portal/llm/src/ipc.rs#L116-L123

Daemon Lifecycle and Connection Handling

Section titled “Daemon Lifecycle and Connection Handling”

The daemon’s main() function follows a strict startup sequence: parse configuration, validate bounds, initialize tracing, load the model into DSP memory, bind the Unix socket, set restrictive permissions, and enter the accept loop. Each step has explicit failure handling — any error during model load or socket bind causes an immediate exit(1) with a logged diagnostic, because a half-initialized LLM daemon has no useful behavior.

Connection Accept Loop

Section titled “Connection Accept Loop”
flowchart TD
    ACC["listener.accept()"] --> SH{"shutdown flag?"}
    SH -->|Yes| DONE["Remove socket, exit"]
    SH -->|No| PC{"SO_PEERCRED<br/>UID == portal?"}
    PC -->|No| REJ["Log + drop connection"]
    PC -->|Yes| SEM{"Semaphore<br/>try_acquire?"}
    SEM -->|Full| DROP["Drop connection<br/>(client gets EOF)"]
    SEM -->|Acquired| SPAWN["thread::spawn"]
    SPAWN --> HDL["handle_connection()"]
    HDL --> RL{"Rate limit<br/>check?"}
    RL -->|Exceeded| ERR["LlmResponse::Error"]
    RL -->|OK| PROC["Process request"]
    PROC --> ACC

Each accepted connection acquires a permit from a tokio Semaphore bounded at 4 concurrent connections (T48 hardening). This leaves headroom for the three expected callers — voice, context, and PCP — plus one ad-hoc client (health check, debug CLI). When the pool is saturated, new connections are immediately dropped rather than queued; clients receive EOF and are expected to retry with backoff. This design prevents thread proliferation and DSP context exhaustion under a connection-flooding scenario.

Sources: portal/llm/bin/portal-llmd/main.rs#L44-L198, portal/llm/bin/portal-llmd/main.rs#L200-L312

Rate Limiting

Section titled “Rate Limiting”

Beyond the connection pool, each accepted connection gets its own RateLimiter instance (token-bucket, 100 req/sec) from portal-common. A hostile client can only exhaust its own bucket — the per-connection isolation prevents one slow or abusive peer from starving others. In the current one-request-per-connection protocol, this guard is defensive; it becomes load-bearing if the protocol is extended to a persistent request loop.

Sources: portal/llm/bin/portal-llmd/main.rs#L52-L58, portal/llm/bin/portal-llmd/main.rs#L207-L245, portal/common/src/rate_limiter.rs#L50-L87

Security Model: Three-Layer Defense

Section titled “Security Model: Three-Layer Defense”

The daemon implements a defense-in-depth strategy with three independent security layers, each addressing a different threat model:

Layer Mechanism Threat Mitigated Failure Mode
Authentication SO_PEERCRED via getsockopt Unauthorized process connecting Connection dropped, logged
Authorization UID-gated privileged operations Non-root requesting Shutdown PermissionDenied error response
Availability Connection pool + rate limiter DoS via connection flooding or request spam Connection dropped / error response

The require_peer_cred function calls getsockopt(SO_PEERCRED) on every accepted socket, verifying the connecting process’s UID matches the portal user (resolved once via getpwnam and cached in a OnceLock). The peer UID is carried into the handler so that the LlmRequest::Shutdown branch can require peer_uid == 0 without a second syscall. Non-root peers attempting shutdown receive LlmErrorCode::PermissionDenied with a diagnostic message.

The socket file itself is chmod 0o600 immediately after bind() to prevent world-read access. Without this, the socket would inherit the system umask (typically 0o022 → 0o755), allowing any local user to connect.

Sources: portal/llm/bin/portal-llmd/main.rs#L110-L119, portal/llm/bin/portal-llmd/main.rs#L152-L165, portal/llm/bin/portal-llmd/main.rs#L290-L308, portal/common/src/ipc.rs#L62-L150

Text Processing and Prompt Safety

Section titled “Text Processing and Prompt Safety”

User input travels through two sanitization stages before reaching the model. The first, sanitize_user_input, strips C0 control characters (except tab, newline, carriage return), the DEL character, and literal two-character escape pairs (\n, \t, \r as typed backslash-letter). This prevents prompt-template injection where a user could type literal escape sequences to break out of the JSON blob that the GenieX bridge constructs. The sanitizer preserves all printable Unicode, punctuation, and backslashes not followed by n/t/r (e.g., Windows paths survive).

The second stage, strip_think_tags, removes Qwen3’s <think> reasoning blocks from model output. Although the Python bridge applies enable_thinking=False at the template level and strips <think> via regex, the Rust-side filter provides defense-in-depth — if the model emits residual reasoning tags despite the template setting, they never reach the client.

Sanitization Stage Location Input Removes Preserves
sanitize_user_input Rust (hexagon_engine.rs) Raw user prompt C0 controls, DEL, \n/\t/\r pairs Printable Unicode, \ + non-ntr
apply_chat_template Python (script()) Messages array Adds system/user roles Structured prompt
strip_think_tags Rust (hexagon_engine.rs) Model output <think>...</think> blocks Final response text

Sources: portal/llm/src/text_utils.rs#L48-L84, portal/llm/src/text_utils.rs#L21-L46, portal/llm/src/hexagon_engine.rs#L53-L80

Client API: LlmClient

Section titled “Client API: LlmClient”

Other Portal subsystems interact with the daemon exclusively through LlmClient — a thin synchronous wrapper that connects per-request (no connection pooling). This design choice reflects the expected workload: at most three callers (voice, context, PCP), each sending infrequent requests. Each generate or health_check call opens a fresh Unix socket, sends the request, reads the response, and closes the connection.

The client maps daemon error responses to typed errors via LlmClientError:

Error Variant Condition Recovery Strategy
ConnectionFailed Socket not found / daemon down Fall back to canned response
Timeout No response within timeout (default 10s) Retry with backoff or fallback
ServerError Daemon returned LlmResponse::Error Inspect LlmErrorCode for retry decision
IoError Network-level I/O failure Retry or degrade gracefully

The primary consumer is the voice pipeline’s ResponsePipeline, which instantiates an LlmClient pointed at /run/portal/llm.sock by default. When the classifier produces a low-confidence intent (not a system command), the pipeline calls llm_client.generate() for a conversational response. If the daemon is unavailable, the pipeline falls back to a canned response — never echoing the user’s raw input.

Sources: portal/llm/src/client.rs#L1-L67, portal/llm/src/client.rs#L88-L202, portal/voice/src/nlu/pipeline.rs#L1-L77

NPU Integration: Why GenieX llama_cpp

Section titled “NPU Integration: Why GenieX llama_cpp”

The Snapdragon X Elite’s CDSP firmware (c1-00046) exposes QNN API version 2.33.0, which is below the 2.37.0+ required by Qualcomm’s QAIRT Genie runtime. This version gap blocks the standard QAIRT C-API path for LLM inference. The GenieX llama_cpp runtime sidesteps this entirely: it uses llama.cpp’s ggml-hexagon backend, which communicates with the DSP through FastRPC/CDSP directly — a completely separate channel that doesn’t depend on the QNN API version check.

Five alternative approaches were tried and failed before this solution was verified:

Approach Failure Mode
QAIRT Genie C API QNN API 2.33.0 < 2.37.0+ required
c1-00069 firmware Rejected by HP TrustZone (PIL error -22)
qnn-genai-transformer-composer GGUF Wrong tensor names for llama.cpp
Phi-3.5-mini pre-compiled bundle Same QNN API version block
GenieX qairt runtime Same QNN API version block

Verified performance on the HP EliteBook Ultra G1q (Snapdragon X Elite, 2026-07-13): prefill at 412 tok/s (NPU-accelerated), decode at 42 tok/s, time-to-first-token between 84–247 ms, and total response time of 0.2–0.5 seconds per query. The Qwen3-0.6B Q4_0 GGUF model occupies 365 MB and remains resident in DSP memory for the daemon’s lifetime.

Sources: portal/npu-runtime/P10_LLM_STATUS.md#L1-L58, portal/npu-runtime/NPU_WORKING.md#L1-L70

Configuration Reference

Section titled “Configuration Reference”

The LlmConfig struct centralizes all tunable parameters. Defaults are calibrated for Qwen3-0.6B with a 2048-token context window and moderate sampling diversity. Configuration flows through three layers with increasing precedence: compiled defaults → CLI arguments → environment variables.

Parameter Default Valid Range Description
model_path /usr/local/share/portal/models/llm/model.rkllm Path with existing parent dir GGUF model file
max_new_tokens 256 1–4096 Max tokens per generation
max_context_len 2048 > 0 Prompt + generation budget
temperature 0.7 [0.0, 2.0] Sampling temperature
top_p 0.9 [0.0, 1.0] Nucleus sampling threshold
top_k 40 > 0 Top-K sampling limit
repeat_penalty 1.1 ≥ 1.0 Repetition penalty factor
content_safety_check true boolean Enable input filtering
socket_path /run/portal/llm.sock Any path IPC socket location
system_prompt Elara persona Any string Default system instruction

CLI arguments: --model-path PATH and --socket-path PATH Environment overrides: PORTAL_LLM_MODEL_PATH, PORTAL_LLM_SOCKET_PATH, PORTAL_LLM_LOG_LEVEL

Sources: portal/llm/src/config.rs#L16-L54, portal/llm/src/config.rs#L56-L133

Systemd Deployment

Section titled “Systemd Deployment”

The daemon runs as the portal user under the audio group, spawned by portal-llm.service. It depends on portal.service (base platform initialization) and reads its environment from /etc/portal/portal-voice.env. The service enforces filesystem restrictions: ReadWritePaths=/run/portal (socket directory) and ReadOnlyPaths=/usr/share/portal /home/portal (model files and venv).

[Service]
Type=simple
User=portal
Group=audio
EnvironmentFile=-/etc/portal/portal-voice.env
Environment=RUST_LOG=portal_llm=info
ExecStart=/usr/local/bin/portal-llmd --model-path /home/portal/qwen3-0.6b-q4_0.gguf
Restart=always
RestartSec=5
ReadWritePaths=/run/portal
ReadOnlyPaths=/usr/share/portal /home/portal

The Restart=always with RestartSec=5 ensures the daemon recovers from Python subprocess crashes or DSP resets within 5 seconds. The environment file must define PORTAL_VENV_DIR pointing at the GenieX virtualenv — the bridge will refuse to start without it.

Sources: portal/systemd/portal-llm.service#L1-L27, portal/systemd/portal-voice.env.example#L1-L23

Testing Strategy

Section titled “Testing Strategy”

The LLM crate employs three tiers of testing, each targeting a different layer of the stack:

Test Tier Scope Key Tests DSP Required?
Unit tests IPC serialization, config validation, text sanitization Round-trip postcard, payload size rejection, think-tag stripping No
Property tests Fuzzing IPC types with proptest LlmRequest roundtrip, LlmErrorCode equality, LlmResponse::Error message preservation No
Integration tests Client-daemon interaction via mock server Generate success, error propagation, connection failure, health check No

The unit and property tests exercise the pure-Rust layers (IPC framing, configuration bounds, text processing) without needing DSP hardware. The integration tests spawn a mock daemon on a per-test Unix socket, verifying client-side error handling and success paths. Tests requiring the hexagon feature (actual model loading, DSP offload) are gated behind #[cfg(feature = "hexagon")] and run only on target hardware.

Sources: portal/llm/src/ipc/tests.rs#L1-L200, portal/llm/src/proptests.rs#L1-L59, portal/llm/tests/client_integration.rs#L1-L200, portal/llm/tests/main_init.rs#L1-L200

Section titled “Related Pages”