Voice/AI · voice

Portal LLM — On-Device Language Model

On-device LLM daemon running Qwen3-0.6B on the Hexagon DSP via GenieX/llama.cpp. Serves generation requests via Unix socket IPC at /run/portal/llm.sock.

portal-llmv0.2.0voice

1.8K

Lines of Code

Tests

Files

Architecture

Four modules: client (LlmClient — Unix socket client for other subsystems), config (LlmConfig — model path, context window, system prompt), ipc (binary protocol: LlmRequest, LlmResponse, LlmStatus, LlmErrorCode), text_utils (build_elara_prompt, strip_think_tags). The hexagon_engine module (feature-gated) wraps the GenieX Python subprocess. The daemon binary portal-llmd is in portal/llm/bin/portal-llmd/.

Overview

portal-llm provides on-device language model inference for Portal's voice assistant (Elara). It wraps GenieX/llama.cpp running on the Hexagon DSP via FastRPC, loading a Qwen3-0.6B Q4_0 model (335MB). The daemon exposes a Unix socket RPC at /run/portal/llm.sock that other Portal subsystems (portal-voiced, portal-context) connect to as clients. The system prompt is 'You are Elara. Answer the user directly in first person. Never narrate or describe actions. Be concise.' Generation latency: 0.3-0.5s per query, ~30 tok/s decode on DSP. Memory: ~900MB DSP footprint with HOSTBUF=0 and n_ctx=512.

Key Types

LlmClientUnix socket client — connects to /run/portal/llm.sock, sends LlmRequest, receives LlmResponse.

LlmConfigConfiguration — model path, context window size (512), system prompt, generation parameters.

LlmRequestIPC request — contains the user transcript/prompt.

LlmResponseIPC response — contains generated text, status, error code.

build_elara_prompt()Builds the Elara system prompt — passes transcript directly (lowercased, no wrapping).

strip_think_tags()Strips <think> tags from LLM output before TTS synthesis.

Modules

client

LlmClient — Unix socket client for connecting to portal-llmd from other subsystems.

config

LlmConfig — model path (/home/portal/qwen3-0.6b-q4_0.gguf), context window (512), system prompt.

ipc

IPC protocol — binary framing over Unix socket: LlmRequest, LlmResponse, LlmStatus, LlmErrorCode.

text_utils

Text utilities — build_elara_prompt (system prompt assembly), strip_think_tags (clean output).

hexagon_engine

Hexagon DSP engine (feature-gated) — wraps GenieX Python subprocess, manages model lifecycle on DSP.

Data Flow

portal-voiced → LlmClient → Unix socket /run/portal/llm.sock → portal-llmd → GenieX Python subprocess → Hexagon DSP → Qwen3-0.6B → generated text → response

Dependencies

Internal (1)

portal-common

External (7)

serde workspaceserde_json workspacepostcard workspacethiserror workspacetracing workspacetracing-subscriber workspacetokio workspace

Feature-gated: hexagon feature enables the DSP engine. External: tokio (async), serde (IPC serialization). Runtime: GenieX Python venv, Qwen3-0.6B model file.