Fusion Engine: Touch Tracking, Intent Classification, and Action Generation

The Fusion Engine is the central nervous system of the Portal input stack — a deterministic intent classifier that transforms raw capacitive touch data into discrete input actions. It sits at the intersection of three concerns: tracking multi-touch state across frames, classifying what a user means by each contact, and dispatching concrete actions to virtual input devices. The engine’s design philosophy is priority-chain classification with zero-allocation hot paths, achieving sub-50μs per-event processing on the Snapdragon A55 cores that power the Spaceboard.

Touch Capture: From evdev Raw Bytes to Normalized Events

The touch pipeline follows a strict three-stage architecture — Capture → Normalize → Track — where each stage has a single, well-defined responsibility. Raw kernel events arrive via the Linux evdev interface using Multi-Touch Protocol B (MT-B), which tracks touch points by slot rather than requiring stateful reconstruction per frame. The EvdevTouchCapture backend opens a whitelisted touch device with O_NONBLOCK, validates that it supports all required ABS_MT_* axes, and queries device capabilities via ioctl to obtain min/max coordinate ranges. Events are read in batches up to the next SYN_REPORT boundary, with out-of-bounds position events silently dropped to prevent malformed input from corrupting the pipeline.

Sources: portal/input/src/touch/capture/backend.rs#L22-L62, portal/input/src/touch/capture/backend.rs#L70-L161

Normalization is performed by DeviceNormalizer, a stateless pure-math transform that maps raw integer coordinates to the [0.0, 1.0] range using the device’s reported x_min/x_max/y_min/y_max capabilities. This avoids hardcoded resolution constants and enables the same downstream logic to work across any touch panel. The transform clamps to [0, 1] to handle edge cases where the panel reports values slightly outside its declared range.

Sources: portal/input/src/touch/normalizer.rs#L14-L63

flowchart LR
    A["evdev Device<br/>/dev/input/eventX"] -->|"RawEvdevEvent<br/>(type, code, value, ts)"| B["SlotTracker<br/>[SlotState; 10]"]
    B -->|"ABS_MT_SLOT<br/>ABS_MT_TRACKING_ID<br/>ABS_MT_POSITION_X/Y"| B
    B -->|"SYN_REPORT"| C["DeviceNormalizer<br/>[0,1] normalization"]
    C --> D["TouchEvent<br/>10 fixed-size slots"]
    D --> E1["Channel A<br/>→ Gesture Worker"]
    D --> E2["Channel B<br/>→ Fusion Worker"]

The SlotTracker is the critical component that bridges raw evdev and structured touch events. It maintains a fixed-size [SlotState; MAX_SLOTS] array (MAX_SLOTS = 10) with zero heap allocation on the hot path. The tracker processes ABS_MT_SLOT to select the current slot, ABS_MT_TRACKING_ID to detect touch start/end, and position updates as they arrive. Only on SYN_REPORT does it emit a fully assembled TouchEvent — a single struct containing all 10 touch point slots, a changed-slot bitmask, a primary slot index, and an aggregate phase (Start/Move/End/Cancel). Post-frame state transitions reset Released slots to Idle and advance Active slot phases to Move, ensuring the next frame starts in a clean state.

Sources: portal/input/src/touch/tracker.rs#L30-L84, portal/input/src/touch/tracker.rs#L135-L231, portal/input/src/touch/types.rs#L14-L138

TouchPhase	Trigger Condition	Semantic Meaning
Start	New tracking_id ≥ 0 on previously idle slot	Finger just touched surface
Move	Position update on active slot	Finger is sliding
End	tracking_id = -1 on active slot	Finger lifted
Cancel	active_count = 0 with no End in changed mask	Device reset or all touches lost

Dual-Channel Fan-Out: Touch Events to Both Classifiers

A fundamental architectural decision is that the Fusion Engine and Gesture Recognizer operate as independent parallel consumers of the same touch stream, not as a sequential pipeline. The ChannelEndpoints struct creates six bounded MPSC channels (A–F) with DropNewest overflow semantics, meaning that when a channel is full, new events are silently dropped rather than backpressuring the producer. The fan_out_touch method sends each TouchEvent to both Channel A (gesture worker) and Channel B (fusion worker) simultaneously, allowing each consumer to maintain its own state machine without gating or filtering the other.

Sources: portal/input/src/channels/endpoints.rs#L14-L81, portal/input/src/channels/constants.rs#L1-L57

Channel	Payload	Producer	Consumer	Capacity
A	TouchEvent	Touch thread	Gesture worker	64
B	TouchEvent	Touch thread	Fusion worker	64
C	GestureEvent::Recognized	Gesture worker	Fusion worker	32
D	ClassifiedIntent	Fusion worker	Output worker	64
E	RenderLayout	Layout worker	Compositor	16
F	GestureEvent (PhaseChanged/Cancelled)	Gesture worker	Layout worker	32

Gesture events from the recognizer are routed by route_gesture_event: Recognized events go to Channel C (reaching the fusion engine for action mapping), while PhaseChanged and Cancelled events go to Channel F (reaching the layout worker for UI feedback). This separation ensures that only completed gestures trigger input actions, while progress indicators flow to the rendering layer independently.

Sources: portal/input/src/channels/endpoints.rs#L90-L106

Thread scheduling is pinned to specific CPU cores using Linux SCHED_FIFO/SCHED_RR policies, with the touch thread at priority 50 on core A55-0 (the most latency-sensitive), and the fusion worker at priority 45 on core A55-1 with a 2ms time slice. This design ensures that touch capture never starves under load from classification work.

Sources: portal/input/src/channels/constants.rs#L23-L49

V21FusionEngine: The Priority Chain Classifier

The V21FusionEngine is the canonical implementation of the FusionEngine trait, designed around a fixed-priority classification chain that evaluates each active touch point against five intent categories in strict order. The engine maintains per-slot tracking state via a [Option<TrackedTouch>; MAX_SLOTS] array, where each TrackedTouch records start position, last position, velocity (computed over a 100ms sliding window), displacement from origin, a position history vector pruned to the same window, and a has_committed_key flag for detecting typing-to-gesture transitions.

Sources: portal/input/src/fusion/v21_engine/engine.rs#L18-L33, portal/input/src/fusion/v21_engine/types.rs#L7-L25, portal/input/src/fusion/engine.rs#L54-L94

The classification chain in classify_touch evaluates each active slot through five priority levels. The first matching priority wins and the slot advances to the next event:

flowchart TD
    START["Active Touch Point"] --> P1
    P1{"P1: Modifiers Active?<br/>(Portal/Warp/Vortex)"} -->|"Yes"| SYS["System Intent<br/>→ SystemAction"]
    P1 -->|"No"| P3
    P3{"P3: Edge Zone +<br/>Inward Motion?"} -->|"Yes"| EDGE["Gesture: EdgeSwipe<br/>(Fusion sole authority)"]
    P3 -->|"No"| C42
    C42{"C42: Committed Key<br/>+ Fast Slide?"} -->|"Yes"| SLIDE["Gesture: Slide<br/>Clear committed key"]
    C42 -->|"No"| P2
    P2{"P2: Touch Start +<br/>Low Displacement?"} -->|"Yes"| TYPE["Typing Intent<br/>→ KeyPress/ModifierHold"]
    P2 -->|"No"| P4
    P4{"P4: High Displacement<br/>(> typing_threshold)?"} -->|"Yes"| POINT["Pointing Intent<br/>→ PointerMove"]
    P4 -->|"No"| P5["P5/P6: Idle<br/>→ NoOp"]

Priority 1 — System (Modifier Override)

When any Portal-specific modifier (Portal, Warp, or Vortex) is active, the engine immediately classifies the touch as a System intent, bypassing all lower-priority checks. Each modifier maps to a distinct system action: Portal → Overview, Warp → SpatialMode, Vortex → ContextSwitch. A deduplication mechanism (SystemActionDeduplication) prevents double-triggering by enforcing a 500ms cooldown per action type, since modifiers can remain held across multiple touch events. Haptic feedback fires on mode change.

Sources: portal/input/src/fusion/v21_engine/classify.rs#L80-L103, portal/input/src/fusion/types.rs#L179-L208

Priority 2 — Typing (Keyboard Hit)

When a touch starts (phase = Start) and displacement is below the typing threshold (default: 0.05, or 5% of screen), the engine marks the slot as committed and resolves the key via slot_keys — a per-frame mapping of slot → evdev code that is populated by the layout layer’s hit-test. If the evdev code maps to a modifier (F13/F14/F15 → Portal/Warp/Vortex), the engine emits ModifierHold. Otherwise, it resolves the KeyCode, stores it in committed_keys for later release, and emits a KeyPress action with an optional character. This is the path that powers the on-screen keyboard.

Sources: portal/input/src/fusion/v21_engine/classify.rs#L146-L182, portal/input/src/fusion/key_code_mapping.rs#L14-L110

Priority 3 — Edge Swipe (Fusion Engine Sole Authority)

A critical design rule (designated C41/C53) is that the fusion engine — not the gesture recognizer — is the sole authority for edge swipe detection. When a touch starts within a configurable edge zone (default: 5% of screen width/height from each border) and exhibits inward motion exceeding the fling minimum displacement, the engine directly emits an EdgeSwipe system action tagged with the edge zone. The gesture recognizer’s vocabulary defines edge swipe patterns but explicitly does not register them, preventing double-classification.

Sources: portal/input/src/fusion/v21_engine/classify.rs#L105-L123, portal/input/src/gesture/vocabulary/mod.rs#L177-L195

C42 — Committed-Key-to-Gesture Transition

When a slot has already committed a key (via Priority 2) but the finger suddenly accelerates above fling_max_velocity (default: 0.8 normalized units/ms), the engine abandons the typing intent and reclassifies as a gesture. It clears the committed key, emits a SystemGesture with a slide gesture pattern, and fires gesture-complete haptics. This prevents accidental key rolls when a user intends to swipe from a key position.

Sources: portal/input/src/fusion/v21_engine/classify.rs#L125-L144

Priority 4 — Pointing

If displacement exceeds the typing threshold, the touch is classified as pointing. The engine computes the delta from the last position, scales it by screen dimensions, and emits a PointerMove action. This is the path for trackpad-style cursor control.

Sources: portal/input/src/fusion/v21_engine/classify.rs#L184-L199

Priority 5/6 — Idle / NoOp

If no priority matches, the engine emits an Idle intent with a NoOp action. This catches palm touches, ambiguous contacts, and any input that doesn’t fit a recognized pattern. The output layer silently ignores NoOp actions.

Sources: portal/input/src/fusion/v21_engine/classify.rs#L201-L208

Touch Lifecycle: Start, Commit, and Release

The process_touch_event entry point (via the FusionEngine trait impl) handles the complete touch lifecycle, not just classification. On TouchPhase::End, it emits KeyRelease intents for every slot that had committed a key, clears those committed keys, and removes the tracked touch entries. This ensures that every key press has a matching release, preventing stuck-key states. Release intents are prepended to the classification results so they arrive at the output worker before any new press intents from the same frame.

Sources: portal/input/src/fusion/v21_engine/trait_impl.rs#L25-L47

The velocity computation uses a 100ms sliding window (VELOCITY_WINDOW_US = 100_000μs). The position_history vector for each tracked touch stores (x, y, timestamp) tuples and is pruned on every update to entries within the window. Velocity is computed as Euclidean distance from the oldest in-window position to the current position, divided by the time delta in milliseconds. This provides a smoothed velocity that resists single-frame spikes while remaining responsive enough for fling detection.

Sources: portal/input/src/fusion/v21_engine/tracking.rs#L11-L33, portal/input/src/fusion/v21_engine/tracking.rs#L41-L75

Gesture Recognition: Independent State Machine Observer

The StateMachineGestureRecognizer operates as a parallel observer that processes the same TouchEvent stream but maintains completely independent state. Its lifecycle follows a five-phase state machine: NoTouch → Possible → InProgress → Recognized | Cancelled. The recognizer does not gate, filter, or intercept touch events for the fusion engine — it simply watches and emits GestureEvents when patterns complete.

Sources: portal/input/src/gesture/state_machine/recognizer.rs#L15-L26, portal/input/src/gesture/mod.rs#L9-L16

stateDiagram-v2
    [*] --> NoTouch
    NoTouch --> Possible: TouchPhase::Start
    Possible --> InProgress: Duration > 50ms OR Progressive Upgrade
    Possible --> NoTouch: Timeout (2500ms)
    InProgress --> InProgress: TouchPhase::Move (accumulate snapshots)
    InProgress --> Recognized: TouchPhase::End + Pattern Match ≥ 0.6 confidence
    InProgress --> Cancelled: TouchPhase::End + No Match OR Superseded
    InProgress --> Cancelled: Timeout (2500ms)
    Recognized --> [*]: Emit GestureEvent::Recognized
    Cancelled --> [*]: Emit GestureEvent::Cancelled

Two timing windows govern multi-touch evolution:

Window	Default Duration	Purpose
Progressive Upgrade	300ms	A 2-finger gesture can upgrade to 3-finger when a new finger lands within this window
Multi-Touch Coalescence	200ms	Additional fingers arriving within this window are absorbed into the current gesture

Pattern matching uses early-exit evaluation (#28): patterns are sorted by descending priority, and the first match above the recognition_min_confidence threshold (default: 0.6) returns immediately without evaluating remaining patterns. Confidence scoring degrades based on how close displacement is to pattern limits and how far the detected direction deviates from the required direction within the 30° tolerance.

Sources: portal/input/src/gesture/state_machine/matching.rs#L52-L111, portal/input/src/gesture/state_machine/handlers.rs#L18-L86, portal/input/src/gesture/state_machine/process.rs#L15-L144

The gesture vocabulary defines 25 Tier 1 patterns (12 single-finger, 6 two-finger, 4 three-finger, 3 four-finger) and 7 Tier 2 patterns (spatial, portal system shortcuts). However, the 4 edge swipe patterns (T1.9–T1.12) are explicitly excluded from registration via registerable_patterns(), enforcing the fusion engine’s sole authority over edge gestures.

Sources: portal/input/src/gesture/vocabulary/mod.rs#L126-L195

Conflict Resolution: Six-Level Priority System

When multiple patterns match the same gesture data, the ConflictResolver applies a six-level cascade:

1. User-remapped gestures (deferred to explicit priority field)
2. App-specific overlays (deferred to explicit priority field)
3. More specific match — higher finger count + longer minimum path wins
4. Tier precedence — Trackpad > Portal > Future
5. Explicit priority field — higher numeric value wins
6. Trigger overlap analysis — if two patterns at the same priority have overlapping finger ranges, overlapping durations, and compatible directions, they are Superseded (both cancelled); otherwise they Coexist

Sources: portal/input/src/gesture/conflict.rs#L33-L106

The “superseded” concept is deliberately narrow: two patterns only conflict when they occupy the same input space (same fingers, overlapping time, same direction). Two swipes in different directions at the same priority coexist peacefully — they are mutually exclusive by their direction constraints.

Action Dispatch: From Intent to uinput

The ActionDispatcher is the final translation layer that converts abstract InputAction variants into concrete virtual device calls. It holds mutable references to a VirtualKeyboard and VirtualPointer (both backed by uinput) and maps each action type:

Sources: portal/input/src/bin/portal-keyboardd/action_dispatch.rs#L27-L66

InputAction Variant	Dispatch Target	Concrete Operation
KeyPress / KeyRelease	VirtualKeyboard	emit_key(key_code, pressed)
PointerMove	VirtualPointer	move_pointer(dx, dy) (scaled by screen dimensions)
PointerClick	VirtualPointer	click(button)
PointerDrag	VirtualPointer	press_button + move_pointer
PointerScroll	VirtualPointer	scroll(axis, amount)
PointerZoom	VirtualKeyboard	Ctrl+Equal (zoom in) / Ctrl+Minus (zoom out)
PointerRotate	VirtualKeyboard	Ctrl+Shift+Right / Ctrl+Shift+Left
SystemAction::Overview	VirtualKeyboard	Ctrl+Up
SystemAction::Launcher	VirtualKeyboard	Alt+F2
SystemAction::AppSwitcher	VirtualKeyboard	Alt+Tab
SystemAction::EdgeSwipe	VirtualKeyboard	Direction-specific key combos
ModifierHold	None	Fusion engine tracks state internally; no uinput dispatch
NoOp	None	Silent no-op

The emit_key_combo helper handles multi-key sequences (modifier presses, key press/release, modifier releases in reverse order) atomically, ensuring the compositor receives the complete chord. Individual device errors are logged but do not abort the dispatch loop, providing graceful degradation.

Sources: portal/input/src/bin/portal-keyboardd/action_dispatch.rs#L68-L244

Fusion Worker Thread: The Runtime Wiring

In the production portal-keyboardd daemon, the fusion engine runs on a dedicated fusion_worker thread that owns a V21FusionEngine instance. The worker receives EnrichedTouchEvent structs (a TouchEvent paired with per-slot evdev key codes from the layout hit-test) over Channel A, calls engine.process_touch_event(), and sends FusionChannelResult (containing the resulting intents and the original touch event for glow hit-testing) over Channel D. The worker uses a 100ms recv_timeout to remain responsive to channel disconnection.

Sources: portal/input/src/bin/portal-keyboardd/app/fusion_worker.rs#L34-L80

The main loop in KeyboardDaemon::main_loop_iteration orchestrates the full cycle: drain DRM page-flip events, read touch events from the capture backend, feed each through the SlotTracker, enrich TouchPhase::Start points with hit-tested key codes, send enriched events to the fusion worker, drain fusion results from Channel D, dispatch intents through the ActionDispatcher, update visual glow effects based on intent type, and render the next frame.

Sources: portal/input/src/bin/portal-keyboardd/app/run.rs#L21-L48, portal/input/src/bin/portal-keyboardd/app/run.rs#L68-L193

Context Management: Dimension and Spatial Awareness

The fusion engine maintains a cached InputContext snapshot that includes the active Dimension (Desktop/VR/AR/Tablet), focused application, spatial layout with zone definitions, and user preferences. The CachedContextProvider wraps this with rate-limited updates and a subscription mechanism — callbacks fire on DimensionChanged, FocusedAppChanged, SpatialLayoutChanged, and UserPreferencesChanged signals. Rate limits prevent context thrashing: dimension changes are rate-limited to 1s, focused app to 200ms, and spatial layout to 500ms intervals.

Sources: portal/input/src/fusion/context.rs#L11-L111, portal/input/src/fusion/context.rs#L113-L180

The context snapshot is embedded in every ClassifiedIntent via build_intent, allowing downstream consumers to reconstruct the exact state of the world at classification time. This is critical for debugging and for the layout engine to make zone-aware rendering decisions.

Sources: portal/input/src/fusion/v21_engine/engine.rs#L91-L107

Modifier System: Portal, Warp, and Vortex

Beyond standard keyboard modifiers (Shift, Ctrl, Alt, CapsLock), the Portal platform introduces three spatial modifiers mapped to evdev codes KEY_F13 (183), KEY_F14 (184), and KEY_F15 (185). These are resolved by evdev_code_to_modifier and tracked in the fusion engine’s ModifierSet. When active, they override the normal classification chain at Priority 1, transforming any touch into a spatial system action. The ModifierHold action type allows these modifiers to be held without producing uinput output — the fusion engine manages their state internally and uses them as classification context rather than forwarding them to the compositor.

Sources: portal/input/src/types/modifier.rs#L7-L111, portal/input/src/fusion/key_code_mapping.rs#L102-L110

Configuration: FusionConfig Defaults

The FusionConfig struct centralizes all tunable thresholds with production-tested defaults:

Parameter	Default	Unit	Purpose
max_touch_points	10	count	Maximum simultaneous touch slots tracked
typing_threshold_ms	50	ms	Maximum duration for typing intent
typing_displacement_threshold	0.05	normalized	Maximum displacement for typing intent (5% of screen)
pointer_drag_threshold	10.0	normalized	Minimum displacement for drag detection
fling_max_velocity	0.8	units/ms	Velocity threshold for gesture override of committed key
fling_min_displacement	0.1	normalized	Minimum displacement for fling/edge-swipe (10% of screen)
edge_zone_size	0.05	normalized	Edge detection zone size (5% of screen)
palm_rejection_enabled	true	boolean	Whether palm rejection pre-filters touches
screen_width	1920	px	Horizontal resolution for coordinate scaling
screen_height	1080	px	Vertical resolution for coordinate scaling

Sources: portal/input/src/fusion/engine.rs#L54-L94

Next Steps

Now that you understand how touch events are classified into intents and dispatched to virtual devices, the natural next steps are:

• Keyboard Daemon: DRM/KMS Direct Scanout, Cairo Rendering, and evdev Parsing — Explore how the visual keyboard surface is rendered and how hit-test results feed into the fusion engine’s slot_keys
• Virtual Input Devices: uinput Bridges to the Wayfire Compositor — Understand the VirtualKeyboard and VirtualPointer implementations that receive dispatched actions
• Spatial Domain Model: Zones, Dimensions, and Assignment Policies — Dive deeper into how zone definitions and dimensions flow into the fusion engine’s context