CI/CD Pipeline: Lint Enforcement, Cross-Compilation, and Release Signing

The Portal project’s CI/CD pipeline is a defense-in-depth system spanning 15 GitHub Actions workflows, 2 pre-commit hooks, and 3 release-signing scripts. Every push and pull request runs format checking, multi-tier clippy analysis, unit/integration tests, cross-compilation to ARM64, secret scanning, and undefined-behaviour detection. Tagged releases trigger a dual-signing chain (Sigstore cosign + minisign Ed25519) with CycloneDX SBOM generation, all staged and uploaded as GitHub release assets. The pipeline is designed so that no production code reaches the main branch without passing strict lint, test, security, and compilation gates, and no binary ships without a verifiable cryptographic provenance chain.

Sources: .github/workflows/check.yml#L1-L87, .github/workflows/release.yml#L1-L136, .github/workflows/test.yml#L1-L26

Pipeline Architecture Overview

The CI/CD system is organized into five functional layers, each triggered by specific GitHub events. Understanding which workflows fire on which events — and what gates they enforce — is essential before modifying any pipeline behavior.

flowchart TB
    subgraph Trigger["GitHub Events"]
        PR["Push / Pull Request"]
        Tag["Tag Push (v*)"]
        Main["Push to main"]
        Rel["Release Published"]
        Cron["Schedule (daily/weekly)"]
    end

    subgraph Quality["Quality Gates (every push/PR)"]
        Check["Check Workflow"]
        Test["Test Workflow"]
        Cross["Cross-Compile"]
        Miri["Miri UB Scan"]
        Semver["Semver Checks"]
    end

    subgraph Security["Security Scanning"]
        GL["Gitleaks"]
        TH["TruffleHog PR Scan"]
        Audit["cargo-audit"]
        Deny["cargo-deny"]
        THDaily["TruffleHog Daily Subsystem"]
        THWeekly["TruffleHog Weekly Full"]
    end

    subgraph Release["Release Pipeline (tag push)"]
        Build["Cross Build ARM64"]
        SBOM["CycloneDX SBOM"]
        Cosign["Cosign Keyless OIDC"]
        Mini["Minisign Ed25519"]
        Verify["Round-trip Verify"]
        Upload["GitHub Release"]
    end

    subgraph Guard["Branch Protection"]
        GuardMain["Guard Main"]
        NotifyRel["Notify Downstream"]
        SyncPR["Sync Mirror Issues"]
    end

    PR --> Check & Test & Cross & Miri & Semver & GL & TH
    Main --> Audit & Deny & GL & TH
    Tag --> Build
    Build --> SBOM --> Cosign
    Cosign --> Mini --> Verify --> Upload
    Main --> GuardMain
    Rel --> NotifyRel
    PR --> SyncPR
    Cron --> THDaily & THWeekly & Audit

Sources: .github/workflows/ directory

Lint Enforcement: Multi-Tier Clippy and Format Gates

Workflow Trigger and Structure

The Check workflow runs on every push and pull request, executing five parallel jobs: fmt, clippy, docs, coverage, and clippy-arm64. Each job checks out the repository with persist-credentials: false (a security best practice that prevents credential exfiltration from within workflow steps) and pins all third-party actions by SHA hash rather than version tag to prevent supply-chain attacks via tag re-pointing.

Sources: .github/workflows/check.yml#L1-L16

Two-Phase Clippy: Production-Strict, Test-Permissive

The most architecturally significant decision in the lint pipeline is the split enforcement model: production code (--lib --bins) is held to a zero-tolerance standard, while test code receives targeted allowances that are documented and reviewed. The production clippy command denies all warnings:

cargo clippy --lib --bins --workspace --all-features -- -D warnings

The test clippy command denies warnings but selectively allows lints that are intentionally permissive in test contexts — unwrap_used, expect_used, as_conversions, and several style lints. The workflow file itself contains an 18-line justification block explaining each allowance. For example, clippy::unwrap_used is allowed in tests because “tests SHOULD panic on setup failure (that is the test signal)” — forcing ? would either require Result in every test function or push the panic into a helper, hiding the failure site.

Sources: .github/workflows/check.yml#L28-L46

Workspace-Wide Lint Configuration

The actual lint rules are defined in the root Cargo.toml under [workspace.lints.clippy], not in the workflow file. This ensures that cargo clippy behaves identically whether run locally or in CI. The configuration follows a priority-based layering system where lint groups are assigned negative priorities to control ordering:

Lint Group	Level	Priority	Rationale
`correctness`	deny	-3	Logic bugs — zero tolerance
`suspicious`	deny	-3	Probable bugs — zero tolerance
`style`	deny	-2	Promoted from warn in Wave 7 T59
`complexity`	deny	-2	Promoted from warn in Wave 7 T59
`perf`	deny	-2	Promoted from warn in Wave 7 T59
`pedantic`	deny	-1	Promoted from warn in Wave 7 T59
`unwrap_used`	deny	default	Prevents panic regressions (W12 T150)
`expect_used`	deny	default	Same rationale
`panic_in_result_fn`	deny	default	Same rationale
`as_conversions`	deny	default	Forces explicit review of FFI casts
`await_holding_lock`	deny	default	Deadlock risk — always a bug

Critically, as_conversions is set to deny at the workspace level so that every new integer or pointer cast surfaces in code review. FFI-heavy crates (npu-runtime, spatial, input, warp, voice, etc.) opt out per-crate via [lints.clippy] as_conversions = "allow" because as is the canonical way to interact with C ABIs and raw pointers. The per-crate opt-outs include TODO comments referencing the workspace-level policy, ensuring the long-term cast-purge plan stays visible.

Sources: Cargo.toml#L160-L250, clippy.toml#L1-L26

Clippy Configuration File

The clippy.toml file provides additional metadata that clippy uses to calibrate lint thresholds. It sets msrv = "1.97.0" (matching rust-toolchain.toml exactly) so that edition-aware lints like manual_strip and uninlined_format_args fire correctly. It also caps excessive-nesting-threshold = 250 and too-many-lines-threshold = 250, aligning with the file-splitting work that ensures no source file exceeds 250 pure lines of code.

Sources: clippy.toml#L1-L26, rust-toolchain.toml#L1-L4

Documentation and Coverage Gates

Two additional quality jobs complement the clippy gates:

The docs job runs cargo doc --no-deps --workspace --all-features with RUSTDOCFLAGS: -D warnings, which promotes broken intra-doc links, missing examples, and rustdoc parse errors to build failures. This is reinforced by the workspace-level rustdoc configuration that denies broken_intra_doc_links.

The coverage job uses cargo-llvm-cov (which replaced tarpaulin after it pulled three RUSTSEC advisories) with --fail-under-lines 85. The 85% threshold is an explicitly documented floor — the remaining ~15% gap lives in QNN/ONNX FFI crates whose C calls defeat llvm-cov’s instrumentation. The workflow comment notes: “Raise only after FFI coverage exclusion is wired into llvm-cov.toml.”

Sources: .github/workflows/check.yml#L48-L75, Cargo.toml#L265-L271

ARM64 Cross-Clippy (Non-Blocking)

The clippy-arm64 job runs clippy against the aarch64-unknown-linux-gnu target using the cross toolchain. It is marked continue-on-error: true, meaning failures produce warnings without blocking merges. This is an advisory gate while the ARM64-specific code paths stabilize.

Sources: .github/workflows/check.yml#L77-L87

Cross-Compilation: ARM64 Production Builds

The Cross-Compile Workflow

Every push and pull request triggers the Cross-Compile workflow, which builds the entire workspace for aarch64-unknown-linux-gnu (the HP Spaceboard’s target architecture). The job installs the cross tool (pinned to git tag v0.2.5) and runs:

cross build --target aarch64-unknown-linux-gnu --release --workspace
cross test  --target aarch64-unknown-linux-gnu --workspace --release || true

Tests run with || true because some test suites require physical hardware (Hexagon DSP, Wayland compositor, DRM devices) unavailable on CI runners. The build step itself is strict — any compilation failure blocks the merge.

Sources: .github/workflows/cross-compile.yml#L1-L28

Cross-Compilation Configuration

Two configuration files govern the cross-compilation environment. Cross.toml defines Docker images for each target triple and specifies which environment variables pass through into the container:

File	Purpose
`Cross.toml`	Docker image selection, `RUSTFLAGS`/`PKG_CONFIG_*` passthrough
`.cargo/config.toml`	Native linker override (`aarch64-linux-gnu-gcc`)

The Cross.toml passthrough list includes RUSTFLAGS, PKG_CONFIG_PATH, and PKG_CONFIG_LIBDIR — these are essential for the container to find the ARM64 cross-sysroot libraries that Wayland, Cairo, and PipeWire need during linking.

Sources: Cross.toml#L1-L13, .cargo/config.toml#L1-L17

Artifact Upload for CI Builds

On every successful cross-compile, five key artifacts are uploaded for inspection:

Artifact	Origin Crate	Purpose
`libportal_spatial.so`	portal-spatial	Wayfire plugin (spatial layout engine)
`portal-wm`	portal-wm	Window manager daemon
`portal-voiced`	portal-voice	Voice pipeline (VAD/STT/TTS)
`portal-llmd`	portal-llm	On-device LLM daemon
`portal-keyboardd`	portal-input	DRM/KMS virtual keyboard

These artifacts allow developers to pull pre-built ARM64 binaries for deployment testing without running a local cross-compile.

Sources: .github/workflows/cross-compile.yml#L19-L28

Release Signing: Dual Cryptographic Chain

Release Trigger and Build

The Release workflow triggers exclusively on tag pushes matching v* (e.g., v0.2.1). It requests contents: write (to create the GitHub release) and id-token: write (required for Sigstore’s keyless OIDC signing flow). The build step cross-compiles the production binaries for ARM64 using the same cross toolchain as CI:

cross build --target aarch64-unknown-linux-gnu --release

Sources: .github/workflows/release.yml#L1-L24

SBOM Generation (CycloneDX)

After building, the workflow generates Software Bill of Materials in both JSON and XML formats using cargo-cyclonedx 0.5.9:

cargo cyclonedx --format json --describe crate --spec-version 1.5
cargo cyclonedx --format xml  --describe crate --spec-version 1.5

The --describe crate flag produces one SBOM per workspace package. The generated .cdx.json and .cdx.xml files are collected from each crate directory into a staging directory using find ... -exec cp. The workflow comment explicitly notes that SPDX format is not produced by cargo-cyclonedx 0.5.x — downstream consumers needing SPDX must use syft separately.

Sources: .github/workflows/release.yml#L26-L39, scripts/generate_sbom.sh#L1-L127

Layer 1: Cosign Keyless Signing (Always Runs)

The first signing layer uses Sigstore cosign with GitHub OIDC for keyless Ed25519 signatures. For each of the four production binaries, the workflow writes three files:

File Extension	Content
`.sig`	Raw Ed25519 signature
`.pem`	Signing certificate (Fulcio CA)
`.bundle`	Rekor transparency log entry (DSSE format)

Consumers can verify any binary with:

cosign verify-blob \
  --certificate <bin>.pem \
  --signature <bin>.sig \
  --bundle <bin>.bundle \
  <bin>

The cosign layer always succeeds because it requires no pre-shared secrets — the signing identity is derived from the GitHub Actions OIDC token, which is attested to the Fulcio certificate authority and logged in the Rekor transparency log.

Sources: .github/workflows/release.yml#L41-L63

Layer 2: Minisign Ed25519 Signing (Conditional)

The second signing layer uses minisign with an offline-generated Ed25519 keypair. This layer is conditional: it only runs when the MINISIGN_SECRET_KEY GitHub Secret is populated. If the secret is empty (as it is currently, since the production keypair has not yet been generated), the step outputs skip_verify=true and the cosign signatures alone provide the trust chain.

When the secret is present, the workflow:

Decodes the base64-encoded secret key to a runner-temp file
Calls scripts/sign_release.sh to sign all staged artifacts
Deletes the secret key file immediately after signing
Runs scripts/verify_release.sh as a round-trip verification to catch any signing failure before artifacts are published

Sources: .github/workflows/release.yml#L65-L107

The `sign_release.sh` Script

The signing script is the single source of truth for what gets signed in both CI and offline builds. It signs five production binaries and any *.cdx.json/*.cdx.xml SBOM files found in the release directory:

Binary	Source	Build Method
`portal-stream`	portal/streaming	C (Makefile)
`portal-voiced`	portal-voice	Rust (cargo)
`portal-llmd`	portal-llm	Rust (–features hexagon)
`portal-keyboardd`	portal-input	Rust (–features keyboardd)
`portal-wm`	portal-wm	Rust (cargo)

The script supports --dry-run for CI staging checks, feeds the password via stdin when MINISIGN_PASSWORD is set (avoiding interactive prompts), and falls back to terminal-based interactive entry for offline builds. Each signed artifact produces a <file>.minisig sidecar.

Sources: scripts/sign_release.sh#L1-L168, scripts/cosign-sign.sh#L1-L167

The `verify_release.sh` Script

The verification script is the CI gate that prevents unsigned or tampered artifacts from being published. It verifies every artifact in the release directory against the public key at keys/portal-release.pub, producing a per-artifact OK/FAIL report and exiting non-zero if any signature fails. This script is also usable by end-users to verify downloaded releases locally.

Sources: scripts/verify_release.sh#L1-L143

Release Upload

The final step uses softprops/action-gh-release (pinned by SHA) to upload all staged assets to the GitHub release. Each binary ships with up to four signature files (cosign .sig/.pem/.bundle + minisign .minisig), plus CycloneDX SBOMs in both JSON and XML with their own minisign signatures. This gives consumers multiple independent verification paths.

Sources: .github/workflows/release.yml#L109-L136

Key Management Policy

The keys/ directory contains only public key material — secret keys are explicitly prohibited from git. The current portal-release.pub is a placeholder (a sentinel value that will fail verification) until the release engineer generates the production keypair on an offline machine. The key policy mandates:

Item	Rule
Public keys (`*.pub`)	Safe to commit, tracked in git
Secret keys (`.sec`, `.key`)	Must never be committed, stored offline
Key password	Must never be hardcoded in scripts or env files
Key algorithm	minisign Ed25519
Key rotation	On compromise or annually, whichever is sooner

The first release that ships with the production minisign key will be cross-signed with cosign keyless signatures (GitHub OIDC). After trust in the minisign key is established, cosign may be retired in a future release.

Sources: keys/README.md#L1-L60, keys/portal-release.pub#L1-L25

Security Scanning: Multi-Layer Defense

Secret Detection: Three-Tier Approach

The project implements layered Data Loss Prevention (DLP) for secret detection across three tiers, each with different trade-offs between speed, false-positive rate, and coverage:

Tier	Tool	Trigger	Coverage	Speed
Pre-commit	Gitleaks + TruffleHog	`git commit`	Staged changes only	Seconds
CI (every PR)	Gitleaks + TruffleHog	PR + push to main	Full diff	Minutes
CI (scheduled)	TruffleHog (subsystem + full)	Daily/weekly cron	Entire git history	Minutes

The pre-commit configuration runs Gitleaks for fast pattern-matching (catches common API keys, tokens, passwords) and TruffleHog for deep verification (only reports secrets confirmed active against live services). Both hooks run in parallel. If either finds a secret, the commit is blocked before it enters history.

Sources: .pre-commit-config.yaml#L1-L35

Gitleaks Configuration

The .gitleaks.toml file extends the default rule set with six project-specific custom rules tuned for the Portal’s threat model:

Rule ID	What It Detects	Rationale
`wayland-socket-path`	Hardcoded `/run/portal/*.sock` paths	Should use runtime discovery
`orange-pi-ssh-key`	Private key PEM blocks	Deployment keys must never be committed
`streaming-endpoint`	Internal IP:port combinations	Deployment targets should be configurable
`drm-device-path`	Hardcoded `/dev/dri/*` paths	Should use libdrm at runtime
`github-token-extended`	GitHub PAT/OAuth tokens	Tokens must use secrets manager
`ip-address-high-entropy`	RFC 1918 IP:port in source	Possible hardcoded deployment target

Each rule includes targeted allowlists (e.g., test fixtures, lock files, compositor configs) to suppress false positives while maintaining strict coverage on source files.

Sources: .gitleaks.toml#L1-L170

TruffleHog: CI and Scheduled Scans

Three TruffleHog workflows provide progressive coverage. The PR scan (trufflehog-scan.yml) runs on every pull request and push to main, scanning only the diff since the base commit (--since-commit). The daily subsystem scan (trufflehog-daily-subsystem.yml) uses a matrix strategy to scan each of the 18 subsystem directories independently with --include-paths filters, running at 06:00 UTC daily. The weekly full repo scan (trufflehog-weekly-repo.yml) scans the entire repository history at 05:00 UTC every Monday.

Both scheduled scans automatically create GitHub issues tagged security on failure, with step-by-step remediation instructions (review logs, rotate credentials, remove from history, close issue). The issue creation logic checks for existing open issues to avoid duplicates.

Sources: .github/workflows/trufflehog-scan.yml#L1-L25, .github/workflows/trufflehog-daily-subsystem.yml#L1-L119, .github/workflows/trufflehog-weekly-repo.yml#L1-L64

cargo-audit and cargo-deny

The Security workflow runs two Rust-specific supply-chain checks. cargo-audit queries the RustSec advisory database for known vulnerabilities in workspace dependencies, running on every push to main, every PR, and daily via cron schedule. cargo-deny enforces four categories of policy:

Category	What It Enforces
`advisories`	No dependencies with unaddressed RUSTSEC advisories
`licenses`	Only allowlisted license types (MIT, Apache-2.0, BSD, ISC, Zlib, MPL-2.0, etc.)
`bans`	Deny multiple versions of the same crate (with documented skip exceptions)
`sources`	Deny crates from unknown registries or git sources

The deny.toml configuration file is extensively documented. Each ignored advisory carries a reason field explaining why it’s suppressed and when it was last verified (all re-verified 2026-07-20/23). Each bans.skip entry groups duplicates by ecosystem cause — for example, “Group A” covers Windows-only target crates irrelevant to ARM64 deployment, “Group B” covers the rand v0.8/v0.9 ecosystem split, and “Group D” covers the syn v2/v3 proc-macro migration.

Sources: .github/workflows/security.yml#L1-L45, deny.toml#L1-L382

Branch Protection: Software-Enforced Guard Main

Because the repository runs on GitHub Free (which does not support native branch protection for private repos), the Guard Main workflow provides software-enforced branch protection on the x-elite-spaceboard branch. It triggers on every push to main and implements a three-stage verification pipeline:

flowchart LR
    Push["Push to main"] --> Classify{"Merge commit?"}
    Classify -->|Yes| CheckApproval["Verify PR approved"]
    Classify -->|No| Revert["Auto-revert commit"]
    CheckApproval --> CheckCI["Verify CI passed"]
    CheckCI -->|Failed| Issue["Create security issue"]
    CheckCI -->|Passed| OK["Allow"]
    CheckApproval -->|Not approved| Issue
    Revert --> Issue

The guard distinguishes between merge commits (PR merges) and direct pushes. For direct pushes, it automatically reverts the commit using git revert HEAD --no-edit && git push. For PR merges, it verifies that the PR had at least one approval and that all CI check suites (excluding Guard Main itself) passed. A single-developer allowance lets the repo owner self-merge without external approval.

Infinite revert loops are prevented by skipping commits authored by github-actions[bot] and commits whose message starts with Revert ". Violations create GitHub issues tagged security-violation with the full audit trail (commit SHA, author, message, reason, action taken).

Sources: .github/workflows/guard-main.yml#L1-L227

Testing and Undefined Behaviour Detection

Unit and Integration Tests

The Test workflow runs on every push and PR, executing three test tiers:

Tier	Command	Status
Library tests	`cargo test --lib --workspace --all-features`	Blocking
Integration tests	`cargo test --test '*' --workspace --all-features`	Blocking
Doc tests	`cargo test --doc --workspace`	Non-blocking (TODO)

All tests compile with --all-features to ensure feature-gated code paths are exercised. Doc tests use continue-on-error: true because some intra-doc examples and link targets are still being resolved.

Sources: .github/workflows/test.yml#L1-L26

Miri: Undefined Behaviour Detection

The Miri (UB scan) workflow runs on every push and PR, targeting the FFI-heavy crates where unsafe Rust interacts with C code, raw pointers, and hardware APIs. Miri executes under strict provenance checking (-Zmiri-strict-provenance), symbolic alignment checking (-Zmiri-symbolic-alignment-check), and disabled isolation (-Zmiri-disable-isolation).

The workflow comment documents a baseline established 2026-07-23: portal-npu-runtime passes 13 unit tests with zero failures and zero ignores. The portal-spatial (95 passed) and portal-input (471 passed) crates are documented as Miri-clean but excluded from CI due to environmental constraints — Miri only supports AF_INET/AF_INET6 sockets (not the AF_UNIX IPC that spatial tests use) and the wayland-sys build script requires dev libraries unavailable on ubuntu-latest.

Sources: .github/workflows/miri.yml#L1-L51

Semver Compliance Checks

The Semver Checks workflow runs cargo-semver-checks on every push and PR to detect breaking API changes across versioned workspace crates. Currently marked continue-on-error: true (non-blocking advisory) while the baseline is being established. Once the workspace passes cleanly, the advisory flag will be removed to make semver violations merge-blocking.

Sources: .github/workflows/semver.yml#L1-L24

Dependency Management

Dependabot watches three package ecosystems on a weekly schedule (Monday), automatically opening pull requests for version updates:

Ecosystem	PR Limit	Label Prefix
GitHub Actions	5	`[build/ci] chore:`
Rust/Cargo	10	`[common] chore:`
Docker (ARM64 sandbox)	3	`[build/ci] chore:`

All Dependabot PRs are labeled with dependencies plus the ecosystem-specific label, making them easy to triage and bulk-merge.

Sources: .github/dependabot.yml#L1-L51

Cross-Repository Integration

Two workflows integrate with the downstream demain-deenson/portal repository:

The Notify Release workflow fires when a GitHub release is published, sending a repository_dispatch event with the tag name, release name, body, and URL to the downstream repo. This triggers the downstream deployment pipeline automatically.

The Sync to demain-deenson Board workflow fires on PR open/close, extracts linked mirror issue numbers from the PR body (via closes #N / fixes #N / resolves #N patterns), and dispatches PR events to the downstream repository for each linked mirror issue. This keeps the issue tracker in sync when PRs are merged.

Sources: .github/workflows/notify-release.yml#L1-L34, .github/workflows/sync-to-demain.yml#L1-L90

Workflow Reference Table

Workflow	Trigger	Blocking?	Purpose
`check.yml`	push, PR	Yes	fmt, clippy (2-tier), docs, coverage, ARM64 clippy
`test.yml`	push, PR	Yes (lib+test)	Unit, integration, doc tests
`cross-compile.yml`	push, PR	Yes (build)	ARM64 release build + artifact upload
`miri.yml`	push, PR	Yes	Undefined behaviour scan (FFI crates)
`semver.yml`	push, PR	No (advisory)	API compatibility regression detection
`security.yml`	push to main, PR, daily	Yes	cargo-audit + cargo-deny + SBOM
`gitleaks-scan.yml`	push to main, PR	Yes	Fast pattern-based secret detection
`trufflehog-scan.yml`	push to main, PR	Yes	Verified secret detection (diff-scoped)
`trufflehog-daily-subsystem.yml`	daily cron	Yes	Per-subsystem full-history secret scan
`trufflehog-weekly-repo.yml`	weekly cron	Yes	Full-repo full-history secret scan
`release.yml`	tag `v*`	N/A	Cross-build, SBOM, dual-sign, GitHub release
`guard-main.yml`	push to main	Enforcement	Branch protection, auto-revert violations
`notify-release.yml`	release published	N/A	Dispatch to downstream repo
`sync-to-demain.yml`	PR open/close	N/A	Mirror issue synchronization
`security.yml` (SBOM job)	tag `v*`	N/A	CycloneDX SBOM artifact upload