Test harness (prototype)

This is the one part of DoesItARM that is built, not just specced: a zero-dependency Swift CLI (apps/doesitarm-harness) that takes a Mac app from “we have a copy” to “here’s the verdict, with evidence.” Everything below reflects what runs today; planned pieces are marked.

Where do the apps run? In a disposable VM by default.

harness test defaults to --driver vm — a disposable macOS VM (tart). The untrusted app under test never runs on the operator’s real machine. Running on the bare host (real Metal, but real prefs/files at risk) is a deliberate, human-approved opt-in (--driver host --allow-host). The VM driver is implemented in code — clone → boot headless → push the harness in over SSH → run the pipeline inside the guest → import evidence back → tear down — but an end-to-end boot needs tart + a base image on the host (the same “needs the real machine” shape as the Screen-Recording grant); when either is absent the driver degrades gracefully to a read-only static-arch read (never a false negative). The VM’s GPU is paravirtual (Virtualization.framework, feature-capped), so real GPU/translation fidelity still comes only from the host or bare-metal — see the where-it-runs options.

System architecture — current vs planned

flowchart TB
  ORC["Harness CLI (orchestrator)<br/>swift run harness · on the operator's Mac"]
  subgraph VM["macOS VM · tart / Virtualization.framework — DEFAULT (--driver vm)"]
    H["harness (in guest)"]
    subgraph DRV["Driver seams (swappable)"]
      UI["UIDriver<br/>ax ✓"]
      REC["EvidenceRecorder<br/>screencapturekit ✓ · screencapture · ffmpeg · none"]
      ISO["IsolationBackend<br/>host ✓ · user"]
    end
    VAPP["App under test<br/>VLC · HandBrake · Audacity"]
    H --> DRV
    DRV --> VAPP
  end
  subgraph HOSTRUN["bare host · real Metal — GATED (--driver host --allow-host)"]
    HAPP["App under test on the real machine"]
  end
  ORC == "--driver vm (default): clone · boot · push harness · run in guest · import evidence" ==> VM
  ORC -. "--driver host (needs human approval)" .-> HOSTRUN
  classDef gated stroke-dasharray:6 4,opacity:0.6;
  class HOSTRUN,HAPP gated;

Build status

Capability	Status
Static arch / Rosetta classification (lipo/otool)	✅ built
Bulk scan of all installed apps (no permissions)	✅ built — 113 apps classified
Acquire via Homebrew cask + locate	✅ built
Launch + window-ready + crash detection	✅ built
Drive UI via Accessibility (menus, buttons, files)	✅ built (needs Accessibility grant)
Audio Unit lane via auval (zero-UI)	✅ built
Record video + screenshots	✅ built — needs Screen Recording grant to capture
Local store (SQLite = D1 mirror, r2/ = R2 mirror)	✅ built
Swappable run driver, recorder + isolation seams	✅ built (vm/host drivers, capture backends, host isolation); user isolation = stub
VM run driver (--driver vm, the default)	✅ built in code — clone/boot/push/run-in-guest/import/teardown; end-to-end boot needs tart + a base image on the host
Host execution gated behind human approval (--driver host --allow-host)	✅ built — refuses without consent
Live D1/R2 sync	◌ dry-run only

Test run pipeline

One invocation of harness test <app> runs this, start to finish — inside the disposable VM guest by default (--driver vm), or on the bare host when explicitly approved (--driver host --allow-host):

flowchart LR
  A["acquire<br/>brew cask / locate"] --> B["static arch<br/>lipo · otool"]
  B --> C["isolation.prepare<br/>host: clean state"]
  C --> D["launch<br/>NSWorkspace"]
  D --> E["record<br/>ScreenCaptureKit"]
  E --> F["drive profile<br/>Accessibility API"]
  F --> G["evidence<br/>video · shots · crash · logs"]
  G --> H["classify<br/>passed / native …"]
  H --> I[("store<br/>SQLite + r2/")]
  I -. "sync (later)" .-> J["D1 + R2"]

The driver pattern

Four seams are pluggable at runtime, so we can A/B the tooling — or run permission-free — with a flag. ✓ = working, (gated) = built but needs human approval, (stub) = wired seam, not implemented. The headline seam is RunDriver (--driver): which machine the test runs on.

flowchart TB
  E["AutomationEngine"]
  E --> D{{"RunDriver (where it runs)"}}
  E --> U{{"UIDriver"}}
  E --> R{{"EvidenceRecorder"}}
  E --> S{{"IsolationBackend (state hygiene)"}}
  D --> d1["vm ✓ (default)"]
  D -.-> d2["host ✓ (needs --allow-host)"]
  U --> u1["ax ✓"]
  U -.-> u2["cgevent (stub)"]
  U -.-> u3["xcuitest (stub)"]
  R --> r1["screencapturekit ✓"]
  R --> r2["screencapture ✓"]
  R --> r3["ffmpeg ✓"]
  R --> r4["none ✓"]
  S --> s1["host ✓"]
  S -.-> s3["user account (stub)"]

--driver (which machine) and --isolation (state hygiene once on a machine) are different axes. The VM driver runs the in-guest harness with --isolation host because a disposable VM is the isolation.

Where-it-runs options

--driver / lane	GPU	Isolation	Approval	Status
vm (tart macOS VM) (default)	⚠️ paravirtual (capped)	full clean-room, disposable	none — safe	built; end-to-end boot needs tart + base image
host (bare host)	✅ real Metal	prefs/containers reset	human (--allow-host)	built — refuses without consent
host + --isolation user	✅ real Metal	separate account state	human	stub
bare-metal worker (prod)	✅✅ full	one app per box	fleet policy	planned

Data model → D1 + R2

Results map 1:1 onto the data model. The local store is built so “connect Cloudflare later” is a flat upload: SQLite is D1, and r2/ file paths are R2 object keys.

flowchart LR
  subgraph M["Domain model"]
    T["Title"] --> SG["Signal"]
    TR["TestRun"] --> SG
    SG --> V["Verdict"]
  end
  subgraph L["Local store · .darm-data"]
    DB[("SQLite<br/>doesitarm.db")]
    FS["r2/ mirror<br/>recordings · logs · diagnostics"]
  end
  M --> DB
  TR -. "evidence keys" .-> FS
  DB -.->|"wrangler d1 execute"| D1[("Cloudflare D1")]
  FS -.->|"wrangler r2 object put"| CR[("Cloudflare R2")]

Component tree

mindmap
  root(("doesitarm-harness"))
    CLI
      setup
      arch
      scan
      test
      plugins
      sync
    Core["CompatHarnessCore"]
      Acquirer
      ArchInspect
      Launcher
      EvidenceRecorder
      IsolationBackend
      Accessibility
      Profiles["Base + VLC/HandBrake/Audacity"]
      Storage["Storage + Schema"]
      VerdictResolver
      AutomationEngine
    Store[".darm-data"]
      SQLite["SQLite (D1 mirror)"]
      r2["r2/ evidence (R2 mirror)"]

What works without permissions

macOS gates UI driving (Accessibility) and capture (Screen Recording) behind TCC grants that can’t be set from the CLI. The arch/scan/plugin lanes need neither, so we collect real data today regardless.

flowchart TB
  subgraph FREE["Works now — no permissions"]
    sc["scan · 113 apps classified"]
    pl["plugins · auval"]
    ar["arch · static Mach-O"]
  end
  subgraph AXG["Accessibility ✓ granted"]
    dr["drive UI · menus · buttons"]
  end
  subgraph SRG["Screen Recording ✗ NOT granted"]
    vid["video clips"]
    shot["screenshots"]
  end
  classDef ok fill:#0f3d2e,stroke:#5ee0a8,color:#dffaf0;
  classDef warn fill:#3a2f12,stroke:#e5c463,color:#fff7e0;
  class FREE,sc,pl,ar,AXG,dr ok;
  class SRG,vid,shot warn;

Planned: beta-OS coverage + a driverless printer lane

Three near-term extensions, from the timeline and data-sources memos:

• macOS 27 “Golden Gate” beta data. To classify apps against the next OS before GA, the harness needs a beta-OS VM. tart create --from-ipsw is blocked by a confirmed Apple Virtualization.framework bug, so the working path is to boot a Tahoe (macOS 26) VM image and upgrade to 27 in-guest via OTA (e.g. the MacStadium Orka Tahoe image, which has a recovery partition). Verify store-distributed apps actually install in the VM before trusting it as a data source.
• A driverless printer/peripheral lane. macOS printing is a software pipeline you can exercise without the physical device — a three-tier check on one Mac mini: lipo static inspection → cupsfilter -e PASS/FAIL → ippeveprinter/ipptool.
• Acquisition guardrail. Don’t build acquisition on disposable Mac App Store VMs (sign-in fails, re-auth per clone, device caps + rate limits, account-termination risk); lead with Homebrew casks + KVR plugins.

Running it

cd apps/doesitarm-harness
swift build
swift run harness scan                  # classify every installed app (no permissions)
swift run harness test vlc              # acquire, launch, drive, record, store
swift run harness test vlc --recorder ffmpeg --isolation host   # swap backends

The full options matrix and decisions live in the repo at apps/doesitarm-harness/docs/driver-architecture-and-tooling-options.md.