A page can show a drone online while it is offline, keep a frozen video labelled “live”, or lose an operator's action after a reconnect. Argus drives the real product in a real browser, injects changing conditions, and decides bug‑vs‑noise by comparing what the screen says with the simulator's ground truth — so it finds real issues with almost no false positives.
Vision‑only testers guess, and guesses produce false positives. Argus does the opposite: it reads what the operator sees (labels, telemetry, the video element, map markers) and compares it against what the system actually knows.
Every product in this challenge sits on the same simulator and control API, so Argus can always obtain independent truth: the true drone status and position, the backend's health, the active faults, and a socket feed subscribed exactly like the cockpit. When the two disagree in a way that harms the user, that's a bug — proven, not guessed.
# the oracle, in one line
UI: "live", distance 75 m, badge connected
TRUTH: stream frozen, drone at 180 m,
simulator disconnected
──────────────────────────────
VERDICT: BUG — stale data shown as live
GROUPED: 3 symptoms · 1 root causeFreshness, video liveness, map position, multi‑user propagation and recovery are all checked this way.
Bugs are decided by comparing the screen with the control API and socket feed. No model opinion in the verdict path → near‑zero false positives.
To check a marker's position, Argus finds the live Cesium viewer through the page's React fiber and reads each entity's real lat/lon and screen box — exact, no screenshot guessing.
Liveness comes from the <video> element's decoded‑frame count and a 16×9 luminance hash — so a frozen stream still labelled “live”, or a label that switched while the picture didn't, is caught.
A HUD overlay is burned into every video showing the step, the UI's claim, the ground truth and the verdict — the video is the reproducible evidence, produced automatically.
One fault's many symptoms merge into a single issue. Guards prevent false alarms — e.g. a control is scrolled into its own container before it can be called “unreachable”.
The same self‑healing element matching used across the project reads the UI by meaning (role, accessible name, context), so small markup changes don't break the probes.
# run one scenario live (a real Chrome opens)
argus live run --only S2
# run the full suite, record every scenario
argus live run
# generate the submission document from the runs
argus live reportEach run writes runs/live/<id>/ — an MP4, screenshots, timed samples and a machine‑readable result.json — then rolls them into EVALUATION.md.
The broader Argus CLI (used elsewhere in the project) also ships:
argus run replay + self-heal + triage
argus explore crawl an app into a map
argus generate write a suite (1 LLM call)
argus diff two builds, no changelog
argus gauntlet mutation benchmark
argus doctor preflight checksArgus is also an MCP server, so a coding agent (Claude Code, Cursor, opencode…) can test its own work in a real browser before opening a PR.
# register once
claude mcp add argus -- argus mcpThe agent writes a feature, then calls verify_change("what I intended"). Argus runs the browser suite and answers BUG with repro steps or INTENDED CHANGE with the release‑note citation — the intent it was given is the oracle.
| MCP tool | what it does |
|---|---|
| run_tests | run the full suite, return verdicts |
| verify_change | test a change against your intent |
| explore_and_generate | build a suite for an app with none |
| last_report | verdicts of the most recent run |
| Component | Role in Argus | From the brief's list |
|---|---|---|
| Playwright | Drives real Chrome — multi‑user contexts, device emulation, offline/throttle, video recording, traces | ✓ recommended |
| Chrome DevTools Protocol | Performance, long‑tasks and JS‑heap metrics for the load / long‑running scenarios | ✓ recommended |
| Cesium | The cockpit's map — Argus reads its live entities for exact marker positions (does not modify it) | starter kit |
| MCP (Model Context Protocol) | Exposes Argus to coding agents as tools | ✓ recommended |
| python‑socketio | Subscribes to the backend feed exactly like the cockpit, for ground‑truth freshness | — |
| OpenCV · imagehash | Perceptual video/frame hashing and the vision tier for canvas surfaces | — |
| FastAPI · pydantic · rapidfuzz | Report/live console, typed contracts, self‑healing similarity scoring | — |
| imageio‑ffmpeg | Stitches multi‑operator recordings side‑by‑side into one evidence video | — |
| OpenRouter · Groq (free tiers) | LLM mesh for the rare reasoning calls (generation, ambiguous triage) — layered with fail‑fast fallback | ✓ recommended |
The runtime verdict path is deterministic. Models are used only where reasoning genuinely helps (writing a suite, judging a truly ambiguous change), never to decide a live bug.
| # | Scenario | Level | Category | Verdict |
|---|---|---|---|---|
| S1 | Stale / delayed / offline data shown as live | L2 | Telemetry / freshness | BUG |
| S2 | Frozen drone video still labelled “live” | L2 | Video & media | BUG |
| S3 | Battery / altitude on a phone | L1 | Responsive UI | PASS |
| S4 | Video tile covers the 2D/3D map switch on a phone | L1 | Responsive UI | BUG |
| S5 | Drone & dock labels overlap on the map | L1 | Map / visual | BUG |
| S6 | Connection loss & recovery | L3 | Network & recovery | PASS |
| S7 | Two operators see the same take‑off | L3 | Real‑time / multi‑user | PASS |
| S8 | Selecting a drone switches telemetry & video together | L2 | Functional / video | PASS |
| S9 | Flight commands need no sign‑in; open CORS | L1 | Security & permissions | BUG |
| S10 | Responsiveness with 16 drones, ~10 sim‑minutes | L3 | Performance / long‑running | PASS |
A clean PASS is a result, not a miss: it shows the tester avoids false positives — a scored criterion.