Test health

Flaky and failed tests collected from every Tests, Develop, Publish, and Visual nightly run, per test, across runs. Generated 2026-10-18T15:56:08.497Z. Data: summary.json, ledger.json, durations.json.

Flaky and failed tests

only tests that need a ticket

Ticket	Test	7 d	30 d	Runs (30 d)	Legs	Last seen	Last error

Where the time goes

Every Playwright E2E leg of every develop push, green or red, from the playwright-durations-* artifact each leg uploads (a pull request's stay in its own run for 3 days). A leg that runs in shards uploads one part per shard and is folded back into one leg here: its wall-clock is its longest shard, since the shards run side by side.

Wall-clock and tests per leg

Wall-clock (minutes) Tests that ran

Budget input: per leg over the window

Nearest-rank P50 and P90 per leg over the develop pushes in the window, one vote per push (a re-run attempt replaces the leg's earlier one), left out when a test failed every attempt, no test ran, a shard of the leg has no part, or the leg ran in another shape than its latest push (a failing test's time is its retries and timeouts; a flaky-only leg counts). Shape is how the leg's latest push ran: whole or in how many shards, and on how many workers; a change to either starts the budget input over. The input a later e2e budget is derived from; every plotted run is in durations.json.

Leg	Shape	Runs	Wall P50	Wall P90	Tests P50	Tests P90	Left out

The heaviest spec files

#	Spec file	Minutes, all legs	Tests	s per test per leg	Minutes per leg, heaviest first

Every test over 3 s

Test	Slowest	Legs over the line

A test needs a ticket once it flaked in + distinct runs within 30 days (the flakiness playbook's rule). flaky = failed, then passed on retry (Playwright; still fails the leg under failOnFlakyTests); failed = did not pass. Jasmine entries carry the runner's isolation verdict when their file was re-run alone. A quarantined Playwright test still runs and reports; its flaky verdict does not fail the leg until the entry's expiry (tests/AGENTS.md, Quarantine). A visual row is a screenshot capture: the file is its spec, the legs are the variants it differed on, and failed means it differed from its golden record beyond regconfig.json's tolerances. It needs a ticket once it differed on + Visual nightly runs, or on that many distinct branches (the nightly counts as develop): nothing in that tier is ever flaky, and a pull request's own intended change, recorded before approval, stays on one branch however often it is pushed. A quarantined capture is listed in visual-tests/visual-quarantine.json (visual-tests/AGENTS.md, Guardrails); an entry names the legs it covers, so a row shows its most recent entry and only the legs outside it count toward the ticket. Entries older than days drop off. Source: .github/workflows/test-health.yml.
Where the time goes: a leg's wall-clock is what the leg paid, retries included; a leg that runs in shards is one record whose wall-clock is its longest shard (what the leg costs when the shards start together), whose tests add up, and whose parts are listed under each run with the span from the first shard's start to the last shard's end when they queued (a push with a shard missing is left out of the budget and keeps the rows it had). Runs stay 30 days. The per-test rows come from each leg's latest develop push and give each test its time from a run where it passed on its first attempt: a test that failed or was flaky in the latest run keeps its time from the run before, because its own time is retries and timeouts, which the table above tracks. Only develop pushes are recorded: a pull request's leg files stay in its own run's artifacts for 3 days.