Flaky Test Diagnoser
Diagnose a test that passes and fails without relevant code changes by reproducing the instability, classifying its trigger, and isolating the shared state, timing, ordering, concurrency, randomness, or environment dependency behind it. Use when CI retries hide failures, a test fails only in the suite, or a failure cannot be reproduced reliably on one machine.
npx agentscamp add skills/flaky-test-diagnoserInstall to ~/.claude/skills/flaky-test-diagnoser/SKILL.md
Diagnoses nondeterministic tests through repeated, reordered, isolated, parallel, and seeded runs. It classifies the flake, identifies the smallest condition that changes the outcome, distinguishes test defects from product races, and reports evidence plus a targeted fix and verification plan instead of treating retries as a solution.
Find the condition that changes a test outcome without guessing or accepting retries as the end state.
Workflow
- Capture the failure signature. Record the test name, assertion or exception, runner, seed, worker count, duration, environment, retry number, and nearby logs. Separate multiple signatures before investigating.
- Measure the baseline. Run the narrowest failing test repeatedly with the same seed and environment. Report run count and failure rate; do not call a test stable after one passing run.
- Vary one dimension at a time. Compare isolated versus full-suite, serial versus parallel, fixed versus random order, cold versus warm process, local versus CI-like settings, and controlled versus real time. Preserve every command and result.
- Classify the trigger. Check for leaked global state, incomplete cleanup, fixed ports, shared files or records, mutable fixtures, clock and timezone assumptions, unseeded randomness, eventual consistency, unordered collections, resource exhaustion, and true concurrency races.
- Find the minimal interference. Bisect the preceding test set or worker configuration when order matters. Identify the smallest predecessor, shared resource, timing window, or environment variable that changes the outcome.
- Distinguish harness defect from product defect. Do not add waits or mocks until deciding whether the system violates a real invariant under a valid schedule. A race revealed by a test is not automatically a flaky test bug.
- Propose the narrowest repair. Prefer deterministic inputs, explicit synchronization, unique resources, complete teardown, event-based waiting, or isolated state. Avoid arbitrary sleeps, wider timeouts, broad serialization, and permanent retries unless they are temporary containment.
- Verify statistically. Repeat the original stress conditions and the broader suite enough times to cover the observed failure rate. Report residual risk when the initial flake was too rare to establish confidence.
WARNING
A longer timeout can make a timing symptom rarer without correcting causality. Fix the state or synchronization contract that made elapsed wall time relevant.
Output
Report the failure signature, reproduction commands, run counts and failure rates, isolated trigger, evidence for test-versus-product classification, recommended fix, and post-fix stress results. If the cause remains unknown, rank the remaining hypotheses and name the next discriminating experiment.
Frequently asked questions
- Are test retries a valid fix for flaky tests?
- No. Retries can keep CI moving temporarily, but they hide the failure rate, consume time, and allow real races to survive. Track retries as debt while isolating and fixing the cause.
- How can I tell whether the test or production code is flaky?
- Change one condition at a time. If controlled time, isolation, cleanup, or deterministic data removes the failure, the harness is suspect. If valid concurrent schedules still violate a product invariant, the test may be exposing a real race.
Related
- 9 Best Claude Skills for Software TestingCompare Claude skills for regression, unit, integration, contract, property, mutation, prompt, and test-data workflows.
- Coverage Gap FinderRun the project's coverage tool and identify the highest-value untested paths — error branches, edge cases, and critical modules — then propose specific test cases for each gap. Use when you have a coverage report but don't know where new tests will pay off most.
- GitHub Actions OptimizerMake a GitHub Actions workflow faster, cheaper, and harder to attack — by profiling where wall-clock and billed minutes actually go, then adding content-keyed caching, matrix/job parallelism, run-cancellation, and path filters, and hardening the supply chain (SHA-pinned actions, least-privilege GITHUB_TOKEN, safe fork-PR handling). Use when CI is slow or queues, when a repo burns Actions minutes, or before trusting a workflow that runs on untrusted pull requests.
- Integration Test DesignerDesign integration tests that exercise components against REAL collaborators — actual database, queue, HTTP boundary — at a deliberately chosen seam, instead of a unit suite that mocks everything or a slow flaky full E2E. Use when bugs slip past green unit tests, when wiring or contracts between layers break in production, or when a mocked DB test passes but the real query/migration/serialization fails.
- Mutation Test RunnerMeasure whether a test suite actually catches bugs by running mutation testing — introduce small faults into the code and check which ones a test kills versus which slip through silently. Use when line coverage is high but bugs still ship, when you suspect tests assert weakly, or to find the exact assertions a suite is missing.
- Test ScaffolderScaffold a test file with sensible cases for a given module or function. Use when adding tests to untested code and you want a fast, structured starting point.