Flaky Test Detective
Kill intermittent test failures at the source — timing, shared state, clock/RNG, races — never with a retry band-aid.
v1.0.0 · ~544 tokens · ⬇ 0 · Updated July 6, 2026
What it does
Diagnoses a flaky test through the usual causes in order (async timing, order-dependence/shared state, time & randomness, concurrency races, external deps, float/hash equality), reproduces the flake before fixing (loop it, force the ordering), and removes the non-determinism at the source. Refuses to mask a flake with retries, longer sleeps, or skip.
Example uses
Fix a randomly failing CI test
A test passes locally but fails one CI run in five, and retrying it has quietly become the team's workaround.
The test 'sends invoice email after payment' in tests/billing.spec.ts fails roughly one CI run in five but always passes locally. Reproduce the flake first — run it in a loop, control the timing — then find the actual source of non-determinism and fix that. No retries, no longer sleeps.Expose order-dependent tests
The suite is green in serial mode but randomly red in parallel, which points to tests sharing state.
Our Jest suite passes with --runInBand but fails intermittently when tests run in parallel. We share one test database across the suite. Run the tests in random order to expose which ones leak state into each other, then isolate them so every test sets up and tears down its own data.Hunt a timezone-dependent failure
A test fails only at certain times of day or on machines in certain timezones — the classic clock non-determinism signature.
date-utils.test.ts fails in CI every night around midnight UTC and on our teammates' machines in Singapore, but never in the US during the day. Freeze the clock, pin the timezone in the test setup, and prove the fix by running the suite with faked times on both sides of the midnight boundary.Install
# 1. Create the skill folder in your Claude setup mkdir -p ~/.claude/skills/flaky-test-detective # 2. Download SKILL.md into it (or move the file you just downloaded) # → ~/.claude/skills/flaky-test-detective/SKILL.md # 3. Claude Code auto-discovers it on next launch.
Inside the skill
--- name: flaky-test-detective description: Find and fix a flaky (intermittently failing) test — the real cause, not a retry band-aid. Use when a test passes sometimes and fails others, when CI is "randomly" red, or when asked "why is this test flaky", "fix the flaky test", "stop the intermittent failures". --- # Flaky Test Detective A flaky test is a real bug — in the test, the code, or the environment. Retrying it just hides the signal. Find the source of non-determinism and remove it. ## The usual causes (check in this order) 1. **Timing / async**: a hardcoded `sleep`, or asserting before a promise/callback settles. → wait for the condition, not a fixed delay. `await` the actual thing. 2. **Order dependence / shared state**: test A leaves state (DB row, global, module cache, temp file) that test B relies on or is broken by. → each test sets up and tears down its own state; run tests in random order to expose it. 3. **Time & randomness**: `Date.now()`, `Math.random()`, timezones, DST, locale. → inject/ freeze the clock, seed the RNG, pin the locale/timezone. 4. **Concurrency / races**: parallel tests hitting the same resource; unordered async results asserted as ordered. → isolate resources per test; sort or await deterministically. 5. **External deps**: real network, real time-of-day, real filesystem. → mock the boundary; a test that needs the internet is an integration test, label it and isolate it. 6. **Floating point / hashing**: exact-equality on floats, or asserting an unordered map's order. ## Prove it before you fix it Reproduce the flake: run the test in a loop (`--repeat`/`for i in $(seq 100)`), or force the bad ordering, or freeze the timing window. A flake you can't reproduce, you haven't fixed — you've just gotten lucky. ## Fix, don't mask - Remove the non-determinism at the source. - **Never "fix" a flake with a retry, a longer sleep, or `test.skip`.** That's turning off the smoke detector. If the flake reflects a real race in the *code*, the code has the bug — fix that. ## Output Root cause (which category), the reproduction you used, the fix, and confirmation it now passes N times in a row.
Changelog
- v1.0.02026-07-03Initial clean-room write.
Frequently asked questions
Is Flaky Test Detective free?
Yes. Flaky Test Detective is free to download and MIT-licensed.
Where do I install Flaky Test Detective?
Place the SKILL.md file in ~/.claude/skills/flaky-test-detective/ and Claude Code auto-discovers it on next launch.
How many tokens does Flaky Test Detective use?
About 544 tokens — it is designed to be token-lean.
We're developers and SaaS builders who use these tools daily in production. Every review comes from hands-on experience building real products — DealPropFirm, ThePlanetIndicator, PropFirmsCodes, and many more. We don't just review tools — we build and ship with them every day.
Written and tested by developers who build with these tools daily.