A test fails, someone reruns it, it passes, and the failure gets filed under "flaky" without anyone checking whether that's true, even though a test that fails one run in twenty and a test that fails one run in three get treated exactly the same once both carry that label, quarantined and ignored, and they're completely different problems.
I wrote a small tool that runs a pytest suite a set number of times and classifies each test by its actual pass rate: stable if it passed every run, consistently failing if it failed every run, and flaky only if the result varied between runs. Putting a test that fails every time in the same bucket as one that fails occasionally buries an actual bug under a label that tells the team to ignore it.
The demo makes the difference pretty obvious. Two Playwright tests hit a page where an element appears after a random delay between 0 and 100 milliseconds. One waits for the element itself and passes every time, because it's checking for the actual condition instead of guessing at when it'll be ready. The other waits a fixed 50 milliseconds and then checks, which works whenever the random delay happens to land under 50ms and fails whenever it doesn't. Running that second test 15 times produced 10 passes and 5 failures. That gives you a measurable failure rate instead of just calling it flaky.
A single CI run only tells you pass or fail. Fifteen runs tell you the odds. Those odds matter when you're deciding whether a test is safe to gate a deploy on. A test failing a third of the time is broken and shouldn't be trusted with anything. A test failing one run in twenty might be worth a look when someone has time, but it isn't the same emergency.
Once you have that number, you can make a better call on a red build: fix it, quarantine it with a known failure rate attached, or admit the test caught a real bug and go find it. The tool is on GitHub, along with unit tests for the classification logic itself.