None Is Not an Error
Sep 12, 2026
search_player_id used to print a message and return None. Testing that meant checking two different things instead of one.
search_player_id used to print a message and return None. Testing that meant checking two different things instead of one.
A test that fails once and a test that fails constantly look identical from a single CI run. Only one of them is actually flaky.
A mocked test suite passed every run for months. The real API's response contract had changed, and nothing caught it.
Same code, same test. The only thing different was the timezone the two machines were running in.
Happy paths should be automated. Once that's done, here's specifically what's left for a person to do.
An AI-generated Playwright test passed every time, but it only checked what the page reported, not what the backend actually did.
I added retry logic to a flaky Playwright suite and the tests went green. I thought I had fixed it. I hadn't.
Playwright's default navigation behavior assumes a traditional page load. Phoenix LiveView doesn't work that way, and your test suite will tell you.
Breaking Playwright suites taught me to automate less and test smarter. What I actually do now to keep QA stable.