When the Mock Is Right and the API Isn't Anymore
Jul 25, 2026
A mocked test suite passed every run for months. The real API's response contract had changed, and nothing caught it.
A mocked test suite passed every run for months. The real API's response contract had changed, and nothing caught it.
Same code, same test. The only thing different was the timezone the two machines were running in.
Happy paths should be automated. Once that's done, here's specifically what's left for a person to do.
An AI-generated Playwright test passed every time, but it only checked what the page reported, not what the backend actually did.
I added retry logic to a flaky Playwright suite and the tests went green. I thought I had fixed it. I hadn't.
Playwright's default navigation behavior assumes a traditional page load. Phoenix LiveView doesn't work that way, and your test suite will tell you.
Breaking Playwright suites taught me to automate less and test smarter. What I actually do now to keep QA stable.