Using AI to Explain a Test Failure You Did Not Write
You open a pull request, CI is red, and the failing test has a name like test_order_fulfillment_idempotency_edge_case_v2. You didn't write it, you don't own it, and the person who did left the team six months ago. The stack trace points somewhere deep in a shared fixture, and the last commit touching that file was a dependency bump. This is one of the most draining situations in test automation — not because the bug is hard, but because the context is missing. In practice, I've found that AI tools can dramatically compress the time it takes to rebuild that context from scratch.
What I'm talking about isn't magic. You're not asking an AI to fix the test for you — you're using it as a first-pass analyst that can read code without ego, summarize what a test is asserting, and surface the most likely failure causes given a stack trace. The key is knowing how to structure the conversation so you get a useful explanation rather than a confident-sounding guess. That distinction matters a lot when the test you're investigating gates a production deploy.
This article walks through the exact prompting patterns I use, the mistakes that produce bad AI output, and how to verify whatever the AI tells you before you act on it. The goal is a repeatable workflow you can apply the next time you inherit a red build you know nothing about.
Learn Node.js, Cucumber, GitHub Copilot, APIs, CI/CD, and modern automation by building a complete framework.
Reading the Stack Trace and Test Body Together With AI
The single biggest mistake I see when people ask an AI to explain a test failure is pasting only the stack trace. A stack trace tells you where the code blew up; it tells you almost nothing about what the test was trying to prove. AI models fill that gap with plausible-sounding assumptions, and those assumptions are often wrong in subtle ways that cost you real time.
The prompt pattern that actually works is a three-part paste:
- The full test function — including any decorators, parametrize markers, and the fixture names it receives.
- The relevant fixture definitions — especially any setup that touches external state, like an API client or a database seed.
- The complete stack trace — not just the last line, but the full traceback so the AI can see the call chain.
With those three pieces, a prompt like this works well:
Here is a failing pytest test, its fixtures, and the stack trace.
Explain in plain English what this test is asserting, what the
stack trace says went wrong, and what the top two or three most
likely root causes are. Do not suggest fixes yet — just explain.
[paste test]
[paste fixtures]
[paste stack trace]
Asking it not to suggest fixes yet is deliberate. When you ask for fixes in the same breath as the explanation, the AI anchors on the first plausible fix it generates and shapes the explanation to support that fix. Separating the two steps keeps the explanation honest.
A pattern that shows up often: the AI will correctly identify that a fixture is creating shared state and flag that as a likely culprit, even when the stack trace doesn't point there directly. That's genuinely useful — it's the kind of thing a senior engineer would spot after reading the code for five minutes, and the AI does it in seconds. For a deeper look at how shared fixture state can cause exactly these kinds of silent failures, the article on sharing test data across a Behave suite is worth reading alongside this workflow.
One more thing: always include the assertion line explicitly in your paste, even if it seems obvious. AI models sometimes misread what a test is asserting when the assertion is buried inside a helper or a custom matcher. If the model misunderstands the assertion, everything downstream in its explanation will be subtly off.
Prompting AI to Distinguish a Broken Test From a Real Bug
Once you have the AI's plain-English explanation of the failure, the next question is the one that actually matters: is the test wrong, or is the code wrong? This is where unfamiliarity with the test hurts most, because you don't have the original author's intent to guide you. AI can help here too, but you have to ask the right question.
After the explanation step, I follow up with a prompt like this:
Given your explanation, what would have to be true about the
application code for this test to be correct and meaningful?
And what would have to be true for this test to be poorly written
or outdated?
This forces the AI to reason from both sides of the fence rather than defaulting to "the test is probably fine, the code broke." In practice, I've seen AI catch things like: a test asserting a specific HTTP status code that was intentionally changed in a recent API version, a fixture that seeds data in a format that no longer matches the schema, or a test that was written against a mock but is now running against a live endpoint with different latency characteristics.
That last case — environment mismatch — is particularly sneaky. If the failure only reproduces in CI and not locally, the AI's explanation alone won't be enough; you need to cross-reference the environment. The patterns for debugging a flaky API test that only fails in CI apply directly here, and pairing that environmental checklist with the AI's code-level explanation covers most of the bases.
A few things to watch for when evaluating the AI's output at this stage:
- Overconfidence about intent. The AI will sometimes say "this test is clearly checking X" when the test name and the assertion are actually in tension. Trust the assertion, not the name.
- Hallucinated library behavior. If the test uses a less common library or an internal utility, the AI may invent method signatures or behaviors. Cross-check anything it says about a specific library call against the actual source or docs.
- Missing context about deploy history. The AI has no idea what changed in the last deploy. If the failure is new, a recent schema or contract change is often the real cause, and the AI can't know that without you telling it. Feed it the relevant diff if you have it.
The goal at the end of this step is a short, written hypothesis — one or two sentences — that you can go verify manually. The AI gives you the hypothesis; you do the verification. That division of labor is what makes this fast without making it reckless.
Turning the AI's Explanation Into a Fix You Can Actually Trust
Once you have a verified hypothesis, you can ask the AI to help you draft a fix — but the way you frame that request determines whether you get something useful or something that passes the test by weakening it. I've seen plenty of AI-suggested "fixes" that amount to loosening an assertion just enough to make the failure go away, which is the worst possible outcome: the test goes green, the real bug ships, and nobody knows.
The prompt I use for this stage is explicit about the constraint:
Based on the hypothesis that [your verified hypothesis here], suggest
a fix. The fix must preserve the original intent of the test — do not
suggest weakening the assertion or skipping the test. If the right fix
is in the application code rather than the test, say so.
That last sentence matters. AI tools default to fixing the test because that's what you handed them. Reminding it that the fix might live in the application code keeps it honest about which layer actually needs to change.
When the AI does suggest a test-level fix — updating a fixture, adjusting a parametrized value, fixing a stale mock — I treat that suggestion the same way I'd treat a suggestion from a junior engineer: read it carefully, understand why it works, and make sure it doesn't accidentally reduce coverage. This is essentially the same discipline described in the guidance around using GitHub Copilot to draft test cases safely — AI output is a starting point, not a finished product.
A few practical habits that make this whole workflow more reliable:
- Keep a running comment in the PR. Paste the AI's explanation (lightly edited) as a PR comment before you push the fix. This gives reviewers context and creates a paper trail for the next person who hits this test.
- Run the test in isolation before and after.
pytest path/to/test.py::test_name -v— confirm it was actually failing, then confirm your fix makes it pass without touching anything else. - Check neighboring tests. If the AI identified shared fixture state as the culprit, other tests using that fixture may be silently broken or silently passing for the wrong reason. A quick scan of the fixture's consumers is worth the two minutes it takes.
The broader pattern here is that AI is most valuable for test failures when it's used as a context accelerator, not an answer machine. You still need to verify the hypothesis, understand the fix, and own the outcome. But compressing the "what is this test even doing" phase from thirty minutes to three minutes is a real productivity gain — especially when you're dealing with a large suite you didn't build yourself. If you're working toward that kind of systematic ownership of an inherited framework, the guide to building production-ready test automation frameworks covers the structural patterns that make individual tests easier to understand and debug in the first place.