Retrying an API Call in Tests Without Hiding Real Failures
Retry logic in test automation sits right at the edge of a trap. On one side, a test that fails every third run because of a transient network hiccup is genuinely noisy — it erodes trust in the suite and eventually gets ignored. On the other side, a test that silently retries until it passes is no longer a test; it's a wishful loop that can hide a real regression behind a green checkmark. I've watched both failure modes cause real problems in production, and the difference between them almost always comes down to how the retry was implemented, not whether retrying was the right call in the first place.
The pattern that shows up most often in codebases I've reviewed is a bare for loop or a blanket tenacity decorator slapped onto a test with no thought about what conditions should trigger a retry, how many attempts are reasonable, or what gets logged when those attempts happen. That approach papers over flakiness without explaining it. The goal of this article is to give you a retry pattern that is honest: one that retries only on the failure modes that genuinely warrant a second chance, surfaces every attempt in the logs, and fails loudly — with full context — when retries are exhausted.
Everything here is in Python, using pytest and the tenacity library, which gives you the most surgical control over retry behavior. The same principles apply if you're using Behave or a custom retry helper — the library is secondary to the design decisions behind it.
Learn Python, Behave, GitHub Copilot, APIs, and CI/CD by building a real framework you can finish in a weekend.
Defining What Actually Deserves a Retry (and What Doesn't)
The first decision is the most important one: not every failure is retry-worthy. Retrying indiscriminately is how you end up with a suite that hides regressions. Before you write a single line of retry code, you need a written policy — even if it's just a comment in the test — that names the specific conditions under which a retry is permitted.
In practice, retry-worthy failures share one characteristic: they are caused by something external to the system under test, and that cause is genuinely transient. The short list looks like this:
- HTTP 429 (Too Many Requests) — a rate-limit hit that will resolve after a backoff.
- HTTP 503 (Service Unavailable) — a downstream dependency momentarily unavailable.
- Connection timeouts — a network blip, not an application error.
- HTTP 502 / 504 gateway errors — load balancer or proxy noise in a shared environment.
The list of things that should never be retried is longer:
- HTTP 400 / 422 — your request is malformed. Retrying won't fix it.
- HTTP 401 / 403 — an auth or permissions failure. Retrying is noise; the credential is wrong.
- HTTP 404 — the resource doesn't exist. Retrying won't create it.
- HTTP 500 — this one is subtle. A 500 can be transient, but it can also be a real application crash. I treat 500s as non-retryable by default and escalate them immediately, because status codes don't always tell the full story — a 500 that resolves on retry is still a 500 that happened, and that's worth investigating.
- Assertion failures on response body content — if the API returned 200 but the payload is wrong, that is a bug. No amount of retrying will change what the server returned.
Here's how to encode that policy in code using tenacity:
import requests
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception
RETRYABLE_STATUS_CODES = {429, 502, 503, 504}
def is_retryable_http_error(exc: BaseException) -> bool:
"""Only retry on transient HTTP status codes, never on client errors."""
if isinstance(exc, requests.HTTPError):
return exc.response is not None and exc.response.status_code in RETRYABLE_STATUS_CODES
if isinstance(exc, (requests.ConnectionError, requests.Timeout)):
return True
return False
@retry(
retry=retry_if_exception(is_retryable_http_error),
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=8),
reraise=True,
)
def call_api(url: str, headers: dict) -> requests.Response:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
return response
The key detail is reraise=True. When all attempts are exhausted, tenacity re-raises the original exception instead of swallowing it into a RetryError. That means your test still fails with the real error, and your CI log still shows exactly what went wrong. The retry is transparent, not a trapdoor.
Also notice that raise_for_status() is called inside the retried function. That means any 4xx or 5xx that isn't in RETRYABLE_STATUS_CODES will raise an HTTPError, is_retryable_http_error will return False, and tenacity will immediately stop and propagate the exception. The retry gate is enforced at the exception level, not as an afterthought.
Logging Every Attempt So Retries Are Visible, Not Silent
A retry that isn't logged is a lie of omission. If your test passes on the third attempt and nothing in the output mentions the two failures that preceded it, you've lost information that could have pointed to a real infrastructure problem. The whole point of structured test logging is to make failure patterns visible — and retries are failure events, even when they eventually succeed.
tenacity provides a before_sleep callback that fires between attempts. Use it to log every retry with enough context to be actionable:
import logging
from tenacity import before_sleep_log
logger = logging.getLogger(__name__)
@retry(
retry=retry_if_exception(is_retryable_http_error),
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=8),
before_sleep=before_sleep_log(logger, logging.WARNING),
reraise=True,
)
def call_api(url: str, headers: dict) -> requests.Response:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
return response
before_sleep_log emits a WARNING-level message that includes the attempt number, the exception, and the time until the next retry. In a CI log, that looks like:
WARNING:__main__:Retrying call_api in 1.0 seconds as it raised
HTTPError: 503 Server Error: Service Unavailable for url: https://api.example.com/orders
(attempt 1 of 3)
That single line tells you: which function retried, why, when, and how many attempts remain. If you see this in a passing build, you know to go look at your environment's stability. If you see it in a failing build, you know the retry policy was respected and the failure is genuine.
For teams that need richer context — like the full response body on a 503 — you can write a custom before_sleep callback instead:
from tenacity import RetryCallState
def log_retry_attempt(retry_state: RetryCallState) -> None:
exc = retry_state.outcome.exception()
attempt = retry_state.attempt_number
wait = retry_state.next_action.sleep # seconds until next attempt
status_code = None
if isinstance(exc, requests.HTTPError) and exc.response is not None:
status_code = exc.response.status_code
logger.warning(
"Retry attempt %d | status=%s | exception=%s | sleeping %.1fs",
attempt,
status_code or "N/A",
type(exc).__name__,
wait,
)
@retry(
retry=retry_if_exception(is_retryable_http_error),
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=8),
before_sleep=log_retry_attempt,
reraise=True,
)
def call_api(url: str, headers: dict) -> requests.Response:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
return response
One thing I want to be direct about: if a test is hitting the retry path on every single run, the retry isn't the fix — it's a bandage. The retry buys you time to investigate; it doesn't replace the investigation. Tracking which tests retry most frequently across your CI runs will surface the chronic offenders fast. Treat a high retry rate as a signal that something in your environment or test setup needs attention.
Keeping Retry Logic Out of Your Test Body (and Why That Matters)
One mistake I see constantly is retry logic written directly inside the test function itself — a for attempt in range(3): loop wrapping the assertion, with a break on success and a continue on failure. It works, technically, but it conflates two completely separate concerns: how to call the API reliably and what to assert about the response. When those two things live in the same block of code, both become harder to read, harder to reuse, and much harder to debug when something goes wrong.
The pattern I recommend is to push all retry logic into the API client layer — a dedicated helper or fixture that your tests call — and keep test bodies focused purely on assertions. Here's what that looks like in a pytest fixture:
# conftest.py
import pytest
import requests
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception
RETRYABLE_STATUS_CODES = {429, 502, 503, 504}
def is_retryable_http_error(exc: BaseException) -> bool:
if isinstance(exc, requests.HTTPError):
return exc.response is not None and exc.response.status_code in RETRYABLE_STATUS_CODES
return isinstance(exc, (requests.ConnectionError, requests.Timeout))
class ResilientAPIClient:
def __init__(self, base_url: str, headers: dict):
self.base_url = base_url
self.headers = headers
@retry(
retry=retry_if_exception(is_retryable_http_error),
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=8),
reraise=True,
)
def get(self, path: str) -> requests.Response:
response = requests.get(
f"{self.base_url}{path}",
headers=self.headers,
timeout=10,
)
response.raise_for_status()
return response
@pytest.fixture
def api_client():
return ResilientAPIClient(
base_url="https://api.example.com",
headers={"Authorization": "Bearer test-token"},
)
Now the test itself is clean:
# test_orders.py
def test_order_list_returns_200(api_client):
response = api_client.get("/orders")
assert response.status_code == 200
assert "orders" in response.json()
The test has no idea retries are happening. If the call succeeds on the second attempt, the test passes and the retry is logged at WARNING level in the output. If all three attempts fail, reraise=True surfaces the original exception and the test fails — with the full traceback pointing at the get method, not buried inside a loop in the test body.
This separation also makes it straightforward to test the retry behavior itself in isolation, without coupling it to a specific endpoint's business logic. And it pays dividends when you're running tests in parallel — centralized retry logic with exponential backoff reduces the thundering-herd effect where many parallel workers simultaneously hammer a rate-limited endpoint.
The final rule I'd leave you with: treat every retry as a yellow flag, not a green one. A test that passes after a retry passed — but it also told you something. Log it, track it, and periodically review which tests are relying on retries to stay green. That review is often where you find the real bugs: the environment instability, the undocumented rate limit, the auth token that expires faster than expected. Retry logic done right doesn't hide those problems. It gives you just enough runway to find them.