Tutorial: Retry with exponential backoff

Why you need retries

Network timeouts, 5xx server errors, and rate limits (429) are normal in distributed systems. Without retries, a single transient failure can cause a lost payment or a broken checkout flow.

The algorithm

delay = min(base × 2^attempt + random_jitter, max_delay)

Parameter	Recommended value	Purpose
`base`	1 second	Starting delay
`max_delay`	30 seconds	Cap so you don't wait too long
`max_retries`	3	Limit total attempts
`jitter`	0–1 second random	Prevent thundering herd

Visual flow

When to retry

Response	Retry?	Reason
`200` with `response_status: approved`	No	Success
`200` with `response_status: declined`	No	Issuer refused — ask for different card
`200` with `response_status: error`	Yes	Gateway/processor issue
`400`, `401`, `404`, `422`	No	Client error — fix the request
`429`	Yes	Rate limited
`500`, `502`, `503`, `504`	Yes	Server error — transient
Timeout / connection error	Yes	Network issue

Complete implementation

import time, random, requests, os, logging

logger = logging.getLogger("paradise")

API_KEY = os.environ["PARADISE_API_KEY"]
BASE = os.environ.get("PARADISE_BASE_URL", "https://api.dev.paradisegateway.net")

def send_with_retry(
    method: str,
    path: str,
    payload: dict | None = None,
    max_retries: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 30.0,
) -> dict:
    url = f"{BASE}{path}"
    headers = {"Content-Type": "application/json", "APIKEY": API_KEY}

    for attempt in range(max_retries + 1):
        try:
            resp = requests.request(
                method, url, headers=headers, json=payload, timeout=30
            )
            body = resp.json()

            if resp.status_code == 200 and body.get("response_status") != "error":
                return body

            if 400 <= resp.status_code < 500 and resp.status_code != 429:
                logger.error(
                    "Non-retryable error %s: %s (correlation_id=%s)",
                    resp.status_code, body, body.get("correlation_id"),
                )
                return body

        except (requests.exceptions.Timeout, requests.exceptions.ConnectionError) as e:
            body = {"error": str(e)}

        if attempt == max_retries:
            logger.error("Max retries exhausted for %s %s", method, path)
            return body

        delay = min(base_delay * (2 ** attempt) + random.uniform(0, 1), max_delay)
        logger.warning(
            "Attempt %d/%d failed, retrying in %.1fs (correlation_id=%s)",
            attempt + 1, max_retries, delay, body.get("correlation_id", "n/a"),
        )
        time.sleep(delay)

    return body

Usage

result = send_with_retry("POST", "/v1/transactions", {
    "type": "sale",
    "amount": 1999,
    "payment_method": {
        "type": "card",
        "card": {
            "pan": "4111111111111111",
            "expiry_month": "12",
            "expiry_year": "2027",
            "cvv": "123",
            "cardholder_name": "John Doe",
        },
    },
})

Testing retries in sandbox

Simulate transient failures by sending requests with intentionally malformed auth to trigger 401, or use test card numbers that produce declines to verify your non-retry path works correctly.

Anti-patterns

Mistake	Why it's bad	Fix
Retry on `400`	The request is broken — retrying won't help	Fix the payload
Retry on `declined`	The issuer said no — retrying charges the customer's card repeatedly	Ask for a different card
No jitter	All your servers retry at the same time	Add random jitter
No max retries	Infinite loop during an outage	Cap at 3 retries
No idempotency key	Retries may create duplicate transactions	Always send `Idempotency-Key`