A timeout is not permission to repeat a write

Reliable retries require an explicit operation identity, an atomic local boundary and a plan for external commits you cannot observe.

Start reading
Conceptual illustration of two request paths meeting at a single guarded commit point.
Conceptual illustration of two request paths meeting at a single guarded commit point.

Name the ambiguity before adding retries

In a hypothetical refund workflow, a service commits a refund but its response never reaches the caller. The caller sees a timeout. Retrying with a fresh operation identity can create another refund; abandoning the request can leave the user uncertain about an action that already happened. Neither choice follows safely from the timeout itself.

Distinguish the business operation from each transport attempt. The operation is the user's authorized intent. Attempts are deliveries of that intent across an unreliable boundary. A retry policy controls timing and load; an idempotency contract controls whether repeated delivery creates another effect. Backoff and jitter do not provide deduplication.

AWS's discussion of idempotent APIs favors an explicit caller-provided request identifier over inferring intent from identical parameters [1]. Two legitimate operations can have identical bodies. An agent must therefore preserve an operation identifier through retries instead of improvising a new one whenever a tool call fails.

Bind the key to a stable request meaning

Scope an idempotency key to the tenant and operation type, and authenticate every attempt. A key is not a bearer credential. Generate it when the business operation is created and persist it before sending the first attempt. Concurrent workers handling that operation must retrieve the same identity.

Store a cryptographic request hash with the key. Canonicalize the validated semantic request before hashing: normalize field ordering, amount representation, defaults and relevant API version. Include fields that change intent, such as payment, amount and currency. Exclude transport metadata that legitimately changes between attempts. Define this canonicalization as part of the contract; arbitrary JSON serialization is not enough.

The hash detects reuse of one key with different parameters; it does not decide whether two requests represent the same intent. Reject a mismatch explicitly. Stripe documents parameter comparison for reused keys and distinguishes API-version behavior [2]. Treat that as an example of a published contract, not a universal promise about every downstream API.

Make local uniqueness and mutation atomic

A cache lookup followed by a write has a race: two workers can both observe no prior request. Enforce uniqueness in durable storage, scoped by tenant, operation type and key. For effects wholly inside one transactional database, commit the deduplication record, mutation and result together. AWS explicitly describes the need for an atomic operation encompassing the token and associated mutations [1].

Illustrative local-only transaction:
BEGIN
  INSERT operation(scope, key, request_hash)
    -- durable UNIQUE(scope, key)
  validate authorization and business preconditions
  apply local mutation
  store stable result and resource identifier
COMMIT

On unique conflict, after the winning transaction resolves:
  load committed operation
  reject a different request_hash
  otherwise return its stored result

This outline omits database-specific conflict and rollback handling. Serialize competing changes to the payment through a lock or checked version; uniqueness on request keys alone cannot enforce a refund limit across distinct operations. A worker must not return an uncommitted result. Recheck current access before exposing a stored response, and document which preconditions are evaluated only on first execution. Returning the prior operation's result is different from rerunning today's business logic against changed state.

Do not extend a local transaction across the network

The preceding transaction cannot atomically commit a refund at an independent provider. Holding a database lock while calling that provider does not close the crash window: the provider may commit before the local process records success. A local uniqueness constraint alone therefore cannot guarantee exactly-once external effects.

Persist an intent and an outbox entry together, then let a worker deliver it using a stable downstream idempotency key. The outbox prevents losing the intent after a local commit, but normally permits repeated delivery. Safety still depends on the provider's documented deduplication behavior and retention window.

Represent an ambiguous response as unknown or pending reconciliation. Query status by a stable provider reference where supported; otherwise route to a controlled review process. Do not announce success or automatically mint a new key. If the external API has neither deduplication nor reliable lookup, that is a real limit: restrict automation or accept an explicitly reviewed risk. A compensating action may itself fail and is not an atomic rollback.

Treat expiry and retry budgets as correctness decisions

Keys cannot necessarily be retained forever, but deleting them changes semantics. A delayed retry arriving after expiry may be treated as a new operation. Stripe's API v1 documentation, for example, permits pruning keys after at least 24 hours and describes a new request when a pruned key is reused [2]. That window is provider-specific, not a recommended universal setting.

Choose retention using the maximum supported retry and reconciliation horizons, queue delays and business lifetime. Consider durable business uniqueness constraints or longer-lived tombstones for effects that must never repeat. Retaining less response data may reduce privacy exposure while preserving an operation identity, but resource lookup must still be possible.

Bound retries independently with deadlines and backoff. Classify validation failures, throttling, transport failures and ambiguous server errors according to the actual endpoint contract. A repeated error may be a stored result rather than a fresh attempt. Exhausting the retry budget should produce a visible unresolved operation, not silently declare the effect absent.

Test the contract at its crash windows

Use a fault harness to interrupt the workflow at causal boundaries, not only to return random errors. The following matrix is illustrative guidance, not a report of measured product behavior.

Retry contract review matrix
ScenarioRequired observation
Concurrent identical keysOne local mutation; consistent operation result
Same key, changed amountExplicit mismatch; no second mutation
Crash after external commitReconciliation; no unguarded new operation
Retry after key expiryDocumented rejection or reviewed duplicate risk

Also test two distinct keys with identical payloads: whether both should execute depends on the business contract. Inspect durable state and provider references, not just response counts. Idempotency trades storage, coordination and reconciliation complexity for safer repeated delivery. It becomes useful when those limits are explicit enough that both the agent and its operators know when to retry, when to inspect and when to stop.

References

Primary sources and further technical context. Guidance should be checked against your own system contracts.

  1. AWS Builders’ Library — Making retries safe with idempotent APIs
  2. Stripe API documentation — Idempotent requests

By Nuhaverse editorial. Engineering guidance for system design and review. Deployment requirements and capabilities are agreed for each engagement; test results are not safety guarantees.

Back to the journal