A failed request invites a simple response: try again. Sometimes that is exactly right. A connection may have dropped, or a service may have been briefly unavailable. But repeating work also consumes time and capacity. A retry needs a reason, a limit and a way to stop.
This sample considers a read request between services. It is a general design discussion, not a description of an employer’s systems.
Start with a budget
The useful question is not “How many retries can we make?” It is “How long is this result still useful to the caller?” A timeout for each attempt is only part of that answer. The overall deadline must also include waiting between attempts.
For a request with a short deadline, spending most of the budget on the first attempt leaves little room to recover. Conversely, a large number of tiny attempts can create more pressure without improving the chance of success.
type RetryBudget = {
maxAttempts: number; // Includes the first attempt
deadlineMs: number; // Includes attempts and waiting
attemptTimeoutMs: number;
};
const budget: RetryBudget = {
maxAttempts: 3,
deadlineMs: 2_000,
attemptTimeoutMs: 500,
};These numbers are illustrative, not recommended defaults. The type records a policy; it does not implement cancellation. An implementation must stop in-flight work when possible, account for elapsed time and pass the remaining deadline to each attempt.
Make repetition safe
A timeout tells us that we did not receive an answer. It does not prove that the remote operation failed. That distinction matters when a request changes state.
For a write operation, safe retries may require a stable idempotency key and a server-side guarantee that repeated requests do not repeat the effect. Simply adding a header on the client is not enough. The service must define how keys are stored, how duplicates are handled and how long that guarantee lasts.
Before repeating an operation, understand what a second successful execution would mean.
Some errors should not be retried at all. An invalid request will not become valid because we waited. Retry policy should distinguish transient failures from problems that require a change to the request or the system.
Leave room to recover
Backoff spreads attempts over time. Jitter prevents many callers from making their next attempt together. Both help only when there is a finite budget and a clear understanding of which layer owns the retry.
If several layers retry independently, a small limit at each layer can multiply into substantial downstream work. Start by identifying the owner, then make its behavior observable: attempts, elapsed time, final outcome and reasons for stopping.
The goal is not to hide every failure. It is to recover from the failures that are safe to repeat, without making a struggling dependency’s job harder.