What is a Webhook Retry Policy?

A webhook retry policy is a set of rules that a webhook system uses to determine whether to resend a webhook if unsuccessful. This configuration specifies the system’s limit on attempts, the pause between each attempt, the criteria for failure, and the conditions for discontinuing. In most policies, exponential backoff is employed to prevent attempts in short time frames, whereas the permanently failing events are sent to a dead-letter queue.

Retry policies are a relevant consideration, given that networks and services may exhibit variations in reliability. The finalization of data transfers during network or service disturbances in delivery is dependent on the presence of an active retry policy. A retry policy prevents data loss from a delivery failure by ensuring a retry after the receiving end doesn’t return a 2xx.

When an event delivery attempt encounters an issue, the system’s policy usually processes this by examining the HTTP status code. Following this, a pause is introduced, consistent with a retry schedule, before the event transmission is reattempted. These are the components:

•       Trigger conditions: the precise HTTP statuses that trigger a retry.
•       Retry schedule: intervals in time between attempts, normally exponential backoff or a value extracted from the Retry-After header.
•       Failure surfacing: it involves functions that enable the identification of issues without an immediate resolution, such as a next retry time indication.
•       Observability and idempotency: logs, metrics, and replay tools for failure analysis, along with idempotent handling to swallow duplicates.

Most retry strategies incorporate a delay in the interval between attempts. The one that is being followed most often is exponential backoff; it prolongs the wait period with each attempt, which, in turn, lowers the burden on a receiver suffering some kind of problem.

This means:

•       Exponential backoff: is a technique where retry attempts are separated by increasingly longer time periods.
•       Adding jitter: varying the wait time by a random element to avoid retry spikes at the same time. Twilio Event Streams implements the idea of jitter to avoid a “thundering herd”.
•       Tiered retries: immediate, short-term, long-term, and dead-letter queue.

This queue is intended for events that have utilized their configured retry processes without reaching a completed status. These events are retained in a designated location, allowing for examination of their specifics, observation of their sequence, or addressing them as situations may dictate.

Rather​‍​‌‍​‍‌ than blindly copying aggressive retry policies, good retry design considers when the system might be overwhelmed by too many retries. That is, it involves carefully thinking through retry limits and interval timing.

•       Prefer exponential backoff plus jitter.
•       Establish well-defined retry limits and stopping criteria.
•       Establish unambiguous status-code rules on which deliveries will be retried and which ones won’t.
•       Idempotency keys relate to the consistent management of reprocessing.

Monitoring practices relate to the identification of discrepancies in pipeline delivery at an early stage. Track metrics that indicate problems, then automatically notify via alerts and link them to the tools used for operation.

•       Dashboards: an overview of delivery health for the last period of time.
•       Manual replay: an option that allows sending events again after the root cause is resolved.
•       Runbooks: they present pre-defined actions for on-call engineers, intended to guide the troubleshooting process.

•       A process of methodically re-trying failed deliveries aims to address instances where events might otherwise go unprocessed.
•       It contributes to a reduced likelihood of event data loss during short-term interruptions and delivery delays.
•       Integrations typically recover from short periods of disruption, with their associated issues often being temporary rather than causing extended outages.
•       The consistent delivery of critical data, such as orders and payments, is a factor influencing the level of confidence in integrations.