A custody provider confirms a transfer and sends a webhook. The receiving application records the credit, but its response is lost before the provider receives it. The provider retries. If the second delivery creates a second credit, a routine transport failure has become a financial discrepancy.
A webhook is an event notification sent to a receiving endpoint. It helps systems react quickly without repeatedly asking for updates. It is not, by itself, a guarantee that a business action happens once, happens in order or happens at all. The integration has to establish those properties around the notification.
Read the actual delivery promise
Providers differ on retry windows, ordering, signature formats and replay support. Alchemy's webhook documentation describes ordered first-time notifications and automatic retries with exponential backoff when delivery fails. The qualification matters: a promise about first-time delivery does not settle the order in which delayed retries and current messages reach a business process.
Stripe's webhook guidance, a useful payment-integration reference, explicitly discusses duplicate events and does not guarantee event delivery order. Neither provider's rules should be assumed to describe another vendor. Record the terms for the specific product, network and event type the institution uses.
Distinguish an event identifier from a delivery-attempt identifier and a business-object identifier. Several deliveries may represent one event. Several events may refer to one transfer. A transaction hash can identify an onchain transaction containing several transfers, so it may be too coarse to identify one ledger movement.
Make receipt durable before acknowledging success
The receiving endpoint should verify the vendor's signature according to the documented method, validate the expected environment and store the accepted event durably. Only then should it acknowledge successful receipt. The acknowledgment tells the sender that it can stop trying; returning it before persistence creates a loss window if the receiver immediately fails.
Keep slow business work outside the narrow receipt path where practical. A durable queue or inbox allows processing to continue after the provider receives its acknowledgment. The queue needs its own retry, monitoring and recovery policy. Moving work into a queue does not remove duplication; it creates another delivery boundary that must be handled.
Retain the original payload, relevant headers and verification outcome under an appropriate access policy. A parsed subset may omit the field needed to investigate a later schema change. Secrets used for verification should be stored separately, with rotation tested rather than assumed.
Idempotency is a property of the business action
Idempotency means repeated attempts produce the same intended effect as a single successful attempt. Checking whether an event identifier has been seen is useful, but the check and the financial change must be coordinated. Two workers can both see an unseen event and both post a credit if the check is merely an earlier read.
Use a persistent uniqueness rule or equivalent transactional control around the receipt and resulting ledger operation. The exact design depends on the system, but it must survive concurrent workers and process failure. An in-memory list disappears during a restart and cannot be the only record preventing a duplicate financial effect.
External side effects require another step. If processing sends a payment instruction or a client message, a local database transaction does not automatically cover the remote system. Preserve an intended operation with a stable identifier, use downstream idempotency support where available and reconcile uncertain outcomes before retrying an action that might already have succeeded.
A hypothetical transfer with a delayed status
Hypothetical example: a treasury application receives a completed-transfer notification and records the transfer as complete. An earlier processing notification, delayed by a delivery retry, arrives afterwards. A naive handler replaces the current status with whatever the newest received message says. The transfer appears pending again.
The right response is not simply to sort all notifications by their arrival times. Arrival time describes transport, not business sequence. Use documented object versions or ordering markers where provided, validate allowed state transitions and retrieve the authoritative object when the available event data cannot resolve the conflict.
Some changes are genuine reversals or corrections. A design that refuses every transition away from completed can also be wrong. The application's state model needs to distinguish stale observations from explicit compensating events. The provider's business semantics, and the institution's ledger treatment, must agree on that distinction.
Plan for events that never reach the inbox
Retries can be exhausted, subscriptions can be misconfigured and an event filter can exclude the activity the team expected to receive. An empty error dashboard does not prove completeness. If the provider never sent an event to that endpoint, the endpoint has nothing to flag.
Run independent reconciliation against authoritative transfer, balance or transaction records. The scope and frequency should follow the consequence of a missed event. A deposit credited to a client and a notification used only for an internal dashboard may require different recovery times.
Reconciliation needs stable identifiers, pagination, overlapping time windows and a rule for corrections. A simple request for records since the last local timestamp can miss late-arriving changes if the provider's time field has different semantics. Document whether the query filters creation time, update time or an onchain event time.
Replay tools need authorisation too. An operator should be able to reprocess the original event without bypassing the normal uniqueness and transition checks. Keep a record of who initiated replay, its scope and the resulting business effects. A recovery tool with an unrestricted force option can undo the protections built into normal delivery.
Network observations also need a correction path. If a provider exposes provisional events and later reversals, preserve their relationship to the original ledger entry. A replacement notification is not permission to delete accounting history. The business process needs a controlled reversal or adjustment with its own traceable identifier.
Test the failure boundaries as a sequence
Replay a valid event twice, deliver older events after newer ones and interrupt the worker between persistence and posting. Exercise an acknowledgment lost after successful work, a signature failure, a schema change and an outage longer than the retry window. Include simultaneous duplicate deliveries, because sequential tests can miss races.
Measure receipt-to-processing delay, oldest unprocessed event, repeated processing attempts, unmatched reconciliation items and time to resolve uncertain outcomes. Monitoring only HTTP success rates rewards a receiver that acknowledges quickly even when its workers are stuck. Operations needs to see the financial backlog.
The distinction between observation and authoritative state is developed in blockchain indexer data quality. The timing questions connect to onchain timestamps and reporting cutoffs. A webhook integration is dependable when the institution can account for the business effect of each relevant event, including the ones it had to discover without a webhook.