Skip to main content
Outbound webhooks are not fire-and-forget. They go through a transactional outbox, which determines exactly what OnTime guarantees you.

​
The pipeline

​
The guarantee: at least once

The outbox row is written inside the same database transaction as the domain change. So:

No event without the change

If the approval rolls back, the outbox row rolls back with it. You never receive a webhook for something that did not happen.

No change without the event

If the approval commits, the outbox row committed too. The event cannot be lost to a process restart — it is a durable row, not an in-memory retry timer.
At least once, not exactly once. A delivery that succeeds but whose acknowledgement is lost will be retried. Your consumer must be idempotent.Deduplicate on the X-Ontime-Delivery header — it is the outbox row ID, stable across every retry of the same delivery. That header is what turns at-least-once into exactly-once on your side.

​
Retries

A delivery is a success when your endpoint returns any 2xx. Anything else — a non-2xx status, a connection failure, a timeout — is a failure and is retried.
Return 2xx fast. Acknowledge receipt, then process asynchronously. A handler that does real work inline risks the 30-second timeout, which OnTime reads as a failure and retries — giving you the duplicate you were trying to avoid.

​
The dead-letter queue

An event that exhausts all eight attempts moves to Failed. It is not deleted.
  • It appears in the delivery log in the admin console.
  • Every api:manage holder receives an in-app notification.
  • It can be replayed — re-queued, sending the original payload unchanged.
The alert fires on exhaustion, not on the first failure. The webhook’s own status flips to Failing immediately, but the notification arrives ~22 hours later.So an endpoint can be visibly broken for most of a day before anyone is told. If you operate a webhook consumer, monitor it yourself rather than relying on the OnTime alarm.

​
Crash recovery

A delivery interrupted mid-flight — the OnTime process restarting between claiming a row and completing it — is reclaimed after a few minutes and retried. The event is not stranded. Claiming uses a database-level lock, so concurrent workers cannot pick up the same row.

​
What else the tick does

The same background tick drains the outbox, runs due scheduled reports, checks for due leave accruals, closes finished days, and detects offline devices.
There is no separate queue broker, worker service or cron service in OnTime’s topology. This matters for one reason: an event’s durability comes from the database row, not from a message broker. A deploy in the middle of a retry cycle loses nothing.