> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ontime.hosai.app/llms.txt
> Use this file to discover all available pages before exploring further.

# The outbox and delivery

> How OnTime guarantees an event that happened is an event you eventually receive — and what your consumer has to do about it.

Outbound webhooks are not fire-and-forget. They go through a **transactional outbox**, which determines exactly what OnTime guarantees you.

## The pipeline

```mermaid theme={null}
flowchart LR
  D[Domain write<br/>e.g. leave approved] -->|same transaction| O[(Outbox row)]
  O --> T[Platform tick]
  T --> H[HTTP POST to your endpoint]
  H -->|2xx| S[Delivered]
  H -->|failure| R{Attempts left?}
  R -->|yes| B[Backoff, retry]
  B --> T
  R -->|no| F[Failed — dead-letter queue]
  F --> A[Alert api:manage holders]
```

## The guarantee: at least once

The outbox row is written **inside the same database transaction as the domain change**. So:

<CardGroup cols={2}>
  <Card title="No event without the change" icon="check">
    If the approval rolls back, the outbox row rolls back with it. You never receive a webhook for something that did not happen.
  </Card>

  <Card title="No change without the event" icon="check">
    If the approval commits, the outbox row committed too. The event cannot be lost to a process restart — it is a durable row, not an in-memory retry timer.
  </Card>
</CardGroup>

<Warning>
  **At least once, not exactly once.** A delivery that succeeds but whose acknowledgement is lost will be retried. Your consumer **must** be idempotent.

  Deduplicate on the `X-Ontime-Delivery` header — it is the outbox row ID, stable across every retry of the same delivery. That header is what turns at-least-once into exactly-once on your side.
</Warning>

## Retries

| Property                | Value                                   |
| ----------------------- | --------------------------------------- |
| **Attempts**            | 8                                       |
| **Backoff**             | Quadrupling, roughly 1 minute → 8 hours |
| **Total window**        | About 22 hours                          |
| **Per-attempt timeout** | 30 seconds                              |

A delivery is a success when your endpoint returns any `2xx`. Anything else — a non-2xx status, a connection failure, a timeout — is a failure and is retried.

<Tip>
  Return `2xx` fast. Acknowledge receipt, then process asynchronously. A handler that does real work inline risks the 30-second timeout, which OnTime reads as a failure and retries — giving you the duplicate you were trying to avoid.
</Tip>

## The dead-letter queue

An event that exhausts all eight attempts moves to `Failed`. It is not deleted.

* It appears in the **delivery log** in the admin console.
* Every `api:manage` holder receives an in-app notification.
* It can be **replayed** — re-queued, sending the original payload unchanged.

<Warning>
  **The alert fires on exhaustion, not on the first failure.** The webhook's own status flips to `Failing` immediately, but the notification arrives \~22 hours later.

  So an endpoint can be visibly broken for most of a day before anyone is told. If you operate a webhook consumer, monitor it yourself rather than relying on the OnTime alarm.
</Warning>

## Crash recovery

A delivery interrupted mid-flight — the OnTime process restarting between claiming a row and completing it — is reclaimed after a few minutes and retried. The event is not stranded.

Claiming uses a database-level lock, so concurrent workers cannot pick up the same row.

## What else the tick does

The same background tick drains the outbox, runs due scheduled reports, checks for due leave accruals, closes finished days, and detects offline devices.

<Note>
  There is no separate queue broker, worker service or cron service in OnTime's topology. This matters for one reason: **an event's durability comes from the database row, not from a message broker.** A deploy in the middle of a retry cycle loses nothing.
</Note>