# Webhooks

Webhooks send status changes to your own endpoints, such as a Slack relay, PagerDuty, or a Lambda function. TunnelHQ sends one event when a monitor goes down and another when it recovers.

## Project webhooks

Set up project webhooks under **Developers → Webhooks**. Add an endpoint URL, choose the events it receives, and optionally give it a secret so you can [verify deliveries](#verifying-deliveries).

Project webhooks are **edge-triggered**: you get exactly one `monitor.down` when a monitor goes from up to down, and exactly one `monitor.up` when it recovers. A new monitor's first check also sends an event: `monitor.up` if it's healthy, or `monitor.down` (or `monitor.degraded`, if it's retrying) if it fails.

:::note[No repeats during an outage]
A project webhook fires on the change of state, not on every failed check. If a server is down for six hours, you get one `monitor.down` at the start and one `monitor.up` at recovery, with no "still down" calls in between. This is what receivers such as Slack, Discord, and PagerDuty expect.
:::

## Events

Subscribe each endpoint to the events it needs.

| Group | Event | Sent when |
| --- | --- | --- |
| Monitors | `monitor.down` | A monitor goes down. |
| | `monitor.degraded` | A monitor fails a check and starts retrying (see below). |
| | `monitor.up` | A monitor recovers, or a new monitor's first check passes. |
| | `monitor.created` | A monitor is created. |
| | `monitor.updated` | A monitor is changed. |
| | `monitor.deleted` | A monitor is deleted. |
| | `monitor.paused` | A monitor is paused. |
| | `monitor.resumed` | A monitor is resumed. |
| Incidents | `incident.created` | An incident opens. |
| | `incident.resolved` | An incident closes. |

### About monitor.degraded

`monitor.degraded` is a retry signal, not a partial-failure signal. It fires when a monitor fails a check and starts retrying before it's declared down, so a monitor with a single location emits it on a brief network blip. It isn't limited to servers that fail in some locations and pass in others.

Every `monitor.degraded` is followed by a `monitor.up` when the monitor recovers, so a receiver never stays stuck in a degraded state. If you only want confirmed outages, subscribe to `monitor.down` and `monitor.up` and leave `monitor.degraded` off. A recovery that only ends a brief retry is delivered only to endpoints subscribed to the degraded event, so it won't reach you.

:::caution[Not the same as the degraded status]
The monitor *status* called degraded is a different thing. It means no location completed a check and at least one was refused outright, with its credentials or certificate rejected. The event is about retrying; the status is a confirmed configuration problem. See [Locations and status](/docs/concepts/regions/#aggregate-status).
:::

## Payload

Every delivery is a JSON `POST` with the same envelope: `event`, `timestamp`, `projectId`, and an event-specific `data` object.

### Status events

`monitor.up`, `monitor.down`, and `monitor.degraded` carry the monitor and the check that changed its state:

```json title="monitor.down"
{
  "event": "monitor.down",
  "timestamp": "2026-06-14T08:05:10.000Z",
  "projectId": 42,
  "data": {
    "monitor_id": "srv_916",
    "monitor_name": "Sydney-I",
    "protocol": "openconnect",
    "protocols": ["openconnect"],
    "failed_protocols": ["openconnect"],
    "host": "203.0.113.15",
    "port": 443,
    "time": "2026-06-14 08:05:10.123",
    "latency_ms": null,
    "connect_ms": null,
    "message": "Tunnel unreachable",
    "failed_stage": "connect",
    "diagnostic": "…",
    "region": "any",
    "retries": 2
  }
}
```

| Field | Description |
| --- | --- |
| `monitor_id` | The monitor's ID, such as `srv_916`. |
| `monitor_name` | The monitor's name. |
| `protocol` | The monitor's protocol. |
| `protocols` | The monitor's protocols, as an array. |
| `failed_protocols` | The protocols that failed, as an array, or `null`. |
| `host`, `port` | The server address. |
| `time` | When the check ran, in UTC, without a time zone marker (`2026-06-14 08:05:10.123`). |
| `latency_ms` | One round trip through the tunnel, in milliseconds, or `null` if the check failed. |
| `connect_ms` | How long the tunnel took to come up, in milliseconds, or `null` if the check failed. |
| `message` | A short summary of the result. |
| `failed_stage` | The [test stage](/docs/concepts/monitors/#the-test-journey) that failed, such as `connect`. |
| `diagnostic` | The checker's detail, such as `amneziawg handshake not established (peer not responding)`. |
| `region` | The location the check ran from, as a region code such as `any` or `pk`. |
| `retries` | The number of failed checks behind the change. |

`timestamp` in the envelope is ISO 8601 with a `Z`. `time` in `data` has no zone marker, so treat it as UTC.

### Incident events

`incident.created` and `incident.resolved` carry every status field above, plus:

| Field | Description |
| --- | --- |
| `incident_id` | The incident's ID, such as `inc_…`. |
| `incident_status` | The incident's status. |
| `incident_started_at` | When the incident opened. |
| `incident_ended_at` | When the incident closed. |

### Lifecycle events

Lifecycle events have a different `data` shape:

- `monitor.created`, `monitor.updated`, `monitor.paused`, and `monitor.resumed` carry the whole monitor object in `data.monitor`. Its `id` is a number, not an `srv_` ID.
- `monitor.paused` and `monitor.resumed` fire however the monitor was paused or resumed: in the dashboard, through the API, or from the MCP server.
- `monitor.deleted` carries `data.monitorId`, the deleted monitor's number.

## Delivery

Every delivery sends these headers:

| Header | Value |
| --- | --- |
| `Content-Type` | `application/json` |
| `User-Agent` | `TunnelHQ-Webhook/1.0` |
| `X-TunnelHQ-Event` | The event name, such as `monitor.down`. |
| `X-TunnelHQ-Endpoint-Id` | The endpoint's ID. |
| `X-TunnelHQ-Signature` | The signature. Only sent when the endpoint has a secret. |
| `X-API-Key-Id` | Only sent when the endpoint is linked to an API key. |

- **Success:** any `2xx` response. Redirects aren't followed, so a `3xx` counts as a failure.
- **Timeout:** each attempt waits up to 5 seconds.
- **Retries:** a failed delivery is retried 3 more times, 1 second apart, for 4 attempts in total.
- **Auto-disable:** after 50 failed deliveries in a row, the endpoint is switched off. It stays off until someone turns it back on; a later successful delivery doesn't re-enable it.
- **Public addresses only:** TunnelHQ delivers only to public addresses. An endpoint that resolves to a private address is refused, and this is checked again on every attempt.

## Verifying deliveries

If you give an endpoint a **secret**, every delivery to it is signed. The `X-TunnelHQ-Signature` header contains `sha256=<hex>`: an HMAC-SHA256 of the JSON body exactly as sent, keyed with your secret. Recompute the HMAC over the raw body and compare the two before trusting the payload. Reject any request whose signature header is missing or malformed: an attacker controls that header, and a genuine delivery to a signed endpoint always sends a well-formed one.

```js title="verify.js"
const crypto = require("crypto");

function verify(rawBody, signatureHeader, secret) {
  const expected = "sha256=" +
    crypto.createHmac("sha256", secret).update(rawBody).digest("hex");
  const received = Buffer.from(String(signatureHeader ?? ""), "utf8");
  const digest = Buffer.from(expected, "utf8");
  // timingSafeEqual throws unless both buffers are the same length, so a
  // missing or truncated header has to be rejected before it gets there.
  // Comparing lengths first leaks nothing: the digest length is fixed.
  return received.length === digest.length &&
    crypto.timingSafeEqual(received, digest);
}
```

## Delivery log

Each endpoint's page (**Developers → Webhooks**, then the endpoint) lists every delivery with its timestamp and response. Filter by **All**, **Success**, or **Failure**, expand a delivery to see what was sent, and redeliver it if you need to. The same page reveals and regenerates the endpoint's signing secret.

To check whether a webhook fired, use the delivery log. The "still down" heartbeat rows in the dashboard are status updates, not webhook calls.

## Notification webhooks (legacy)

There's a second, older webhook under **Integrations → Add Integration → Webhook**, inherited from the monitoring engine TunnelHQ is built on. Like the other notification channels, it fires on changes of state.

Repeated reminders during an outage come from a per-monitor setting, not from the webhook: **Resend Notification if Down X times consecutively** (0 turns it off). Set it in a monitor's settings, or for several monitors at once in the bulk settings on the Monitors page. It repeats the alert on every channel that monitor uses, not only webhooks.

:::tip[Which should I use?]
For new integrations, use **project webhooks**. They're edge-triggered, signed, retried, and logged.
:::