Webhooks and events

What this page covers

This page is for engineers of other systems who want to receive something from mmune, or push something into it. It covers what mmune sends out (alert webhooks, Slack, Teams, PagerDuty, ServiceNow incidents, an email digest, and Jira and ServiceNow tickets) and what mmune accepts (probe heartbeats, discovery agent submissions, and change data capture, available on request). It ends with a table that tells you which mechanism to pick for a given job.

Prerequisites: a running mmune installation you can reach (the examples use https://mmune.example.com), an account with the write permission to configure channels, and an HTTPS endpoint of your own that can receive a JSON POST. Placeholders such as <TOKEN> and <SECRET> stand for real values you must not commit. REST conventions (authentication, error envelopes, rate limits, pagination) are defined in the API overview and are not repeated here.

Three things are worth knowing before you read on.

What you might expect	What mmune does
A registration API where you subscribe a URL to named events	There is no subscription API. You receive pushes through the webhook notification channel, which has one URL for the whole installation.
Signed webhook deliveries	Deliveries are not signed. The webhook channel can attach one static token header that you choose. Use that header together with a network allow-list on your receiver.
Subscribing to mmune's internal activity from outside	There is no network subscription. Use the notification channels, the REST API, the cursor endpoint or WebSocket.

Which mechanism to use

I want to	Use	Notes
Get a JSON POST in my own system when mmune detects drift, an unreachable connection, a quarantine or an SLA breach	The webhook notification channel	One URL per installation. Up to 3 attempts, spaced a fraction of a second apart (a receiver that hangs can stretch one delivery to about 30 seconds). See Outbound alert webhook.
Page someone	The pagerduty channel	Needs an Events API v2 routing key.
Post to a chat room	The slack or teams channel	Incoming webhook URLs.
Open a ticket automatically	An alert rule with the create_ticket action, or the opt-in servicenow channel	Jira and ServiceNow only. See Ticketing hand-offs.
Get a daily summary by email	The email_digest channel	Needs SMTP settings.
Send mmune alerts to Splunk, Datadog or another platform	Receive the webhook and forward it	See Forwarding to other platforms.
Read current state or history whenever I need it	Poll the REST API	See the API overview and health and monitoring.
Know that something changed so I can refetch	Poll GET /api/v1/events/cursor	Returns three counters. Cheap to poll every 10 seconds. See Polling for changes.
Show live status in a UI I own	Subscribe over WebSocket	See websockets.
Let an AI agent ask questions about estate health	The mmune-trust MCP server	See Using mmune from an AI agent.
Tell mmune that a remote probe node is alive	POST /api/v1/probes/heartbeat	Authenticated with X-Probe-Key. See Probe heartbeats.
Submit discovery results from a network mmune cannot reach	The discovery agent endpoints	See Discovery agent submissions.
Have mmune react immediately to row-level changes in Postgres	Change data capture	Available on request. See Change data capture.

Outbound alert webhook

How a webhook gets sent

When mmune detects something, it applies your alert rules (suppress, raise severity, route to named channels, create a ticket), applies the optional severity floor, checks whether the finding is held by a containment quarantine, and then sends the alert to each channel that is enabled and configured. A channel that is disabled or has no endpoint is skipped without any network call.

The channels are slack, webhook, pagerduty, servicenow, teams and email_digest. The generic JSON webhook is the one named webhook.

Register your endpoint

Registering is a channel configuration, not a subscription. You set the URL and turn the channel on. The call needs a token with the write permission.

curl -s -X PUT https://mmune.example.com/api/v1/notifications/channels/webhook \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true, "endpoint_url": "https://alerts.example.com/mmune/alerts"}'

The response confirms the stored state. The URL itself is never returned, only whether one is set.

{
  "message": "Channel configuration saved",
  "config": {
    "channel": "webhook",
    "enabled": true,
    "configured": true,
    "endpoint_url_set": true,
    "source": "db",
    "circuit_breaker": {"state": "closed", "consecutive_failures": 0}
  }
}

The call replaces both fields, so sending it without endpoint_url clears the saved URL. An environment-set URL still applies; the response reports only the saved value. You can also set the URL from the environment of the backend with MMUNE_ALERT_WEBHOOK_URL. A saved value wins over the environment.

The static auth header is environment-only. You cannot set it through the API.

Variable	Default	Effect
MMUNE_ALERT_WEBHOOK_URL	unset	Endpoint used when no value is saved through the API.
MMUNE_NOTIFICATION_WEBHOOK_ENABLED	channel on	Enable flag used when no value is saved through the API. Accepts 1, true, yes, on.
MMUNE_ALERT_WEBHOOK_AUTH_TOKEN	unset	If set, mmune sends this value as a request header on every delivery.
MMUNE_ALERT_WEBHOOK_AUTH_HEADER	Authorization	Name of the header that carries the token.

The token value is sent exactly as written. If your receiver expects Bearer abc, put Bearer abc in the variable. Reads (GET /api/v1/notifications/channels and GET .../channels/webhook) need read.

What mmune sends

mmune makes a POST with a JSON body. The only headers it sets are Content-Type: application/json plus the optional auth header above. There is no event-type header, no delivery id header and no timestamp header. Everything you need is in the body.

The eight fixed fields come first, followed by additional detail fields of the alert at the top level of the same object.

Field	Type	Meaning
source	string	mmune for findings, notifications_api for test sends.
event	string	What produced the alert. Values are listed below.
integration_id	string or null	The integration the finding belongs to.
severity	string	Free-form. The values in use are info, low, warning, medium, high, critical.
message	string	Human-readable description.
title	string	Short title, mmune <drift_type> for findings.
detected_at	string or null	ISO 8601 timestamp of detection, when the detector supplied one.
alert_id	string	Stable id for the logical alert. Use it as your idempotency key.
drift_type	string	Detail field. Present on finding alerts. For example structural_change, value_drift, connection_unreachable.
object_name	string	Detail field. The table or object affected. May be empty.
field_name	string	Detail field. The column or field affected. May be empty.
impact_estimate	object	Detail field. Present when an impact estimate was attached. Carries tier, inputs (such as row_count and downstream_count), rationale and estimated_cost.

An illustrative body for a structural schema change on a Postgres integration. Values are illustrative.

{
  "source": "mmune",
  "event": "watchdog_structural_finding",
  "integration_id": "erp-postgres",
  "severity": "medium",
  "message": "structural schema diff (watchdog)",
  "title": "mmune structural_change",
  "detected_at": "2026-10-04T09:12:41.508231+00:00",
  "alert_id": "6f1c0d52-3c1e-4d8a-9a53-0f4f8f6c2b11",
  "drift_type": "structural_change",
  "object_name": "public.customers",
  "field_name": "email",
  "impact_estimate": {
    "tier": "high",
    "inputs": {"row_count": 1250000, "downstream_count": 7},
    "rationale": "7 downstream assets depend on this column",
    "estimated_cost": null
  }
}

To correlate connection loss and recovery in your own system, key on integration_id plus drift_type.

The event values

For findings, event is the producing component with _finding appended.

event	Sent when
watchdog_structural_finding	The watchdog sees a structural schema difference on a database integration.
watchdog_value_drift_finding	Column value statistics move outside their learned range.
watchdog_airflow_finding	An Airflow DAG finding.
watchdog_kafka_flow_finding	A Kafka consumer lag finding.
watchdog_flow_finding	A finding from flow detection, which carries a confidence and its signals.
watchdog_connection_finding	An integration becomes unreachable, or reachable again.
sap_alert_runner_finding	An SAP alert check stores and sends a finding.
reconciliation_finding	A reconciliation run produces a finding.
standing_pii_finding	A scan reports new standing PII findings.
containment_owner_notify	Assets owned by a team were quarantined downstream of an incident.
containment_recovery_summary	A recovery re-validation sweep finished.
alert	Incident SLA breach alerts, which use the default value.
test_delivery	A test send from POST /api/v1/notifications/channels/{channel}/test.

Metric threshold alerts go to Slack, Teams and the email digest only. They are never sent to the webhook channel.

Authenticating the sender

There is no signature scheme on outbound deliveries. mmune does not compute an HMAC, does not add a timestamp, and does not send a signature header. The only credential mmune can attach is the static header described above, so authentication is a shared secret that travels with every request. Terminate TLS in front of your receiver, compare the header value in constant time, and restrict the receiver to the network addresses your mmune installation sends from.

Python:

import hmac
import os

EXPECTED = os.environ["MMUNE_WEBHOOK_TOKEN"]  # the same value as MMUNE_ALERT_WEBHOOK_AUTH_TOKEN


def is_from_mmune(header_value: str | None) -> bool:
    return header_value is not None and hmac.compare_digest(
        header_value.encode(), EXPECTED.encode()
    )

TypeScript:

import { timingSafeEqual } from "node:crypto";

const expected = Buffer.from(process.env.MMUNE_WEBHOOK_TOKEN ?? "");

export function isFromMmune(headerValue: string | undefined): boolean {
  if (!headerValue) return false;
  const given = Buffer.from(headerValue);
  return given.length === expected.length && timingSafeEqual(given, expected);
}

Delivery behaviour

Aspect	Behaviour
Timeout	10 seconds total per attempt.
Success	Any HTTP status below 400.
Failure	Any status of 400 or above, a timeout, or a connection error. Client errors such as 404 are retried like any other failure.
Attempts	3 by default (MMUNE_ALERT_RETRY_MAX).
Backoff	Fixed pauses of 0.05 seconds, then 0.1 seconds, then 0.2 seconds for any further attempts, so about 0.15 seconds of waiting across 3 attempts. Each attempt also has its own 10 second timeout, so a receiver that hangs or times out can keep one delivery going for about 30 seconds. The attempts finish in well under a second only when the receiver fails fast.
After the last failure	The result is recorded as failed in the delivery log. A failed delivery is not sent again later.
Circuit breaker	After 5 consecutive failed deliveries (MMUNE_ALERT_CIRCUIT_FAILURES) the channel opens. Further alerts are skipped and logged as skipped_circuit_open. After 60 seconds (MMUNE_ALERT_CIRCUIT_RESET_SECONDS) one probe delivery is allowed. If it succeeds the breaker closes. Breaker state is kept per channel and resets when mmune restarts.
Concurrency	At most 4 deliveries in flight per channel (MMUNE_ALERT_MAX_INFLIGHT_PER_CHANNEL).
Ordering	No ordering guarantee. Alerts are sent as detections happen, and concurrent detections interleave. Do not assume detected_at is increasing, and note that it can be null.
Deduplication	mmune remembers recent alert_id values and drops a repeat. With MMUNE_ALERT_DEDUP_WINDOW unset, a repeated id is dropped until it falls out of the 512 most recent ids or mmune restarts. With the variable set to a number of seconds, a repeat is dropped only within that window.
Severity floor	If MMUNE_ALERT_MIN_SEVERITY is set to one of the values below, alerts under it are not sent to any channel. The ranks are info 0, low 1, warning and medium 2, high 3, critical 4. An unrecognized value means no floor.
Alert rules	A rule with the suppress action stops all outbound sends for matching findings. A route_channels rule restricts the send to the listed channels. See health and monitoring.
Containment	A finding held by an active containment quarantine is not sent to any channel. It is recorded as skipped_contained.

Because the retries come within a fraction of a second of each other (plus up to 10 seconds of waiting per attempt on a receiver that hangs) and the receiver gets no later chance, a receiver that is down for more than a moment misses the alert on the webhook channel. The delivery log (below) tells you which alerts were missed. If you cannot tolerate that, poll the REST API for open findings and incidents as a reconciliation step.

Idempotency for the receiver: treat alert_id as the key for one logical alert and expect the same id to arrive again after mmune restarts, because deduplication is not kept across restarts. The id is the finding's drift id when there is one, otherwise <integration_id>:<drift_type>:<object>:<field>.

Set MMUNE_ALERT_DEDUP_WINDOW (for example 300) so recovery alerts are delivered.

Test it and see what happened

Send a test through the channel:

curl -s -X POST https://mmune.example.com/api/v1/notifications/channels/webhook/test \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"message": "hello from mmune", "severity": "low"}'

{"channel": "webhook", "status": "ok", "attempts": 1, "error": null, "ok": true}

Your receiver gets event set to test_delivery, source set to notifications_api, alert_id set to test-webhook, title set to mmune notification test, and integration_id and detected_at null. A channel that is not configured answers 200 with status set to skipped_unconfigured or skipped_disabled.

Repeated tests are skipped while the duplicate-suppression rule applies; set a short window to test repeatedly.

To see what happened to real deliveries:

curl -s -H "Authorization: Bearer <TOKEN>" \
  "https://mmune.example.com/api/v1/notifications/deliveries?channel=webhook&status=failed&limit=20"

Each row has id, alert_id, channel, status, attempts, error and created_at. The status values are ok, failed, queued, skipped_unconfigured, skipped_disabled, skipped_deduped, skipped_contained and skipped_circuit_open. The error text has the auth token replaced with ***. Full endpoint details are in health and monitoring.

Receiver examples

FastAPI:

import hmac
import os

from fastapi import FastAPI, Header, HTTPException, Request

app = FastAPI()
EXPECTED = os.environ["MMUNE_WEBHOOK_TOKEN"]
seen: set[str] = set()  # use a durable store in production


@app.post("/mmune/alerts")
async def receive_alert(request: Request, authorization: str | None = Header(None)):
    if authorization is None or not hmac.compare_digest(
        authorization.encode(), EXPECTED.encode()
    ):
        raise HTTPException(status_code=401, detail="bad token")

    alert = await request.json()
    alert_id = alert["alert_id"]
    if alert_id in seen:
        return {"status": "duplicate"}  # still a 2xx, so mmune counts it as delivered
    seen.add(alert_id)

    # Hand off quickly. mmune waits at most 10 seconds for this response.
    print(alert["severity"], alert["event"], alert.get("integration_id"), alert["message"])
    return {"status": "ok"}

Express with TypeScript:

import express from "express";
import { timingSafeEqual } from "node:crypto";

const app = express();
app.use(express.json());

const expected = Buffer.from(process.env.MMUNE_WEBHOOK_TOKEN ?? "");
const seen = new Set<string>(); // use a durable store in production

app.post("/mmune/alerts", (req, res) => {
  const given = Buffer.from(req.header("authorization") ?? "");
  if (given.length !== expected.length || !timingSafeEqual(given, expected)) {
    return res.status(401).json({ error: "bad token" });
  }

  const alert = req.body as {
    alert_id: string;
    event: string;
    severity: string;
    integration_id: string | null;
    message: string;
  };
  if (seen.has(alert.alert_id)) {
    return res.status(200).json({ status: "duplicate" });
  }
  seen.add(alert.alert_id);

  console.log(alert.severity, alert.event, alert.integration_id, alert.message);
  return res.status(200).json({ status: "ok" });
});

app.listen(3000);

Return a 2xx for duplicates. A 4xx or 5xx is counted as a failure and mmune retries it immediately.

Forwarding to other platforms

mmune does not send findings, metrics or logs directly to Splunk or Datadog. If you need those platforms to see mmune alerts, run a small receiver, such as the examples above, that takes the webhook body and forwards it in the format the platform expects.

Other notification channels

These use the same retry behaviour, circuit breaker and delivery log as the webhook channel. Configure them with the same PUT /api/v1/notifications/channels/{channel} call. What endpoint_url holds, and the environment fallback for each, is in the table in health and monitoring.

Channel	What mmune sends	Notes
slack	POST to an incoming webhook URL with text and one section block containing the title and message, with an emoji by severity.	10 second timeout.
teams	POST of an Office 365 MessageCard to an incoming webhook URL, with facts for severity, affected object, integration and field.	Colour by severity. Not interactive.
pagerduty	POST to https://events.pagerduty.com/v2/enqueue with event_action set to trigger, dedup_key set to the alert id, and a payload with summary (cut to 1024 characters), source, severity and custom_details.	Mapped severities: critical to critical, high and error to error, medium and warning to warning, low and info to info. The channel only triggers. It does not send resolve events.
servicenow	Creates an incident through the ticketing configuration.	Off by default. See Ticketing hand-offs.
email_digest	Alerts are queued and a scheduled job sends one HTML and text email through SMTP.	The queue accepts the alert and reports queued. Needs MMUNE_SMTP_HOST and recipients in MMUNE_SMTP_TO. Optional: MMUNE_SMTP_PORT (587), MMUNE_SMTP_USE_TLS (true), MMUNE_SMTP_USER, MMUNE_SMTP_PASSWORD, MMUNE_SMTP_FROM (mmune@localhost), MMUNE_EMAIL_DIGEST_INTERVAL_HOURS (24). If SMTP fails, pending alerts stay queued and carry into the next window.

Ticketing hand-offs

mmune can create tickets in Jira or ServiceNow. Configure one provider with POST /api/v1/ticketing/config (provider, base_url, credentials, default_project or default_table, close_ticket_on_resolve), check it with POST /api/v1/ticketing/test, and create a ticket on demand with POST /api/v1/ticketing/tickets. The endpoint reference is in health and monitoring.

Two things create a ticket without a person asking, and a third lets an operator do it by hand.

1. An alert rule with the create_ticket action. When a finding matches, mmune creates a ticket and links it to the open incident. The ticket description includes the top downstream assets from lineage and, when MMUNE_FRONTEND_BASE_URL is set, a link to /alerts.
2. The servicenow notification channel. It is off until you enable it (MMUNE_NOTIFICATION_SERVICENOW_ENABLED=true or a saved channel with enabled true). Saving ticketing credentials alone does not turn it on. Each incident gets correlation_id set to mmune:<alert_id>, and mmune looks that id up first, so a re-fired alert does not create a second incident.
3. On demand, from the console or with POST /api/v1/ticketing/tickets.

Status flows back too. mmune checks linked tickets every 300 seconds by default (MMUNE_TICKET_SYNC_INTERVAL_SECONDS, minimum 30) and resolves the mmune incident when the ticket reaches a terminal state. When an operator resolves an incident that has a linked ticket, mmune adds a comment to the ticket and, if close_ticket_on_resolve is true, tries to move it to done. This polls your Jira or ServiceNow. It does not need an inbound webhook from them.

Inbound

Probe heartbeats

A probe node is a remote process that reports that it is alive. The endpoint is exempt from the bearer token check and authenticates with a shared secret instead.

Item	Value
Endpoint	POST /api/v1/probes/heartbeat
Header	X-Probe-Key: <SECRET>, which must equal the MMUNE_PROBE_KEY environment variable on the server
Body	node_id (required), hostname (optional), capabilities (object, optional)
Success	200 with {"status": "ok"}
Errors	503 if MMUNE_PROBE_KEY is not set on the server, 401 if the header is missing or wrong

curl -s -X POST https://mmune.example.com/api/v1/probes/heartbeat \
  -H "X-Probe-Key: <SECRET>" \
  -H "Content-Type: application/json" \
  -d '{"node_id": "probe-dc1-01", "hostname": "dc1-probe-01", "capabilities": {"discovery": true}}'

The first call for a node_id creates a record, and later calls update hostname, capabilities and last_seen_at. GET /api/v1/probes lists nodes, newest first, and is a normal bearer-token read. mmune does not send heartbeats on your behalf, so whatever runs on your probe node has to send them.

Discovery agent submissions

A discovery agent is a process you run inside a network that mmune cannot reach directly. An admin registers it with POST /api/v1/discovery-agents/register, which returns a registration_token. The agent then sends Authorization: Bearer <registration_token> to its own routes: POST /api/v1/discovery-agents/{agent_id}/heartbeat, GET and PUT .../config, POST .../discoveries (discovered integrations, optionally with schemas) and POST .../kafka-flow-samples (consumer lag readings). The token only works for its own agent_id. Request and response shapes are in integrations and discovery.

mmune does not accept inbound webhooks from SaaS products. If you need another system to tell mmune about a change, use the discovery agent endpoints above, or change data capture.

Change data capture

Change data capture is available on request. Ask the mmune team to enable and configure it for your installation.

Polling for changes

GET /api/v1/events/cursor returns three integers named findings, lineage and health. A counter goes up when something in that domain changes (detections, alerts, health signals, incident and containment changes, lineage edits). Compare it with the last value you saw and refetch the matching REST data only when it moved. It carries no content. It needs a bearer token like every other route.

curl -s -H "Authorization: Bearer <TOKEN>" https://mmune.example.com/api/v1/events/cursor