Monitoring & Alerting

Umari exposes Prometheus metrics so you can watch the runtime in production and get paged when a projector or effect stops running or falls behind. The metrics are pull-based (you scrape them), scoped to liveness and lag, and designed around one goal: catching silent failures before they become data problems.

This chapter covers the /metrics endpoint, the metrics it exports, a ready-made vmui dashboard, and vmalert rules that notify you over Discord.

The /metrics endpoint

The runtime serves metrics in Prometheus text format at:

GET /metrics

Authentication

/metrics is part of the authenticated API surface. When UMARI_API_KEY is set (as it should be in production), scrapers must send the bearer token:

curl -H "Authorization: Bearer $UMARI_API_KEY" http://localhost:3000/metrics

Without the header the endpoint returns 401. Keep this in mind when configuring your scraper; see Scraping below.

Collection interval

Two kinds of metrics are exported:

• Event-driven metrics (failures, restarts, backoff, progress timestamps) are recorded inline as things happen and are always available.
• State-derived gauges (up, positions, lag) are refreshed by a periodic collector.

Control the collector with:

Metrics reference

Every metric is labelled with module_type (projector, effect, or command) and name (the module name). umari_module_info additionally carries version.

Two design facts shape how these are used:

• Lag is query-aware. A projector or effect only subscribes to the events matching its query. A narrowly-subscribed module that is fully caught up legitimately sits far below the global event-store head, so lag is measured against each module’s own query head (umari_module_query_head_position), not the global head. This avoids false lag. Alert on umari_module_lag, never on head - last_position.
• Projectors and effects fail differently. Projectors do not auto-restart: a dead projector stays down (umari_module_up == 0) until reactivated, which is the primary silent-failure case. Effects do auto-restart with exponential backoff, so a crash surfaces as climbing umari_module_restarts_total and umari_module_backoff_seconds rather than a permanent down state.

Scraping with VictoriaMetrics / Prometheus

Point your scraper at /metrics and pass the API key as a bearer token. A vmagent or Prometheus scrape config looks like:

scrape_configs:
  - job_name: umari
    scheme: https
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials: "your-umari-api-key"
    static_configs:
      - targets: ['ops.example.com']

vmalert (below) queries VictoriaMetrics, not the runtime directly, so the bearer token only needs to live in the scrape config.

Dashboard (vmui)

A prebuilt dashboard for vmui lives at docs/monitoring/umari-runtime.json. Load it by pointing VictoriaMetrics at the directory containing it:

victoria-metrics --vmui.customDashboardsPath=/path/to/dashboards

It appears under the Dashboards tab in vmui. The dashboard has five rows:

1. Health — modules currently down (umari_module_up == 0), and active/down counts per type.
2. Lag & freshness — modules behind their query head, and time since last progress.
3. Failures & restarts — recent failures, effect restarts, and backoff.
4. Event store & throughput — ingestion rate, head position, per-module progress rate.
5. Deployments — modules per running version, so you can correlate incidents with rollouts.

The panels follow a “plot problems, not inventory” approach: filters like == 0 and > 0 mean most panels are empty when everything is healthy and light up with exactly the offending module when something breaks. This also keeps them under vmui’s per-panel series limit.

Two things to know:

• vmui renders only line graphs. There is no stat, gauge, or table panel type. For those, use Grafana with VictoriaMetrics as a Prometheus data source.
• The overview panels assume a single runtime instance. Aggregations like sum(...) by (module_type) drop the scraper’s instance label. If you run multiple runtime instances, add instance to the by(...) clauses so they don’t merge.

Alerting with vmalert + Discord

vmalert evaluates alerting rules against VictoriaMetrics and sends firing alerts to Alertmanager, which delivers them to Discord via its native receiver.

Alerting rules

Save these as umari-alerts.yml and pass them to vmalert with -rule:

groups:
  - name: umari-runtime
    # Evaluate at least as often as the runtime's metrics-interval (default 15s).
    interval: 30s
    rules:
      # The runtime (or its /metrics endpoint) stopped reporting entirely.
      # umari_event_store_head_position is always present while the runtime is up,
      # so its absence means the scrape target is down.
      - alert: UmariRuntimeUnreachable
        expr: absent(umari_event_store_head_position)
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Umari runtime is not reporting metrics"
          description: "No umari metrics have been scraped for 3m. The runtime process or its /metrics endpoint is down."

      # A projector died. Projectors do NOT auto-restart, so any sustained down
      # state needs manual reactivation. This is the primary silent-failure alert.
      - alert: UmariProjectorDown
        expr: umari_module_up{module_type="projector"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Projector {{ $labels.name }} is down"
          description: "Projector {{ $labels.name }} has been down for 2m. Projectors do not auto-restart and must be reactivated."

      # An effect is down for longer than its max backoff (~10m), meaning it is
      # not recovering on its own. Transient restarts are expected and won't fire.
      - alert: UmariEffectNotRecovering
        expr: umari_module_up{module_type="effect"} == 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Effect {{ $labels.name }} is not recovering"
          description: "Effect {{ $labels.name }} has been down for 15m, longer than its max restart backoff. It is stuck."

      # An effect keeps crashing and restarting. Catches crash loops without
      # firing on a single transient restart.
      - alert: UmariEffectCrashLooping
        expr: increase(umari_module_restarts_total{module_type="effect"}[15m]) > 5
        labels:
          severity: warning
        annotations:
          summary: "Effect {{ $labels.name }} is crash-looping"
          description: "Effect {{ $labels.name }} restarted {{ $value | humanize }} times in the last 15m."

      # A module is a long way behind its own (query-aware) event head.
      # Tune the 1000 threshold to your event volume.
      - alert: UmariModuleLagging
        expr: umari_module_lag{module_type=~"projector|effect"} > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.module_type }} {{ $labels.name }} is lagging"
          description: "{{ $labels.module_type }} {{ $labels.name }} is {{ $value | humanize }} events behind its query head and has stayed behind for 10m."

      # A module has been continuously behind its query head for 15m without ever
      # catching up: genuinely stuck rather than just slow. Uses min_over_time
      # instead of a lag threshold because lag is sampled from two independent
      # sources (query_head advances the instant an event is appended;
      # last_position advances a beat later, once the module delivers, handles,
      # and commits it), so a healthy module momentarily reads a lag of 1 or 2.
      # A healthy module drains back to 0 between events, so its minimum lag over
      # any 15m window is 0; a stuck one never drains and its minimum stays > 0.
      # This needs no magic threshold and is unaffected by different modules
      # sitting at different positions, since lag is measured per query head.
      - alert: UmariModuleStalled
        expr: min_over_time(umari_module_lag{module_type=~"projector|effect"}[15m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.module_type }} {{ $labels.name }} is stalled"
          description: "{{ $labels.module_type }} {{ $labels.name }} has been continuously behind its query head for 15m without catching up (min {{ $value | humanize }} events behind)."

      # A module recovers but keeps dying: flaky, worth investigating even though
      # it is currently up.
      - alert: UmariModuleFlapping
        expr: increase(umari_module_failures_total[1h]) > 3
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.module_type }} {{ $labels.name }} keeps failing"
          description: "{{ $labels.module_type }} {{ $labels.name }} died {{ $value | humanize }} times in the last hour."

The rules map directly onto the failure model described above:

• UmariRuntimeUnreachable — the runtime or its /metrics endpoint stopped reporting entirely.
• UmariProjectorDown — a projector has been down for 2m. Projectors don’t self-heal, so this is critical.
• UmariEffectNotRecovering — an effect has been down longer than its max backoff, so it is genuinely stuck.
• UmariEffectCrashLooping — an effect keeps restarting.
• UmariModuleLagging / UmariModuleStalled — a module is far behind its query head, or has stayed continuously behind for 15m without ever catching up.
• UmariModuleFlapping — a module recovers but keeps dying.

The thresholds (lag > 1000, restart/failure counts) are starting points — tune them to your event volume.

Alertmanager Discord receiver

Alertmanager has a native Discord receiver (v0.25.0+), so no bridge is needed. Save this as alertmanager.yml and replace the webhook placeholder with your channel’s webhook URL (Discord: Channel → Edit → Integrations → Webhooks → New Webhook → Copy URL):

route:
  receiver: discord
  group_by: ['alertname', 'module_type', 'name']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Re-notify critical alerts more aggressively.
    - matchers:
        - severity="critical"
      receiver: discord
      repeat_interval: 1h

receivers:
  - name: discord
    discord_configs:
        # Paste your Discord channel webhook URL here, or use webhook_url_file to
        # load it from a file so it stays out of version control:
        #   webhook_url_file: /etc/alertmanager/discord_webhook
      - webhook_url: 'REPLACE_WITH_DISCORD_WEBHOOK_URL'
        send_resolved: true
        title: '{{ if eq .Status "firing" }}🔥 FIRING{{ else }}✅ RESOLVED{{ end }}: {{ .CommonLabels.alertname }} ({{ .Alerts | len }})'
        message: |-
          {{ range .Alerts -}}
          **{{ .Labels.severity | toUpper }}** — {{ .Annotations.summary }}
          {{ .Annotations.description }}
          {{ end }}

Wiring it together

Run Alertmanager and vmalert alongside VictoriaMetrics. A docker-compose sketch:

services:
  alertmanager:
    image: prom/alertmanager:latest
    command: ['--config.file=/etc/alertmanager/alertmanager.yml']
    volumes:
      - ./monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro

  vmalert:
    image: victoriametrics/vmalert:latest
    command:
      - -rule=/etc/vmalert/umari-alerts.yml
      - -datasource.url=http://victoriametrics:8428   # VM that scrapes umari
      - -notifier.url=http://alertmanager:9093
      - -remoteWrite.url=http://victoriametrics:8428   # persist alert state across restarts
      - -remoteRead.url=http://victoriametrics:8428    # restore it on boot
    volumes:
      - ./monitoring/umari-alerts.yml:/etc/vmalert/umari-alerts.yml:ro

-remoteWrite/-remoteRead are optional but recommended: without them a vmalert restart drops all in-flight alert for: timers.

Using Matrix instead

Alertmanager has no native Matrix receiver. To deliver to Matrix, run a bridge — matrix-hookshot with a generic webhook, or the matrix-alertmanager relay — and point a webhook_configs receiver at it instead of discord_configs. The alerting rules stay identical.