Alerting

Talaia alerts through Pushover. Configure PUSHOVER_TOKEN and PUSHOVER_USER; with no Pushover credentials configured, alerts are logged to stdout only (handy locally).

Semantics

• Normal/high suites use in-run retries (default: 1 retry after 30 seconds).
• Emergency suites rerun once after a 60-second wait following the first failed attempt. This replaces ordinary retries. Still failing → emergency 2; recovered → a normal 0 short-outage warning.
• up → down: one Pushover alert at the suite’s configured priority, listing the failing tests and the first error lines, with a link back to the status UI (TALAIA_PUBLIC_URL).
• While down: re-alert at most every alert.cooldown_minutes (default 60, set per suite in its suite.yml). Repeat alerts show how many consecutive failures the suite has accumulated. Escalation from a lower delivered priority bypasses cooldown.
• down → up: cancel any emergency repeats, then send a normal 0 recovery message.

Priority configuration

Set the policy in each suite’s suite.yml. All tests in that suite share it:

alert:
  priority: 2                    # 0 normal, 1 high (default), 2 emergency
  cooldown_minutes: 60
  emergency_retry_seconds: 60    # Pushover repeats, not test reruns; minimum 30
  emergency_expire_seconds: 1800 # repeat lifetime; 1..10800 seconds

Existing suites without an explicit priority retain high priority 1. Emergency messages repeat until acknowledged, expired, or cancelled on detected recovery. Defaults are one repeat per minute for up to 30 minutes; Pushover caps repeats at 50. Acknowledgement stops that message’s repeats, but does not mute future still-down reminders from Talaia. Recovery is detected on the next suite run, not by Pushover’s repeats.

Confirmation reruns the whole suite with the same safety gates, even if retries: 0. Both attempts count toward the footprint, and wall duration includes the one-minute wait; allow for this in budget.max_duration_ms. A short outage is recorded as a passed run with two attempts and a summary of the initial failure. An already-down suite receives the usual recovery message instead of an extra short-outage warning.

Failed delivery does not start a cooldown. Pending emergency cancellation, recovery delivery and short-outage warnings survive in SQLite for retry on later runs. Keep the data volume across restarts.

If a suite declares a version_url, the monitored app’s deployed version is fetched after each run and tagged on runs and alerts — so an alert can answer “which deploy caused this?”.

Missing configuration is an error, not a silent skip

Env vars listed in a suite’s env.required that are unset on the server cause the run to be recorded as error — visible in the UI and metrics — rather than the suite quietly testing nothing.

These configuration/harness errors notify at low priority -1, including for emergency suites; they do not use emergency confirmation. A configuration error during confirmation does not establish recovery or a confirmed emergency. Runtime errors such as suite timeouts use the suite’s selected failure priority. Footprint-budget warnings also use -1, with their own daily cooldown.

Heartbeats (dead-man switch)

Alerting on failures only helps while Talaia itself is running. For each suite you can set heartbeat_env in suite.yml to the name of an env var holding a healthchecks.io ping URL. After every run Talaia pings it — the plain URL on a pass, the /fail endpoint on a failure — so healthchecks.io pages you if the pings stop arriving (scheduler dead, host down, cron misconfigured), not just when a check goes red.