Writing a suite

A suite is a directory under projects/<project>/<suite>/ (any nesting depth) containing a suite.yml and plain pytest test_*.py files. The suite id is the path relative to projects/ — e.g. bikecrm/backend.

Adding a suite

1. Copy projects/_template/ to projects/<project>/<suite>/.
2. Edit suite.yml (name, schedule, required env vars) and set enabled: true.
3. Write test_*.py pytest tests. Fast (<60s), independent, idempotent.
4. Add the suite’s env vars to .env.example (documented, with placeholders) and the server’s .env (real values).
5. uv run --env-file .env talaia run <project>/<suite> to verify, then redeploy.

suite.yml reference

Run these steps from backend/. A missing schedule leaves a suite manual-only. The configuration defaults are shown below; the copyable template deliberately sets enabled: false until you finish editing it:

name: Template suite            # display name in the UI (default: suite id)
enabled: true                  # disabled suites are skipped by `run --all`, crontab and the UI
schedule: "*/15 * * * *"        # cron expression consumed by `talaia crontab` (supercronic)
timeout_seconds: 300            # hard kill per pytest attempt
retries: 1                      # in-run retries before the run counts as failed
retry_delay_seconds: 30
budget:
  max_requests: 100
  max_duration_ms: 60000
  max_http_ms: 45000
alert:
  priority: 1                   # 0 normal, 1 high (default), 2 emergency with 60s confirmation
  cooldown_minutes: 60          # min gap between repeat alerts while the suite stays down
  emergency_retry_seconds: 60   # Pushover repeat interval, minimum 30
  emergency_expire_seconds: 1800 # repeat lifetime, maximum 10800

env:
  required: []                  # env vars that must be set; missing -> run recorded as "error"
    # - MYPROJECT_BASE_URL
    # - MYPROJECT_USERNAME
    # - MYPROJECT_PASSWORD

# heartbeat_env: HC_URL_MYPROJECT   # env var holding a healthchecks.io ping URL (dead-man switch)
# version_url: https://api.example.com/api/version/   # GET'd after each run; tagged on runs/alerts

Two optional extras:

• heartbeat_env names an env var holding a healthchecks.io ping URL. Talaia pings it after every run — the plain URL on a pass, the /fail endpoint on a failure — so healthchecks.io alerts you both when the suite goes red and when Talaia stops running at all.
• version_url is fetched (GET) after each run; a version/release field from a JSON response (or the plain response body) is recorded on the run and included in alerts, so an alert can answer “which deploy caused this?”.

Surface checks and flow checks

A surface suite checks public health/version endpoints and page shells using product-specific markers. Keep env.required: [] and default URLs in the test module, with optional environment overrides. This gives a credential-free baseline. A shell check does not prove login or checkout works. Add dedicated test-account flows to make those additional claims.

See Alerting for confirmation and Footprint for budgets and schedule limits. Handwritten pytest is trusted code: suite authors must implement safety gates and cleanup; the runner is not an arbitrary-code sandbox.

Conventions

• A suite is a leaf directory. Never put a suite.yml in an ancestor of another suite — the parent’s pytest run would collect the child’s tests.
• Black box only. Suites run with cwd = the suite directory and must not import anything from the monitored project. API and browser access only.
• Fast, independent, idempotent. Keep the whole suite under ~60 seconds; each pytest attempt is killed at timeout_seconds.
• Env var naming: <PROJECT>_<THING> (e.g. BIKECRM_BASE_URL). Declare them in env.required so a misconfigured server shows up as an explicit error run instead of a confusing failure.
• Stagger schedules between suites of the same project (e.g. 0,30 vs 5,35) so they do not hammer the app at the same instant.

Safety rules (these tests hit production)

• Test accounts only. Every suite authenticates against a dedicated test account/tenant. Never point a suite at a real customer account.
• Prefix and clean up. Any entity a test creates must be named with a SMOKE- prefix and deleted by the same test when possible.
• Gate every write flow. Before exercising any write against production, assert the expected sandbox conditions explicitly (for example: the test business slug is the expected one and irreversible modes are disabled). A suite must refuse to run against anything else.
• Quarantine irreversible flows. Flows that leave undeletable data belong in low-frequency (e.g. daily) suites, never in the every-10-minutes ones.
• No secrets in the repo. Credentials come from env (.env on the server, gitignored). Document every new var in .env.example with a placeholder.