Production Runbook

Start here only once you already have a working local brain — scaffolded, started, and answering in the dashboard chat. If you do not, do getting-started.md first; nothing below replaces it, and every step here assumes the repo it produces.

End-to-end checklist for putting that brain on a fresh server. Linear narrative; the deep reference for each step lives in operations.md — this doc only sequences them.

Assumes:

• The brain lives in its own GitHub repo (separate from engine; EaaF model).
• An engine release is already published to GHCR (ghcr.io/omnislash-ai/engine:X.Y.Z). If none is published yet, ask the engine maintainer to cut a release first.
• A fresh Linux host is reachable as root@<server-ip> (Hetzner default), at least 4 vCPU / 8 GB RAM / 80 GB disk — the minimum server the engine’s production limits are sized for. A smaller host works with the overrides in operations.md § Server size; a larger one can raise them there.

Steps 1–3 are account setup and dominate the clock. From step 4 on it is a sequence of commands, each of which exists in omnislash --help or in gh.

1. Provision Postgres

Pick one — managed container or external (BYOI):

• External (recommended for solo brains) — create a Supabase project at https://supabase.com → copy the pooled connection string. You’ll paste it into Infisical as DATABASE_URL in step 2. The engine auto-detects BYOI and excludes the managed-postgres container. TLS handling: see operations.md § Database (managed vs external).
• Managed container — skip; you’ll set POSTGRES_PASSWORD in Infisical instead, and the compose includes a postgres service.

For external Redis the same dichotomy applies (REDIS_URL vs REDIS_PASSWORD).

2. Infisical project + machine identity

In Infisical:

1. Create a project (e.g. outreach-cat), default env prod.
2. Fill the prod environment with the secrets listed in operations.md § Infisical (OPENAI_API_KEY, WEBHOOK_SECRET, the two CONNECTIONS_* keys, DATABASE_URL from step 1, and the TELEGRAM_* trio if this brain has a Telegram persona).

   The three generated keys — WEBHOOK_SECRET, CONNECTIONS_TOKEN_ENC_KEY, CONNECTIONS_STATE_HMAC_KEY — already exist in your local .env, minted by omnislash init. Copy those exact values into Infisical. Minting new ones for production is fine only on a brain with no live OAuth connections; once there are any, a different CONNECTIONS_TOKEN_ENC_KEY orphans every stored token.

3. Create a Universal Auth machine identity (docs) with access to that prod env.
4. Copy down:
   • INFISICAL_PROJECT_SLUG (e.g. outreach-cat)
   • INFISICAL_PROJECT_ID
   • INFISICAL_ENV_SLUG (prod)
   • INFISICAL_CLIENT_ID
   • INFISICAL_CLIENT_SECRET

3. GitHub tokens

One token, a repo secret you set in step 5:

• OMNISLASH_PAT — a classic PAT with the read:packages scope. CI’s pnpm install reads @omnislash-ai/* from npm.pkg.github.com with it, and the image build pulls the private engine base image with it. Registry-access detail: brain-dev-guide.md § Setup.

There is no second token, and nothing GHCR-shaped is ever stored on the server. The server never pulls the engine base image — its layers arrive inside your brain image, built in CI. The one pull on the host is your own brain image, and the deploy script logs in for it on every deploy with the workflow’s automatic GITHUB_TOKEN — then logs out again in its cleanup trap.

Watch the trailing newline when pasting via GitHub UI — set it with printf '%s' '<token>' | gh secret set <NAME> to avoid \n poisoning the auth header.

4. Server setup

One command, from the brain root:

cd my-brain
omnislash init-server <server-ip>

The host must be Debian or Ubuntu, amd64 or arm64 — Docker and Node.js are installed from the vendors’ own apt repos and the Infisical CLI from a per-architecture release tarball, and nothing else is served by all three.

It derives what it can from this brain — the project name, and whether the host kernel needs relaxing for bwrap isolation — asks for the step-2 machine identity if INFISICAL_CLIENT_ID / INFISICAL_CLIENT_SECRET are not already in your .env, prints the plan, and on your confirmation brings the host up to the baseline: base packages, swap, Docker, the app user with docker access, Node + the Infisical CLI, ~/.infisical-credentials, a CI deploy key, the brain-sync identity key, /opt/<project> + /opt/caddy-extra + ~/repos, a self-signed cert in webhook IP-mode, and a firewall. Every step is idempotent — re-running it on a live host is safe, and is how you converge a host you changed by hand.

Before the confirmation the plan also prints the host’s cores, RAM and disk and how they measure against the minimum server. Below it, set the overrides it names (they are the first row of operations.md § Server size) in Infisical before step 6’s first push.

Flags cover everything it cannot derive — --project, --ssh-user (default root), --app-user (default omnislash), --swap (GB, default 4; 0 leaves the host’s swap exactly as it is), --bot-mode polling|webhook, --webhook-port (default 8443), --domain, --no-github, --yes. omnislash init-server --help lists them.

When it finishes, your .env carries PROD_SERVER_HOST and PROD_PROJECT_DIR, and most of step 5 is already done. The private deploy key is never printed — it travels from the host into the repository secret without passing through your terminal or any log. What each step of the baseline does, and how to do it by hand instead, is in operations.md § Server setup.

Still manual after the baseline: agent CLI auth — claude auth login / codex login --device-auth interactively (the shorter spellings are the TUI’s and never authenticate), and only if some role in your constitution/models.json actually resolves to a CLI harness. A scaffolded brain runs every engine system role on the API chains, so this is opt-in. Easiest path: deploy the brain first, then cd my-brain && omnislash auth-agents from local. See operations.md § Agent CLI auth.

5. Configure the brain GitHub repo

Create the repo (e.g. github.com/<org>/<brain-name>) and push the brain omnislash init committed for you.

Step 4 already wrote these — the six entries it had the values for, plus the brain-sync deploy key:

Step 4 also printed the host key’s SHA256: fingerprint. If your provider shows host-key fingerprints (serial console / system log), compare them now: the provisioning connection was a first contact, and a mismatch means something answered in the server’s place — rotate the Infisical machine identity and re-run.

If gh was missing, logged out, or the repo did not resolve, init-server printed each of these as a command instead of writing it — run those, or install gh, gh auth login and re-run omnislash init-server <server-ip>.

The rest only you have. Each row is the command:

printf, not echo: a token pasted from a browser carries a trailing newline, echo adds one of its own, and a newline inside an Authorization header is what turns a valid PAT into a baffling 401.

Then verify the whole set before you push:

omnislash doctor

The Secrets section carries a GitHub Actions config check that reads the names the workflows in your repo actually reference and compares them against what the repo holds — so a missing value shows up here rather than in a red deploy. It follows ownership: once you create a deploy.yml.override / ci.yml.override sentinel, it grades your committed workflow instead of the one the engine would have written.

Full reference: operations.md § GitHub configuration. GITHUB_TOKEN is automatic; no setup needed.

The workflows that consume these already exist in your repo: omnislash init generated .github/workflows/deploy.yml and ci.yml, and every omnislash upgrade regenerates them. They authenticate to the engine registry through actions/setup-node, which is why your committed .npmrc carries no token line — see contract.md § 1.

6. First push

git push                          # → CI builds image → rsync → docker compose up
gh run watch                      # follow deploy

The schema needs no manual step, managed or BYOI: the deploy runs a one-shot db-migrate service before any app service, applying the engine migration line and then yours. Watch it in the deploy logs; a failing migration stops the deploy rather than starting apps against a half-applied schema.

Every deploy after this one is a few minutes of downtime. The deploy stops the world: it stops every engine service (the queue worker first finishes the jobs it is running), migrates, then starts the new version — nothing of the previous version runs beside the new one (operations.md § Brain CI/CD). Deploy when a few minutes without the bot and the dashboard are acceptable.

To check from your machine (optional):

infisical export --env=prod --format=dotenv | grep DATABASE_URL
DATABASE_URL=postgresql://... omnislash db status

Declaring your own tables and generating migrations: database.md.

If — and only if — some role resolves to a CLI harness, run cd brain && omnislash auth-agents from your local machine after the first green deploy to interactively log in Claude Code / Codex on the host (mounted into worker containers via ~/.claude and ~/.codex).

Sanity-check before declaring done

• gh run watch shows the deploy job green.
• ssh omnislash@<server-ip> 'docker compose -f /opt/<project>/docker-compose.prod.yml ps' shows worker, scheduler, monitoring, webhooks, dashboard healthy (plus bot if this brain has a Telegram persona).
• The dashboard answers, and a new chat there gets a streamed reply — the same check that ended getting-started.md, now against production.
• Bot responds to /start in Telegram — Telegram brains only.
• Grafana reachable via ssh -L 3001:localhost:3000 omnislash@<server-ip> → http://localhost:3001 (or https://grafana.<domain> in domain mode — the vhost sits behind the same ADMIN_USER / ADMIN_PASS_HASH basic auth as every other admin subdomain, so the browser asks for those first and Grafana’s own login second; the tunnel path bypasses Caddy and needs only the Grafana login).

Day-2 ops (update brain code, update engine version, refresh agent auth, backups, log access): operations.md § Day-2 Operations.

Alerting & auto-remediation

Production ships Grafana Unified Alerting preconfigured. Every alert fans out to two receivers, each at its own cadence, grouped by alert name:

• The engine’s notice gate (humans) — the alert POSTs to the webhooks app (POST /engine/alerts, authorized with WEBHOOK_SECRET), and each alert becomes one notice about its fingerprint in TELEGRAM_STATUS_CHAT_ID: posted once, Grafana’s every-interval resend folded into it (↻ ×N since …), marked ✅ Resolved when Grafana resolves it. Critical is a failure, a warning news — so notify.level: failures keeps warnings out of the chat while the resolver below still hears them.
• Engine webhook (auto-resolver) — the alert payload POSTs back into the deployment’s webhooks app (POST /event/grafana, authorized with WEBHOOK_SECRET). Resolved events are delivered too.

Three rules — the bot, the webhooks app and Redis down — carry notify: direct and also go straight to Telegram (plain text, sent by the chief’s bot; its tokenEnv is what the deploy renders into the contact point): the engine cannot carry news of its own delivery path’s outage.

Cadence: critical alerts group within 10s and repeat every 1h; warnings group within 1m (30m between group updates) and repeat every 12h. repeat_interval on the resolver route is the re-summon rate — a persistent alert re-spawns the remediation agent once per interval, which is why warnings deliberately keep the slower 12h cadence. Have the resolver pass the alert’s fingerprint as notification_send’s subject, and its re-triage of the same alert replaces its earlier report instead of posting a new one.

Brains with no chief’s bot — no Telegram surface (bots: []), or every bot on another role — need no Telegram credentials for alerting: the deploy automatically strips the direct Telegram contact point and its route from the emitted Grafana provisioning, and the notice gate posts nowhere, so alerting degrades to resolver-only instead of crash-looping on empty placeholders.

Auto-remediation pipeline

The resolver leg flows through the generic ingest pipeline:

Grafana  →  POST /event/grafana                      (Bearer WEBHOOK_SECRET)
         →  defineEventSource('grafana').extract()   (brain config)
         →  Redis dedup + external-events audit trail
         →  ingest event                             (source=grafana, type=alert:firing)
         →  Hook: grafana-alert.hook.md              (transport)
         →  Agent: necromancer.agent.md
                   + skill: disaster-recovery

The remediation agent consults its disaster-recovery skill for principles, decision procedure, diagnostic matrix, and runbooks — then decides whether to auto-fix (idempotent actions), open a PR (gh pr create) for code/config changes, or escalate to a human via ask_user_question (see human-input.md).

Responsibility split — keep it clean when authoring yours:

• Hook — transport only: carries the Grafana payload and names the agent.
• Agent — identity only: capabilities, model, skill attachment.
• Skill — the whole disaster-recovery procedure: principles, decision tree, tool matrix, runbook catalog, guardrails, report format.

Brain projects wire this up at standard paths:

• {brainRoot}/omnislash.config.ts — ingest.sources + the Grafana extract()
• {brainRoot}/ingest/grafana.ts — webhook body → event shape
• {brainRoot}/features/<feature>/grafana-alert.hook.md — hook (transport; any feature directory)
• {brainRoot}/features/<feature>/necromancer.agent.md — agent role (identity; keep it in a feature you do not switch off)
• {brainRoot}/skills/disaster-recovery/SKILL.md — skill (procedure + runbooks)

Scope: all severities reach the resolver. The agent is the triage layer — it carries the alert catalog with per-rule runbooks, a hard blocklist and default-deny, so pre-filtering by severity in the alert routing only hid rules from it. Narrowing scope again is a routing-config change (drop a route in the Grafana notification policy), no code.

Dedup: the brain’s extract() keys idempotency on groupKey + status + hourBucket — the first fire per hour triggers the resolver, re-fires within that hour are dropped. That collapses duplicate POSTs; the actual re-summon rate is the route’s repeat_interval above.

Delivery-doctrine background (one message per owner, silence on clean success): notifications.md.