Brain Operations: Deploy & Day-2

The deploy/ops reference for a brain: one-time server setup, secrets, CI/CD, runtime persistence, and day-2 operations. production-runbook.md is the linear runbook that sequences this; this doc is the reference it links into. It starts from a brain that already runs locally — getting-started.md is how you get one. Registry access and engine upgrades live in brain-dev-guide.md; the runtime contract (package shape, G/R/W paths) in contract.md; your own tables and migrations in database.md.

How it works

Engine release (by the engine maintainer)
  → ghcr.io/omnislash-ai/engine:X.Y.Z  (private base image, read with OMNISLASH_PAT)

Brain repo (push to main)
  → GitHub Actions:
      1. docker build (FROM engine base) → ghcr.io/<org>/<brain>:sha
      2. Infisical → env vars (DATABASE_URL etc. for BYOI detection)
      3. omnislash compose --prod --image sha → .omnislash/docker-compose.prod.yml
      4. rsync .omnislash/ → server
      5. ssh: infisical export → .env, docker compose pull && up -d

.env is generated on the server from Infisical at each deploy — never stored in CI or git.

Server size

Minimum recommended server: 4 vCPU / 8 GB RAM / 80 GB disk (Hetzner CX32 or equivalent), with the 4 GB swap omnislash init-server creates by default. The engine’s production limits (deploy.resources in its compose templates) are sized for it:

• every memory ceiling is at most half its RAM, so one runaway service cannot take the memory the rest of the stack needs;
• every CPU ceiling is at most its cores, and every engine service (bot, worker, pipelines, scheduler, monitoring, mcp, webhooks, dashboard, brain-sync, the migration one-shot) gets at least one core: each boots by stripping its whole TypeScript module graph, a deploy cold-boots them all at once, and an interactive service at a quarter core turned one second of CPU into four of waiting;
• Redis’ container limit stays a third above its maxmemory, and Restate’s RocksDB budget at most half its container.

A limit is a ceiling, not a reservation: an idle service costs nothing under it, and added up the default stack’s ceilings exceed the minimum server several times over — by design. What has to fit the host is what the stack actually uses; the host panel of health_overview shows it.

The engine checks a host against this minimum in two places: omnislash init-server prints the host’s cores, RAM and disk before you confirm the plan, with the overrides to set when it is below; and the Host resources doctor check (doctor_check, the dashboard’s Doctor page) grades the host the deployment runs on and every container’s real ceilings — warning below the minimum, and on any memory ceiling that is missing or at or above the host’s RAM, which can never bite. Whether a service sits at its CPU ceiling is the Container CPU At Limit alert’s.

By server size

AGENT_EXECUTION_CONCURRENCY and PIPELINES_MEMORY move together: every concurrent CLI agent is a subprocess inside pipelines (~250 MB idle, more while it works), so the container must hold the concurrency plus its coordinator.

Server-size knobs

Every limit worth changing with the server is an env knob, read by compose on the host at every deploy. Set it in the production env (Infisical — the .env the deploy exports); it takes effect on the next deploy. A service or a field without a knob is changed in docker/compose.prod.yml instead (docker/compose.prod.yml.example shows the shape, brain-dev-guide.md § Production troubleshooting the merge rules) — the knob is the first choice because it needs no file.

The observability stack (Grafana, Loki, Prometheus, Alloy, cAdvisor, the exporters) and the one-shots keep fixed limits: they are sized by the engine’s own telemetry, not by the brain’s work. Change one through docker/compose.prod.yml if you must.

Server setup (one-time)

cd my-brain
omnislash init-server <server-ip>

Idempotent, one command, run from the brain root: base packages, swap, Docker (Docker’s apt repo, signing key pinned by fingerprint), the app user, Node, the Infisical CLI (pinned version, sha256-verified) + credentials, the CI deploy key, the brain-sync identity key, /opt/<project> + /opt/caddy-extra + ~/repos, optional self-signed TLS, and UFW. It derives the project name and the bwrap-isolation decision from omnislash.config.ts, takes everything else as a flag, reads the host’s cores, RAM and disk before you confirm and grades them against the minimum server (§ Server size — below it, the plan names the overrides to set before the first deploy), and finishes by writing what it produced into the brain’s .env and into the GitHub repo (see § GitHub configuration). Re-running it on a live host is safe — every step below skips if it is already done. Already-provisioned hosts keep the Docker they have — the Docker step skips when docker is on PATH, so this hardens hosts provisioned from here on.

The host must be Debian or Ubuntu on amd64 or arm64. Docker and Node.js come from the vendors’ own apt repos and the Infisical CLI from a per-architecture release tarball; none of the three serves anything else, and get.docker.com — which used to paper over derivatives — is gone. Anything outside that set is refused before the run touches the host, naming what is missing; install those three by hand and re-run and the rest of the baseline proceeds normally.

The manual equivalents are below, as reference for a host you are converging by hand.

Docker

ssh <user>@<server-ip>
DISTRO=$(. /etc/os-release && echo "$ID")                                  # ubuntu or debian
SUITE=$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL "https://download.docker.com/linux/$DISTRO/gpg" -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Must print 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 — the fingerprint the engine
# pins. Without this the key is trusted because TLS said so, and apt then trusts
# every package the repo ever serves.
gpg --show-keys --with-colons --fingerprint /etc/apt/keyrings/docker.asc | awk -F: '$1=="fpr"{print $10; exit}'

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/$DISTRO $SUITE stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update -o APT::Update::Error-Mode=any
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

The docker group is root on this host. A member can run docker run -v /:/host, which mounts the host filesystem into a container it controls — there is no narrower grant that still lets docker compose up apply a compose file with host bind mounts, so the deploy account genuinely needs it. The consequence travels with every credential that opens that account: the DEPLOY_KEY secret below is a root credential on this server, and so is anything that can read your repository’s secrets. Branch protection on the default branch is part of the deployment contract, not a preference.

This is the accepted trust model: the host is the security boundary. Run one brain per server (or VM) — never several brains or clients on one machine — since anything that reaches the deploy account owns the whole host.

omnislash init-server does the same, and enforces the fingerprint check rather than printing it — see bootstrap.sh §0. Error-Mode=any matters: without it apt-get update answers a source it could not fetch with a warning and exit 0, leaving a broken docker.list behind to fail every later apt-get on the host. (The -o form, not the --error-on=any flag it backs — that flag is apt 2.1.16+, and older apt rejects an unknown long option outright while ignoring an unknown -o key.)

SSH key for CI deploy

ssh-keygen -t ed25519 -f deploy_key -N ""
cat deploy_key.pub >> ~/.ssh/authorized_keys   # add public key to server
# add the private key as the DEPLOY_KEY secret in GitHub repo settings
cut -d' ' -f1,2 /etc/ssh/ssh_host_ed25519_key.pub   # the host's own key → DEPLOY_HOST_KEY variable
# CI verifies the server against DEPLOY_HOST_KEY on every deploy — there is no keyscan

Brain repo access (GitHub App)

Steady state: the brain repo needs no key of its own. It authenticates the same way every other connected repo does — the deploy’s GitHub App mints a repo-scoped installation token that expires in an hour, delivered in the environment (GH_TOKEN plus GIT_CONFIG_*) and never spliced into a remote URL.

Ordering constraint — the first deploy cannot use the App. The App is registered through an agent’s git_request_app card or the dashboard, and both (like every other app service) wait on brain-sync: service_healthy, which brain-sync reaches only after it has cloned the brain repo. So a fresh server bootstraps on the ssh lane below and moves to the App afterwards. The same applies to the commit identity: GIT_AUTHOR_NAME / GIT_AUTHOR_EMAIL are derived from the registered App, so until one exists they must be set in .env — brain-sync fails fast at boot with no identity from either tier.

First deploy (no App yet):

1. Generate the server’s ssh identity and register its public half as a deploy key with write access on the brain repo — omnislash init-server generates it, registers the public half through gh, and prints the key itself when it could not. By hand:

   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""   # default name → ssh auto-uses it
   # register ~/.ssh/id_ed25519.pub on each ssh remote; on GitHub that is
   # repo → Settings → Deploy keys → Add, with "Allow write access" CHECKED
   # (brain-sync pushes, and a deploy key is read-only by default)
   ssh-keyscan -t ed25519 -H github.com >> ~/.ssh/known_hosts   # avoid first-pull prompt

2. Set BRAIN_GIT_REMOTE to the ssh form (git@github.com:<owner>/<repo>.git) and set both GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL in .env. Deploy.

Then move it onto the App:

3. Ask an agent to register the App (git_request_app — it posts one card whose link goes through GitHub’s registration screen and on to its install screen), or use Dashboard → Connections › Repos (/connections/repos). Install it on the account that owns the brain repo; if you install with “Only select repositories”, make sure the brain repo is one of them. git_overview then shows the brain repo’s lane.

4. Change BRAIN_GIT_REMOTE to the https form (https://github.com/<owner>/<repo>.git) and redeploy. GIT_AUTHOR_* may now be dropped — the pair is derived from the App as <slug>[bot]. An ssh:// or git@… remote stays legal; it just keeps the brain repo on the ssh lane.

   The redeploy is load-bearing, not incidental: brain_state and brain_baseline are named volumes that survive it, so the checkouts inside them still carry the ssh URL they were first cloned with. brain-sync’s bootstrap re-points both at the new BRAIN_GIT_REMOTE on the way up — and because every sandbox inherits the baseline’s origin, that repair reaches agent checkouts too. A checkout that is a different repository (a repo rename, a re-pointed remote) is deliberately left alone and reported instead.

5. Run the doctor and read the Brain git remote row. Run it from inside the deploy — docker compose exec brain-sync omnislash doctor — because only that vantage can see the two volumes; from the host the row says so.

   • App-served (installation token) — done, nothing else to set up.
   • App-served (installation token) — no brain checkout visible from here — the lane is proven, but this vantage has no brain_state / brain_baseline to assess. Re-run it inside brain-sync.
   • brain checkout still on the pre-App remote — the configuration moved but a checkout did not: brain-sync’s pull/push and every sandbox clone still run on the host ssh key, and no installation token is minted for them. Fix: docker compose up -d --force-recreate brain-sync.
   • brain checkout is not the repository BRAIN_GIT_REMOTE names — the tree is a different repository (or has no origin at all). The engine will not re-point it. Back up anything unpushed and clear the volume so brain-sync re-clones; if you renamed the repo on GitHub, that is what this is.
   • no installation covers <owner> — the App is not installed there (or the brain repo was not selected); brain-sync cannot clone, pull or push.
   • no GitHub App is registered for this deploy — there is no App yet to install anywhere; ask an agent to register it (git_request_app), or use the dashboard’s Connections › Repos (/connections/repos).
   • brain remote is on the ssh lane — the remote is not an https GitHub URL.

The ssh lane is permanent, not just a bootstrap step: it serves every remote no installation token can authenticate — an ssh BRAIN_GIT_REMOTE, attached repos on ssh or non-GitHub hosts, and git: skill sources. The key lives in the server’s ~/.ssh (which the containers mount read-only). Inside the agent jail ~/.ssh is masked with an empty tmpfs and unmasked only for spawns whose checkout no App token can serve.

No GHCR login on the server. The server never pulls the engine base image — its layers arrive inside your brain image, built in CI. The one pull on the host — your own brain image — is authenticated per-deploy by deploy.sh with the workflow’s automatic GITHUB_TOKEN, whose cleanup trap logs out again. A standing docker login ghcr.io on the host would be both redundant and erased by the first deploy.

Deploy directory

mkdir -p /opt/<project-name>

CI rsyncs .omnislash/ contents (compose, caddy, monitoring, scripts, n8n) here. Must match DEPLOY_DIR in GitHub variables.

For caddy-enabled brains (the default), also pre-create the extra-vhost dir, owned by the deploy user:

install -d -o <deploy-user> -g <deploy-user> /opt/caddy-extra/sites.d /opt/caddy-extra/www

It’s the bind source for caddy’s /etc/caddy/extra mount (admin-supplied extra vhosts + static roots), deliberately a sibling of the deploy dir so rsync --delete never touches it — which puts it in root-owned /opt, so it must be created root-side. deploy.sh verifies it on every deploy. Brains with services.caddy: false skip it entirely.

Code repos

Repos beside the brain are attached at runtime, not declared in env:

mkdir -p ~/repos                              # host side of the volume
omnislash git attach <owner>/<repo>           # writes the row; --slug renames it (same attach on the dashboard's Connections › Repos, /connections/repos)
omnislash git repos                           # the attached rows + their sync state, then what the installations can reach

Container volume: ~/repos → /repos. Each attached repo is cloned to /repos/<slug> — by brain-sync at boot for repos already attached, and by the repo-sync sweep for repos attached later, so an attach needs no redeploy. A clone or fetch failure is written to the row and printed by omnislash git repos (sync_status / last_error). omnislash doctor reports a repo whose checkout is missing, and fails any scenario whose agent.repo names a slug that is not attached.

Infisical

All secrets are managed via Infisical. .env is generated from Infisical at each deploy — never edited manually on the server.

1. Create a project (e.g. my-project), default env prod.
2. Add secrets to the prod environment.

Four variables are hard-required for every brain — OPENAI_API_KEY, WEBHOOK_SECRET, CONNECTIONS_TOKEN_ENC_KEY, CONNECTIONS_STATE_HMAC_KEY — and the Telegram vars join them only when the config declares a bot persona: TELEGRAM_STATUS_CHAT_ID, TELEGRAM_ALLOWED_IDS and each bot’s tokenEnv (TELEGRAM_BOT_TOKEN for the roster synthesized when bots is omitted). That list is not maintained by hand: your brain’s own .env.example, generated from the live engine env schema by omnislash init, is the authoritative copy, and an uncommented NAME= line in it is a variable this brain must supply. Everything below the required rows is deployment-shaped.

3. Create a Universal Auth machine identity with access to prod — you’ll get client-id and client-secret.

Database (managed vs external)

Same dichotomy for Redis: managed → REDIS_PASSWORD; external → REDIS_URL=redis://... (redis container excluded).

BYOI detection is automatic: omnislash compose --prod reads env vars (from Infisical in CI) and excludes managed containers when external URLs are present.

TLS verification. DATABASE_SSL_REJECT_UNAUTHORIZED defaults to auto: Supabase pooler (*.pooler.supabase.com), Supabase direct (*.supabase.co), and Neon (*.neon.tech) hosts are auto-relaxed because their intermediate certs are not in Node’s bundled CA store. Other hosts stay strict. Override with true/false, or point DATABASE_SSL_CA_PATH at a CA bundle (e.g. AWS RDS global-bundle.pem) to keep strict verification with a self-managed cert chain.

Server: install the Infisical CLI

The CLI lives on the server and exports .env from Infisical on every deploy. One-time setup:

ssh <user>@<server-ip>

# The release tarball is the only channel with anything to check the bytes
# against: the apt/deb repo is outdated, and npm's package downloads this same
# tarball from a preinstall script that verifies nothing. The release's own
# `checksums.txt` is unsigned and comes from the same origin as the tarball, so
# it catches a corrupted transfer and nothing else — the digests below are the
# ones the engine pins in `bootstrap.sh` §0, reviewed and in git.
V=0.43.128
A=$(dpkg --print-architecture)          # amd64 or arm64
case "$A" in
  amd64) SHA=a3f460be321ad46fefba99cba883bcc601d0f18b02849d2d30ae9b398a8d99dc ;;
  arm64) SHA=e4fffc20191135c539ab0b086a7a5e8b8bee08db3defde172aa380c768210628 ;;
  *) echo "no pinned digest for $A" >&2 ;;
esac
curl -fsSL -O "https://github.com/Infisical/cli/releases/download/v${V}/cli_${V}_linux_${A}.tar.gz"
echo "${SHA}  cli_${V}_linux_${A}.tar.gz" | sha256sum -c -
tar -xzf "cli_${V}_linux_${A}.tar.gz" infisical
sudo install -m 0755 infisical /usr/local/bin/

cat > ~/.infisical-credentials << 'EOF'
INFISICAL_UNIVERSAL_AUTH_CLIENT_ID=<client-id>
INFISICAL_UNIVERSAL_AUTH_CLIENT_SECRET=<client-secret>
EOF
chmod 600 ~/.infisical-credentials

Verify (this is exactly what CI does on each deploy):

set -a && source ~/.infisical-credentials && set +a
INFISICAL_TOKEN=$(infisical login --method=universal-auth \
  --client-id=$INFISICAL_UNIVERSAL_AUTH_CLIENT_ID \
  --client-secret=$INFISICAL_UNIVERSAL_AUTH_CLIENT_SECRET \
  --domain=https://eu.infisical.com --silent --plain)
infisical export --token=$INFISICAL_TOKEN --domain=https://eu.infisical.com \
  --projectId=<your-project-id> --env=prod --format=dotenv

You should see your secrets printed as KEY=VALUE. To update: omnislash upgrade to an engine version whose pin is newer, then re-run omnislash init-server — the step is version-guarded, so it reinstalls. By hand: repeat the block above with the newer version and the digest that engine version pins.

Agent CLI auth

Worker containers run Claude Code and Codex as child processes. Auth on the host, mounted into containers via volumes:

npm i -g @anthropic-ai/claude-code @openai/codex
claude auth login           # prints a browser URL — paste the code back
codex login --device-auth   # prints a URL plus a one-time code

Those exact spellings matter: a bare claude login is forwarded to the interactive TUI as a prompt and never authenticates, and codex auth is the TUI form. They are the same argv the engine’s own sign-in spawns.

Mounted auth dirs: ~/.claude → /home/node/.claude, ~/.codex → /home/node/.codex, ~/.ssh (read-only, for the ssh lane — mounted into the container, masked inside the agent jail), ~/.gitconfig. Refresh from local with cd brain && omnislash auth-agents (see Day-2 Operations).

OpenCode ships in the image too, as a third CLI harness, but takes no login step and no mount: it reads provider API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, …) from the deployment’s .env, which the container already carries. omnislash doctor reports it as a warn rather than a pass — there is no credential file to inspect.

Setting those keys for OpenCode is safe for routed turns: a <harness>:<router>:<model> turn blanks ANTHROPIC_API_KEY (Claude) or overrides OPENAI_API_KEY with the router’s own credential (Codex) for that subprocess. Without it a key you set here would win — Claude Code reads ANTHROPIC_API_KEY before ANTHROPIC_AUTH_TOKEN — and silently send routed traffic to the direct vendor under a model name only the router knows.

DB migrations

No manual step, managed or external. The compose always renders a one-shot db-migrate service — it consumes DATABASE_URL when you have one and simply drops the local-postgres dependency. It runs before bot/worker/scheduler and applies the engine migration line (PG schema omnislash) first, then yours (db/migrations/, PG schema public).

Each migration commits in its own transaction with its journal row inside it, so a failure resumes exactly where it stopped. A failing migration exits non-zero, the deploy stops, and your apps keep running the previous image instead of crashlooping against a half-applied schema.

To apply ahead of a deploy, or to check state:

infisical export --env=prod --format=dotenv | grep DATABASE_URL
DATABASE_URL=postgresql://... omnislash db status    # applied / pending, per line
DATABASE_URL=postgresql://... omnislash db migrate   # apply pending

The db-migrate service is not excluded when DATABASE_URL is set — only omnislash start --skipDb removes it.

Writing your own migrations: database.md.

Webhook TLS

Default delivery is polling (no extra setup). For webhooks, two modes:

Domain mode (recommended)

Auto-TLS via Let’s Encrypt. Set in Infisical:

PUBLIC_DOMAIN=yourdomain.com
ACME_EMAIL=admin@yourdomain.com
TELEGRAM_WEBHOOK_URL=https://bot.yourdomain.com
TELEGRAM_WEBHOOK_SECRET=<random-string>

DNS: create A records for bot., app., grafana., bullboard., nocodb., n8n., restate., webhooks., artifacts. subdomains → server IP (a single *.yourdomain.com wildcard is simplest). Open ports 80 + 443 (ufw allow 80/tcp && ufw allow 443/tcp). Do NOT set TELEGRAM_WEBHOOK_CERT_PATH — Telegram verifies the CA-signed cert via public CA. Caddy auto-provisions and persists certs in the caddy_data volume.

Admin basic auth — in domain mode the admin dashboards (dashboard, Grafana, Bull Board, NocoDB) become publicly reachable, so Caddy enforces basic auth. The webhooks. vhost is deliberately not behind it — GitHub, your event sources and OAuth providers call it server-to-server; every route there authenticates itself (shared WEBHOOK_SECRET bearer, a source’s own verify, the GitHub App’s HMAC) and caps the request body before reading it:

docker run caddy:2-alpine caddy hash-password --plaintext 'your-password'

ADMIN_USER=admin
ADMIN_PASS_HASH=$2a$14$...   # output from caddy hash-password

Without these, protected vhosts reject all requests.

IP mode (self-signed cert)

For servers without a domain:

mkdir -p certs
openssl req -newkey rsa:2048 -sha256 -nodes \
  -keyout certs/webhook.key -x509 -days 3650 \
  -out certs/webhook.pem -subj "/CN=<SERVER_IP>" \
  -addext "subjectAltName=IP:<SERVER_IP>"

TELEGRAM_WEBHOOK_URL=https://<SERVER_IP>:8443
TELEGRAM_WEBHOOK_SECRET=<random-string>

Caddy terminates TLS → bot runs plain HTTP behind the proxy.

Brain CI/CD

package.json + .npmrc

@omnislash-ai/* packages are declared once in peerDependencies with a registry-pin to the engine release (exact, e.g. 0.152.0). Brain commits a three-line .npmrc at the repo root — pnpm flags plus the engine registry scope, and no auth token. Full shape, the two pnpm flags, and why one declaration suffices: contract.md § 1. pnpm-lock.yaml is committed for local-install determinism only — the Docker image does NOT consume it (contract.md § 4).

Dockerfile + workflow files are engine-generated

omnislash upgrade generates the Dockerfile, .dockerignore, and .github/workflows/{deploy,ci}.yml, version-matched to the engine. Do not edit them manually — they’re overwritten on the next upgrade. Customizing the build: brain-dev-guide.md § Dockerfile Customization.

Two artifacts split the deploy responsibility:

This split is deliberate: the workflow file is a thin GitHub-Actions wrapper (run the brain gates, validate the deploy target, checkout, install, secrets, ssh, rsync, a single bash deploy.sh); the deploy flow itself — including auth (infisical login, docker login ghcr.io) and cleanup — lives in deploy.sh, which rides along in the rsync on every push. So engine fixes to the deploy flow reach you via the next omnislash compose --prod — no workflow regen. To own the workflow file end-to-end, drop an empty .github/workflows/deploy.yml.override sentinel; omnislash upgrade then leaves deploy.yml alone.

The two halves share one contract: the Deploy step pipes its parameters (GHCR_USER, GHCR_TOKEN, INFISICAL_DOMAIN, INFISICAL_PROJECT_ID, INFISICAL_ENV_SLUG) into deploy.sh on the ssh command’s stdin, one NAME=value line each, closed by ===OMNISLASH-DEPLOY-ENV-END===. They are not on the ssh command line, because that line is argv of a real process on your server and argv is world-readable through ps to every local account for the whole deploy — the GHCR token is a live registry credential. If you own deploy.yml via the .override sentinel you must mirror that Deploy step: a deploy.sh that receives no block fails loudly within 30 s telling you to run omnislash upgrade, rather than deploying half-configured.

Notes on the deploy flow:

• The deploy waits on your gates. deploy.yml’s first job is the same gates job ci.yml runs on pull requests — your type-check / lint / test scripts, then omnislash doctor --check pipelines (every enabled pipeline module imported and validated) — and the deploy runs only when it passes, manual dispatches included. A red gate leaves the host on the previous version and Deploy Drift reports the undeployed code. Details: brain-dev-guide.md § CI/CD.
• omnislash doctor --check env validates every required env var (engine + brain) before compose generation — missing vars fail the deploy fast. Its sibling --check pipelines runs earlier, in the gates of ci.yml and deploy.yml, on a runner with no secrets. --check engine needs no brain at all: it answers whether the installed engine itself is whole (every template its package shipped — system roles, eval methods, planner and loop prompts — is where its code reads them) — run it first when doctor fails role or eval-judge checks on a laptop but not on the host.
• infisical export refreshes .env on the host atomically (tmp file + mv).
• The deploy stops the world, so every deploy is downtime — a few minutes: the drain, the migrations, one boot. compose --prod writes the plan to .omnislash/deploy-plan (stop|kill|run <names> lines), and deploy.sh follows it: a service your release removed from the compose file stops first (it would otherwise keep running the old code until up removes it), then the services that accept new work (bot, webhooks, mcp, dashboard, scheduler, and any service of yours on the brain image), then the queue worker, which finishes the jobs it is running (up to 60 s; jobs still waiting stay queued for the new version), then pipelines. In-flight invocations of the engine’s own Restate services are killed (a kill Restate does not confirm fails the deploy before the migrations, and the previous version starts again), and the engine re-derives that work from its rows when it boots; your pipeline runs are not killed. Then brain-sync stops, the migrations run, and one docker compose up -d --wait --remove-orphans starts everything in depends_on order, in parallel, and waits for every healthcheck. Postgres, Redis, Restate, Caddy and the monitoring stack keep running throughout. Nothing of the previous version runs while the new one migrates or boots, so a release never has to stay compatible with the version before it (contract.md § 8).
• A failed deploy. A failure before the migrations starts the previous version again (its containers were only stopped). From the migrations on there is no rollback: a failed migration leaves the engine stopped until a fixed deploy, and a failed start leaves the new version up as far as it starts. The deploy log says which. A migration’s output lives in the deploy’s log — the run --rm container is gone afterwards.
• Caddy holds requests across a restart. Every reverse_proxy in the shipped Caddyfile retries a failed dial for up to 30 s (import upstream_restart), so a request that lands while its service is being replaced waits instead of getting a 502.

GitHub configuration

Repository variables (Settings → Secrets and variables → Actions → Variables):

Both DEPLOY_HOST and DEPLOY_DIR reach a shell on your server (cd $DEPLOY_DIR, and rsync’s remote path), so the deploy job validates them against those charsets in its first step and fails there — before checkout — on anything else. omnislash init-server writes conforming values; a hand-edit in the GitHub UI is what the check is for.

The deploy job pins the server to DEPLOY_HOST_KEY (StrictHostKeyChecking yes, known_hosts composed from DEPLOY_HOST + DEPLOY_HOST_KEY) instead of scanning it on every run — a fresh runner per deploy made that scan a trust-on-first-use with a new first use every time. The first step rejects an empty or non-key value up front.

Engine version is not a repository variable — it’s derived inside the workflow from peerDependencies["@omnislash-ai/cli"] in the brain’s package.json (the single source of truth, contract.md § 1). omnislash upgrade bumps that pin; the next deploy picks up the new base image automatically.

Repository secrets:

GITHUB_TOKEN is automatic — used for GHCR login (push brain image, pull engine base image).

omnislash init-server sets the six it produces itself — DEPLOY_HOST, DEPLOY_DIR, DEPLOY_HOST_KEY, DEPLOY_KEY, INFISICAL_CLIENT_ID, INFISICAL_CLIENT_SECRET — plus the brain-sync write deploy key, and prints the exact gh command for each of the rest. omnislash doctor’s GitHub Actions config check (section Secrets) then compares what the repo holds against the names these workflows actually reference, so the answer to “is CI configured” comes from the workflows rather than from this table — and once you take one over with a .yml.override sentinel, from your version of it.

Deploy

cd brain
git push    # → build image → generate compose → rsync → docker up
gh run watch

Manual deploy (alternative)

When CI/CD is not set up or you need to deploy by hand:

cd brain
omnislash build                                       # → <project-name>:latest
omnislash compose --prod --image <project-name>:latest
docker save <project-name>:latest | ssh $DEPLOY_HOST 'docker load'

# rsync configs; the `P` (protect) filters keep --delete from wiping host runtime state:
# artifacts/ (W-set), certs/ (self-signed webhook TLS), .env (regenerated from Infisical).
rsync -az --delete --force \
  --filter='P /artifacts/***' --filter='P /certs/***' \
  --filter='P /.env' --filter='P /.env.*' \
  .omnislash/ $DEPLOY_HOST:$DEPLOY_DIR/

# deploy.sh acquires INFISICAL_TOKEN internally from ~/.infisical-credentials on the host.
# Its parameters go on STDIN, never on the ssh command line — that line is argv of a real
# process on the server and `ps` shows it to every local account. The end marker is
# mandatory: without it deploy.sh reports the caller as out of date and stops.
printf '%s\n' \
  "GHCR_USER=$GHCR_USER" \
  "GHCR_TOKEN=$GHCR_TOKEN" \
  "INFISICAL_DOMAIN=$INFISICAL_DOMAIN" \
  "INFISICAL_PROJECT_ID=$INFISICAL_PROJECT_ID" \
  "INFISICAL_ENV_SLUG=$INFISICAL_ENV_SLUG" \
  '===OMNISLASH-DEPLOY-ENV-END===' \
| ssh $DEPLOY_HOST "cd $DEPLOY_DIR && bash deploy.sh"

deploy.sh logs in to GHCR itself to pull your brain image, so it wants a GHCR identity from whoever calls it. In CI those are github.actor + the automatic GITHUB_TOKEN; by hand, your GitHub username and a token with read:packages. It logs out again on every exit path — nothing is left behind on the host.

Runtime persistence

Brain is split by mode in production (full classification: contract.md § 2):

• R-set — image-baked, read-only at runtime (omnislash.config.ts, tools/, db/, agent configs). Changes only via image rebuild → redeploy.
• G-set — git-backed, lives in the brain_state named volume (memory/, constitution/, skills/, features/). Two-way synced with the brain git remote.
• W-set — writable host bind, not in git (artifacts/).

The engine repairs broken entity frontmatter in the G-set — and tells you when it did. If a *.heartbeat.md / *.scenario.md / *.hook.md / *.agent.md stops parsing — almost always an unquoted free-text value containing : , e.g. description: Weekly report: summarize the week — the scheduler quotes the offending scalar and writes the file back, on boot and on every content tick. Only files that fail to PARSE are touched — a file whose YAML is fine but whose fields the engine rejected is reported, never rewritten. brain-sync then commits that as an ordinary diff.

Frontmatter is YAML 1.2, read with the same parser the engine writes with. Two consequences worth knowing. A time-like value is a string: schedule: 12:30 is "12:30", not the number 750 that YAML 1.1 would have made it — which is what used to happen when the engine’s own schedule-writing tool set a time, quietly unscheduling the scenario it had just edited. And tabs are not indentation: a frontmatter block indented with a tab is a parse error rather than a document with the nested key silently reparented to the top level. Parse errors name the line and column of the break.

What it guarantees is narrow and worth reading precisely: the rewritten file parses. It does NOT guarantee the change means what you meant. description: Weekly report: summarize the week and agent: role: reviewer are the same YAML syntax, and quoting is right for the first and wrong for the second — nothing in the engine can tell them apart. That is exactly why every repair is announced to your status channel and lands in git as a revertable diff: review the diff, and revert it if the value was meant to be structured. A candidate repair the engine cannot prove parses is discarded and the file left untouched, and anything it cannot fix is reported too — a file that does not parse is an entity that silently never runs. Until it is fixed either way, a scheduled entity keeps firing on its previous schedule rather than dropping off the schedule entirely.

Git is the single source of truth for the G-set. The brain-sync daemon owns brain git: at bootstrap it sparse-checks-out the G-set and pulls; then every BRAIN_SYNC_INTERVAL_MS (default 5 min) it commits writer changes, pulls (rebase), and pushes. All app services see the G-paths via subpath mounts off the same volume; only brain-sync mounts .git.

Local dev                          Prod (brain_state volume, sparse-checkout)
    │  git push ──→  brain-sync: fetch + rebase every tick (memory, constitution,
    │                            skills, features)
    │                       agents write memory, dashboard edits features/* + constitution
    │  git pull  ←── brain-sync: commit(G-set) + push every tick

Required in Infisical: BRAIN_GIT_REMOTE (brain repo URL — https for the GitHub App lane, ssh for the key lane; enables sync), BRAIN_GIT_BRANCH (default main). Sync itself has no manifest toggle — the daemon fills brain_state before any constitution exists to read, so BRAIN_GIT_REMOTE is the switch. What MANIFEST.yaml does carry (editable by dashboard/agents, live — no restart): scheduler.enabled (the brake over scenario crons and heartbeats), reconcile, and the whole memory section. The brain’s timezone is not there: it is locale.timezone in omnislash.config.ts.

Conflict auto-resolution. When brain-sync’s rebase onto origin conflicts (you and an agent edited the same file), it aborts the rebase (local commits preserved), force-pushes those commits to the engine-reserved branch brain-sync/conflict, notifies the status channel, and spawns a single git-conflict-resolver agent in a clone of its own that rebases that branch onto yours and pushes. The daemon checks with git that the pushed tip is a new commit on your branch — made after the export — before adopting the result, then deletes the side branch. At most one resolver runs at a time. If the agent keeps failing you get a 🆘 naming the branch — fetch brain-sync/conflict, rebase it onto your branch and push; the next resolver run finds nothing left to replay, confirms your push and the episode closes.

Manual sync. No need to wait for the interval: tell your agent to sync — the brain_sync engine tool (the brain.sync op, capability brain:sync) commands the daemon to run a full tick (commit local brain changes → pull → push) immediately and reports the outcome. The dashboard has the same control: “Sync now” in the overview sync badge. One tick is bidirectional, so a single sync both picks up your fresh push and lands agent-written changes (memory, constitution) in the remote.

Volume map:

Rsync --delete safety: CI ships .omnislash/ to $DEPLOY_DIR/ with rsync --delete --force --filter='P /artifacts/*** /certs/*** /.env /.env.*'. The P (protect) filters keep --delete from wiping host-only runtime state. A brain compose override that adds a new host bind must be added to this protect list.

Day-2 Operations

Restarting by hand

One service:

cd $DEPLOY_DIR
docker compose -f docker-compose.prod.yml up -d --no-deps --wait <svc>

A new version by hand follows deploy.sh § 3: stop the services .omnislash/deploy-plan lists, in its order, then docker compose -f docker-compose.prod.yml run --rm -T db-migrate (blocks; non-zero on failure), then up -d --wait. Never migrate under running services: a release owes the previous version nothing. BullMQ jobs stay safe in Redis while the worker is down.

Update brain code

cd brain && git push                  # CI/CD
# manual: cd brain && git pull && omnislash build && omnislash compose --prod ...

Update engine version

An engine upgrade is an agent’s job: it runs omnislash upgrade (pin, install, regenerate the engine-owned files, write upgrade/TASK.md), then follows upgrade/TASK.md to the end: checks against production, one commit per entry, the gates, the deploy, the after-deploy steps. There is no upgrade workflow. The whole protocol, the deploy’s downtime and what to do when it fails: upgrading.md. The command’s own detail and the escape hatch when the CLI itself is broken: brain-dev-guide.md § Updating Engine.

cd brain
pnpm omnislash upgrade            # latest published engine (or: omnislash upgrade 0.152.0)
# then follow upgrade/TASK.md; its step 5 pushes, and the push deploys

A field a release makes required in models.json

The task’s gates run omnislash doctor before the deploy. upgrade converges engine-owned artifacts. It never touches constitution/models.json — that file is yours, and when an engine release makes a new field required there, the copy you already have is the one that needs a hand edit. The doctor is what tells you, naming the entry and the field:

pnpm omnislash doctor             # `Models file` fails: models.3.contextWindowTokens: expected number

contextWindowTokens is the current instance of this: every model entry needs its maximum input context window in tokens, taken from the vendor’s model docs. It is what sizes an agent’s static prompt prefix against the window on the dashboard’s Context budget page, and the ceiling the chat composer’s meter uses when the runtime reports none. There is no defensible default — a guessed window turns every percentage the engine shows into fiction — so the loader refuses a file without it instead of degrading to a blank number. Fill it in, commit, then deploy.

timeoutMs — when one of your models is legitimately slow

An entry may also declare timeoutMs: how long one attempt at that model may take before the engine abandons it and falls over to the next model in the chain. Omit it and the model takes the engine default of 15 000 ms.

Declare one when a model’s first output legitimately arrives late — a high-effort reasoning model can think for tens of seconds while emitting nothing the engine counts as output. Below that model’s real latency the engine cannot tell thinking from dead: it discards the generation (which your provider still bills), retries, and eventually falls over to a weaker model, on every single call. Raising timeoutMs on that one entry is the fix; nothing else in your file changes.

Two things to know before you pick a number:

• The unit is milliseconds. "timeoutMs": 45000 is forty-five seconds. The loader rejects anything under 1 000 precisely because "timeoutMs": 45 is an easy way to make every call to that model fail instantly.
• It multiplies with retries. A cheap model gets up to 3 attempts, so a 45 s timeout on one is up to 135 s of waiting before the chain moves on. Generous timeouts belong on models that are slow and expensive enough to be retried once, not on the cheap ones.

The value is yours per model — the engine has no opinion beyond its default, and no dashboard control renders this field, so it is a hand edit to constitution/models.json like the rest of the entry.

Credentials: what expires, and who rotates it

Every credential your deployment holds, where it lives, and whether anything expires. /system/credentials in the dashboard is the live version of this table — statuses, real expiry dates and a copy-pasteable remedy per row, with Fix with agent on every row that needs attention — and the Credentials check on /system/doctor is its alarm (worst-of, so it shows on the overview badge; its hint lists the same flagged rows).

Three things the dashboard does not probe, and how it answers instead of guessing:

• CI secret values and their expiry. GitHub discloses a secret’s value to nobody, and the brain image contains no workflow files to read (.github/ is excluded from the image). Those rows read unverified — not checked, and not checkable — with the command to rotate each one. They never turn the health badge yellow, because a signal that is always on is not a signal.

• Whether a plain API key still works. Probing an LLM key costs a request; the inventory reports set / not set and leaves liveness to the dashboard’s Providers page (/agents/models/providers).

• Whether an idle CLI harness can still log in. A harness refreshes its access token only when it runs, and checking from outside would itself be a refresh (which rotates the single-use token). So nothing probes it; a token that lapsed more than a day ago is graded by what the deployment’s own turn history recorded since — every agent step on a harness writes it down when that harness’s own sign-in is rejected:

  • no rejection — unverified: nothing has tried the harness, the normal state of a fallback in a chain like [claude, codex] that nothing has needed lately. The badge stays green;
  • a rejection — invalid, a failure: the login is dead, and a chain is quietly finishing that work on its next model. Sign the harness in again.

  expired (a failure) is kept for a session that can no longer mint tokens at all. A rejected sign-in also reaches you the moment it happens, as the chat notice described in the CLI-agent setup guide. A step routed through a model router signs in with the router’s key, not the harness login, and does not count.

The one credential you can delete rather than rotate. If the GitHub App covers every repository you have attached, GH_TOKEN is not serving anything — the App lane already mints repo-scoped tokens that expire in an hour and renew themselves. Ask an agent — git_overview names the lane of every repository the deploy works in, and while none reads ambient the PAT serves nothing — or open /connections/repos, where checkouts no App installation serves sit in their own section at the end. A PAT is still required for an ssh brain remote or a non-GitHub remote.

When “expiring soon” starts. Fourteen days ahead, by default. It is your call, not the engine’s:

// omnislash.config.ts
export default defineProject({
  credentials: { expiryWarningDays: 30 },
});

Refresh agent CLI auth

From chat, no shell needed. /connect → 🤖 Agent sign-in shows every harness’s session and signs one in where you already are: pick the harness, open the link it sends, and either reply with the code the page gives you (Claude Code) or type the one-time code into the page (Codex, which then finishes by itself). The engine re-reads the credential file afterwards and tells you what it holds. The agent:auth_issue notice that told you a credential died carries a Sign in button into this same flow.

The sign-in runs inside the pipelines container — the only one that may touch those files — as a Restate Virtual Object keyed by harness, so two people signing in the same harness at once serialize instead of invalidating the token family between them.

From the dashboard. Terminal → pipelines, then the harness’s native login (claude auth login, codex login --device-auth) — the terminal execs into the one container that may write those files.

From a terminal. omnislash auth-agents SSHes into the host, runs the same interactive login inside the pipelines container, and writes refreshed credentials to the host-mounted ~/.claude / ~/.codex. Whichever way you sign in, the next CLI agent spawn picks up the new auth — no compose restart needed.

omnislash auth-agents              # all CLIs declared in models.json (skips if the OAuth session is alive)
omnislash auth-agents claude       # only claude
omnislash auth-agents -f           # --force, re-login even if the session is alive

You only need this when the session (refresh token) is gone. A merely expired access token is normal — Claude/Codex refresh it themselves on the next agent spawn, and omnislash doctor shows it as a refreshes on next run warning, not a failure. Don’t re-login to “fix” that: an interactive login rotates the single-use refresh token and can knock out agents that are mid-run.

Reconnect a dead OAuth integration

An OAuth connection dies when the provider rejects its refresh token. You will get a chat message naming it:

🔌 Connection expired: strava — its refresh token was rejected…

Nothing is silently broken-but-pretending: every tool that needs that credential now fails with “No active connection for strava … it needs re-authorization”, and the dashboard’s Connections › Services row shows the dead status. Two ways to fix it — plus one that may fix it before you get there.

Before you start: it may reconnect itself. A provider declared with autoAuthorize: true in omnislash.config.ts gets a robot attempt first: the engine hands the authorize URL to a browser agent, which logs in with <PROVIDER>_LOGIN_EMAIL / <PROVIDER>_LOGIN_PASSWORD, clears an emailed one-time code if there is one, approves the consent screen, and completes the flow. If it works, the consent card below resolves itself and you never touch it. If it does not — wrong password, a CAPTCHA, no code arrived — nothing is lost: the card is still sitting there, and options A and B below work exactly as written. Robot authorization is best-effort by design.

A. Ask the bot (works with zero inbound ingress). Tell it to reconnect the integration. It calls hitl_request_connection and sends you a consent card with an authorization link.

1. Tap the link, authorize at the provider.
2. The provider redirects your browser to the engine’s callback URL. On a server with no public ingress that page will NOT load — an error, a blank page, “cannot connect”. That is expected and harmless. The authorization server never talks to your engine; it only bounced your browser.
3. Copy the full URL from the address bar (it looks like http://localhost:3200/oauth/callback?code=…&state=…) and reply to the consent card with it.
4. The bot forwards it to the engine internally and confirms: ”✅ Authorization completed”. The agent that was waiting resumes automatically.

The link is valid for 30 minutes while it is pending. But a paste spends it — the state nonce, the PKCE verifier and the card binding are all one-shot, and they are consumed even when the exchange then fails. So if a paste comes back with an error, do NOT re-open the same link (it can only fail again): fix what the error names and ask for a fresh card.

B. From the brain repo (--prod) — same flow, driven by the CLI on your own machine. It prints the authorize URL and then prompts you to paste the address bar back, exactly like the chat loop:

cd my-brain

# a brain-declared provider (always deployment-wide — `dash:default`)
omnislash connect provider strava --prod

# an MCP server — token lookups are PER-SUBJECT, so re-authorizing a dead row means
# naming its subject. --subject is REQUIRED with --prod: the default is evaluated
# inside the container, so omitting it would bind the row to `cli:node`.
omnislash connect mcp https://mcp.notion.com/mcp --subject tg:user:42 --prod

Under the hood --prod is exactly:

ssh -t $PROD_SERVER_HOST \
  "cd $PROD_PROJECT_DIR && docker compose -f docker-compose.prod.yml exec webhooks \
     omnislash connect provider strava"

• It needs PROD_SERVER_HOST + PROD_PROJECT_DIR in the brain .env — the same two vars omnislash auth-agents uses, over the same ssh seam, and the two omnislash init-server writes there for you.
• It must be the webhooks service: the CLI dials the gateway at WEBHOOKS_INTERNAL_URL, defaulting to http://localhost:3200, and only inside that container is the default the gateway itself. WEBHOOK_SECRET is in its env too.
• ssh -t + docker compose exec (never -T) — the paste prompt needs a TTY, and --prod carries one end to end. exec, never run: a --rm oneoff abandoned at the prompt leaks and keeps holding the service’s mounts.
• The raw ssh … docker compose exec … line above still works when you are already on the box.

The chat notice that announces a dead connection spells out the --prod command, subject included.

The exec’d CLI runs in the webhooks container’s cgroup. Engine ≥ the release carrying --prod raises that service’s compose limits (1G / 64 pids) to fit it — brains on an older image should omnislash upgrade and redeploy before leaning on --prod, or the paste prompt shares a 512M cap with the live gateway.

Registering the redirect_uri. A fresh no-ingress brain must still register a redirect URI at the provider, and it must string-match what the engine sends. Nothing ever FETCHES it, so any address works — use a loopback URL on a port that cannot collide with a local dev stack (omnislash dev binds 3200), e.g. http://localhost:53200/oauth/callback, and set WEBHOOKS_PUBLIC_URL to the matching origin. WEBHOOKS_INTERNAL_URL (what services actually dial) stays http://webhooks:3200.

Database: backups, SSL, local access

Backups (managed postgres). External managed DBs (RDS, Neon, Supabase) rely on the provider’s backups. For managed-container mode, a postgres-backup sidecar runs pg_dump on BACKUP_SCHEDULE (default @daily) and rotates by day/week/month (BACKUP_KEEP_DAYS/WEEKS/MONTHS). Dumps live on the postgres_backups volume. Run the first backup immediately after deploy — otherwise the sidecar healthcheck stays red and the postgres-backup-missing alert fires:

docker compose exec postgres-backup /backup.sh
docker compose exec postgres-backup ls /backups/last/

Restore (stream between containers, no host paths):

docker compose exec postgres-backup cat /backups/last/omnislash-latest.sql.gz \
  | gunzip | docker compose exec -T postgres psql -U omnislash omnislash

Run restore drills monthly — untested backups are hope, not backups.

SSL to remote postgres. DATABASE_SSL_REJECT_UNAUTHORIZED (default true) and DATABASE_SSL_CA_PATH (optional PEM bundle) control TLS. Local connections and ?sslmode=disable skip SSL. AWS RDS: curl https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem > rds-bundle.pem then set DATABASE_SSL_CA_PATH. Neon/Supabase/Cloud SQL use a system-trusted CA — leave it unset.

Local → prod DB access for migrations / Drizzle Studio / ad-hoc SQL:

There is no --prod flag: every omnislash db subcommand reads DATABASE_URL from the environment when it’s set.

# BYOI (managed DB) — whitelist your IP in the provider firewall first
export DATABASE_URL='postgresql://user:pass@prod-host:5432/omnislash?sslmode=require'
omnislash db status           # applied / pending migrations, per line
omnislash db migrate          # apply pending migrations
omnislash db studio           # Drizzle Studio
psql "$DATABASE_URL"

# Self-hosted prod (managed postgres in our compose) — via SSH tunnel
ssh -L 15432:localhost:5432 $DEPLOY_HOST
export DATABASE_URL='postgresql://omnislash:<POSTGRES_PASSWORD>@localhost:15432/omnislash?sslmode=disable'
omnislash db migrate

❌ Never run omnislash start / omnislash dev against a prod DATABASE_URL — the bot/worker/scheduler start writing prod state from your laptop (double HITL notifications, duplicate heartbeats, half-finished runs). Allowed against prod: omnislash db migrate, omnislash db status, psql. omnislash db push and omnislash db studio are not available from a brain at all — both are drizzle-kit wrappers that ship only in the engine source tree, so they answer ENGINE_NOT_FOUND; use psql (or your provider’s console) to browse.

Restate state and container recreation

Restate keeps its durable state — the journals of every suspended invocation, virtual-object state, the Bifrost log — in the restate_data volume, mounted at RESTATE_BASE_DIR (/restate-data). A docker restart keeps it; so does a deploy that leaves the restate service definition untouched.

Recreating that container with an empty data dir wipes it, and the wipe is silent: Restate logs Cluster 'localcluster' has been automatically provisioned and comes up healthy with nothing in it. Every in-flight agent, workflow and pipeline invocation is gone; the sweepers then fail those rows with orphaned (no restate record, heartbeat lost) and cancel the HITL approvals hanging off them. Nothing reports an error, because from Restate’s point of view a brand-new cluster is a perfectly good cluster.

Engines before this fix mounted the volume at /target while Restate wrote to /restate-data, so the state was never in the volume at all — it lived in the container’s writable layer and died on the first deploy that touched the restate service. If you are upgrading across that fix, the first deploy still loses whatever is in flight: stop the container, copy its data into the volume, then deploy.

docker stop <brain>-restate-1
docker cp <brain>-restate-1:/restate-data /tmp/restate-data
docker run --rm -v <brain>_restate_data:/dst -v /tmp/restate-data:/src alpine \
  sh -c 'cp -a /src/. /dst/'

The copied data remembers the node that wrote it, and that node’s name was the old container’s hostname — the single directory you just copied is named after it (e.g. 39c9d061bac5). Restate refuses to resume another node’s data under a different name ([RT0002] node-name is required: The working directory '/restate-data' contains data from node '39c9d061bac5' but the default node name is '6e82def849a8') and the container crash-loops, holding every service that depends on it in Created. The engine pins RESTATE_NODE_NAME to restate for a volume it provisions itself; a migrated volume needs the old name pinned instead, in docker/compose.prod.yml, for as long as that volume lives:

services:
  restate:
    environment:
      RESTATE_NODE_NAME: 39c9d061bac5 # the directory name under /restate-data

Whatever is lost anyway, reconcile (below) is the lever that squares the rows with reality.

A workflow key submitted again after that — the orchestrator’s next tick re-dispatching a vacancy it no longer sees as handled — arrives on the same Restate invocation id: a workflow run id is a function of the key. The engine reopens that id’s pipeline_invocations row as its new life (status back to running, outcome cleared, the earlier life’s steps and events kept as history) rather than adopting the closed row, so the approvals the re-run posts hang off a live parent. Engines before this fix adopted the closed row, and the parent-terminal HITL sweep cancelled every card of the re-run within a minute.

Reconcile stuck runs and pipelines

The scheduler already sweeps both every 5 minutes. These two commands are the “fix it now” lever after an admin restate kill, a Restate re-provision, or a deploy that left rows behind — and, more often, the only way to see the verdict before acting on it.

# workflow_runs: re-drive runs whose Restate attempt is gone, and parked runs with no waker left
omnislash workflow reconcile              # dry-run — prints what it WOULD re-drive, dispatches nothing
omnislash workflow reconcile --apply      # actually dispatch the resumes
omnislash workflow reconcile --json       # raw report instead of human output (jq / automation)

# pipeline_invocations: drain removed services, sync terminal status, flag orphaned rows
omnislash pipeline reconcile              # same three flags, same default

Both default to --dry-run — nothing is written until you pass --apply, and --apply wins if you type both out of habit. Both read DATABASE_URL and RESTATE_ADMIN_URL from the environment (workflow also RESTATE_INGRESS_URL, since it re-drives through the run handler rather than writing rows), and pipeline additionally needs your brain root in view — it derives the removable service set from your .pipeline.ts files. So on a server, run them inside a container that already has all of it:

docker compose -f docker-compose.prod.yml exec pipelines omnislash workflow reconcile

omnislash workflow reconcile deliberately takes no lock: overlapping the scheduler’s own pass is safe, because every candidate is re-verified per run and exactly one recover is admitted. Its report calls out separately any run the batch liveness read flagged as lost but the per-run recheck found LIVE — that is a stale batch read, not an orphan.

Logs, scaling, restart

# Logs
docker compose -f docker-compose.prod.yml logs -f bot

# Grafana / Prometheus via SSH tunnel
ssh -L 3001:localhost:3001 -L 9090:localhost:9090 $DEPLOY_HOST   # → http://localhost:3001

# Scale workers
docker compose -f docker-compose.prod.yml up -d --scale worker=5

# Restart one service / stop (keep data) / full down (keep volumes)
docker compose -f docker-compose.prod.yml restart bot
docker compose -f docker-compose.prod.yml stop
docker compose -f docker-compose.prod.yml down

Troubleshooting