Agent Observability: list → get → history → tail

How a brain (or its agents) watches other agents: which read tool answers which question, how agent identity works across chats, and the cursor recipe for monitoring a running child. Spawning and driving agents (agent_task, agent_create + agent_enqueue + agent_await) is covered in contract.md § isolation and in the with-background-tasks prompt module; this page is about reading.

All the read tools here (agent_summary, agent_prompt and agent_export too) are runtime:read / grant-free — no HITL prompt, safe to call from any agent at any time. They are engine ops: the dashboard’s Sessions pages read exactly the same answers.

The layered model

Four tools, ordered by cost and granularity. Read top-down, stop as soon as you have your answer:

agent_await and agent_task are the blocking variants — both return the status snapshot agent_get carries as current (plus timedOut / deleted flags). agent_prompt answers the exact prompt a turn was dispatched with; agent_export the whole record as one JSON document.

Identity: names vs ids

Agent names are unique per chat scope, not globally. A chat agent legally has one row per (chat, bot) — three chats talking to nova = three live rows named nova. So every read addresses an agent by its id; only agent_get also takes a name, resolved as:

1. Your call originates from a chat → that chat’s agent wins.
2. Otherwise: a unique live match resolves; several matches are refused with their ids instead of picking an arbitrary row:

3 agents are named "nova": ses_018f… (chat 42:bot1, running); ses_018e… (chat 99:bot1, idle); ses_0190… (global, idle). Pass `id` instead.

Every agent_list row carries id and every spawn result agentId — carry it forward instead of re-resolving by name.

agent_list

One row per session (or per lineage / run with group):

{ id, name, displayName, role, feature, status, word, source, modelRef, groupKey, boss,
  createdAt, lastActiveAt, deletedAt, usage, pendingInputs, pendingRequests,
  oldestRequest, interception, parentId, runId, origin, sessionCount, recent }

• word is the status in words — idle, working, needs_you, stuck, failed, done. stuck is a working session with no activity (no stream event, no row write) for its role’s stuck threshold — see below; a working session with a request it opened waiting on a person says needs_you, never stuck.
• usage ({ costUsd, tokens }, lifetime, from the spend ledger), pendingInputs (messages queued in its inbox), pendingRequests (questions, grants, approvals it opened that wait on a person — pending, not one held for a parent agent; the inbox’s own set), oldestRequest ({ id, summary } of the oldest of them, or null), interception (who holds it, when a person took it over), parentId (the agent that spawned it), runId (the run that started it), origin (a ref to where it came from: a run, a parent session, a heartbeat, a hook, a cycle or a chat).
• Order: the Boss session, then live sessions, lead every sort — newest / oldest (by last activity), name, status.
• group: none (default — one row per session), lineage (one row per logical agent, its groupKey, with sessionCount and its newest ≤ 24 sessions in recent), run (the sessions one run started).
• Filters (AND): status, word (the words as each session says them — a lineage row then shows its session that says the word) and source (lists), role, feature (the sessions of the roles a feature ships; engine = the system: roles), groupKey (e.g. "hb:digest" — every tick of a heartbeat), q (a part of the name), run (the sessions of one run, any execution id), entity (the sessions of an entity’s runs), parentId (what an agent spawned), intercepted: true, a window of activity (window or from / to); deleted: true includes soft-deleted sessions.
• Paginated: limit 1..100 (default 20), nextCursor → cursor under the same group and sort.
• counts: true adds, on the first page, counts: how many rows each word would list under the same filters (words) and how many in all (total) — exactly what the same call filtered to that word returns.

Counts without the rows — per status, the working ones, the lineages, with words: true the lineages by the word they say now, how many a person holds now (intercepted), and for a window (default 7d) the sessions started, failed and waiting with what agents spent beside the window before (previous); with by: "role" each role’s sessions and newest activity — come from agent_summary.

When is a session Stuck?

Running with no activity for longer than its role’s threshold. The threshold is yours: engine default 600 s → constitution/MANIFEST.yaml sessions: { stuckAfterSeconds } → a role’s own sessions: { stuckAfterSeconds } frontmatter (a CLI agent compiling a repo is quiet far longer than a chat persona). Observation only — nothing stops or retries a stuck session.

What happens to a turn a deploy cuts off?

It goes on. A deploy (or a crash) stops the process the turn’s agent ran in; once the engine is back, the turn continues on the same session with a note from the engine: the turn was cut off, harness subagents and tool calls in flight are gone, these sub-agents of yours are still alive, check what is done before redoing it. Each turn is continued at most sessions.continueAfterRestart times (engine default 3 → constitution/MANIFEST.yaml sessions: { continueAfterRestart } → a role’s own sessions:; 0 = never); past it the turn ends interrupted (Failed · “Interrupted”), and a chat turn says so in the chat. A stopped turn (cancelled) is always someone’s stop, never a restart.

While an agent’s turn is running, brain-sync holds a push that would deploy (it still commits) — at most defineProject({ brainSync: { deployHoldMinutes } }) (default 60) from the oldest held commit; 0 = never hold. A push of only runtime-only paths (.md, memory/, brainSync.runtimeOnly) is never held, and neither is a manual sync (brain_sync, the dashboard’s Sync now).

agent_get — the record and the cheap peek

One call, one agent ({ id } or { name }, a soft-deleted one included by id): the agent_list row plus

• the model it runs on (modelRef), the CLI harness its turns start on (harness — resolved as dispatch resolves it: a role’s chain through models.json, a routed step whose router has no key skipped; null for an API model), what its role gives without an override (roleModelRef), its own override (modelOverride), its grant (capabilities, extraCapabilities, capabilityCeiling), repo, runIn, systemPrompt, plannerMethod, its last textOutput and error;
• contextFill — how full its live conversation’s context window is, always present, one of three states:
  • { state: 'measured', tokens, model, window, windowSource, at } — the newest measurement of the conversation, whatever the turn’s outcome (completed, stopped, interrupted, failed or paused for a question); model is the model that call ran on, window that model’s window (windowSource: observed — reported by the runtime for that model — or registry — contextWindowTokens in models.json; both null when neither knows it, then show bare tokens);
  • { state: 'compacted', preTokens, at } — the context was compacted after the last measurement (preTokens: what that measurement said, or null); the next turn measures again;
  • { state: 'none' } — nothing measured yet (a new or just-cleared conversation);
• reapAt — when a deleted session’s row is reaped;
• mailbox — what waits in its inbox;
• scheduled — messages scheduled to it; watching — the resource watches it holds (watch_create): what it is waiting for, as { watchId, resource, expiresAt, pending, deliveriesLeft } each — an idle agent with a watch is waiting, not done. The full ledger (what was delivered, what was filtered and why — its own App’s echoes, untrusted authors, checks on a replaced commit) is watch_list { agentId }; watch_list { resource } answers who waits on one PR and watch_list { runId } what a run waits on (its agents’ watches and the pipeline’s own waits) (features.md § Events System);
• current — the monitoring snapshot:
  • lastActivityAt (ISO) — recent activity while running, completion time once terminal;
  • currentItem — the in-flight item (tool name, title), null once it stopped working;
  • lastAssistantText — last assistant message, capped at 1000 chars. The fastest way to see what an agent last said or where it got stuck;
  • pendingHitl — the interaction the agent is paused on, null when there is none: { hitlId, kind, questions, deliverAt }. questions is the asked question list for kind: 'question' (null for a grant request or other kind); deliverAt (ISO) is the latest a question held for its parent reaches a person (an escalation hands it over sooner), null when it went to a person at once. What to do with it: human-input.md § Questions from sub-agents;
  • lastToolName, partialUsage — tokens / cost / duration from the most recent completed turn;
  • cursor — stream position of the newest observed event; feed it straight into agent_tail.

agent_history — the transcript

agent_history { id, limit, cursor } returns the conversation as the live stream speaks it — events (messages, tool calls with input and result, turns, plans, requests), oldest first on the page, newest page first; nextCursor pages back. limit is 1..100 (default 50).

• conversations lists its archived conversations with why each ended (clear — a /clear started a new one; compact — a compaction set rows aside); pass one as conversationId to read it.
• episodeId reads one chat episode’s turns.

agent_tail — event forensics

Cursor-based read of the agent’s output stream, projected to compact events:

{ id, at, type, itemType, title, toolName, detail, status, truncated }

• detail carries the main text payload — tool input on item.started, tool result on item.completed, error messages, retry reasons — capped at maxChars (default 500, max 4000) with an explicit … [truncated, N chars total] marker. Full text lives in agent_history.
• No afterId → the most recent events (tail semantics, aligned with agent_get’s current.cursor). With afterId → forward scan strictly after it.
• Default event set covers turns, items (started + completed), tool progress, sub-tasks, HITL, model reroutes and runtime warnings/errors/retries; it excludes per-token content.delta noise. Pass types to override.
• maxEvents 1..200 (default 50).

agent_prompt — what the model was handed

agent_prompt { id, eventId?, against? } — the exact system prompt (system, with its hash) and turn text (user) of the agent’s newest turn, or of the turn whose user message is eventId. against: "previous" adds what moved since the dispatch before (against.diff, unified hunks): the system prompt is byte-stable by contract, so a change means the cached prefix was rewritten. against: "definition" diffs that turn’s system prompt against what the agent’s role composes now — its role file, resolved grant and instruction append as they stand — so an edit to the role since the turn ran shows as hunks. What only a turn knows (a member’s capability ceiling, an unattended or pinned run, a jailed skills path) is composed as an owner’s attended turn on the live brain, so such a turn also differs in those sections. A role that no longer resolves is refused (invalid).

Monitoring recipe

1. agent_enqueue { agent_id: "<child id>", prompt: … }
2. s = agent_get { id }                       → status, current.cursor
3. while s.status is running:
     t = agent_tail { id, afterId: cursor }
     …read t.events, cursor = t.cursor; repeat while t.hasMore…
     (or block: agent_await { agent_id, timeoutMs } → snapshot + timedOut)
4. terminal → agent_get for current.lastAssistantText / error / current.partialUsage,
   agent_history for the full transcript.
   awaiting_input → read current.pendingHitl; agent_answer / agent_escalate, then
   agent_await again (human-input.md § Questions from sub-agents).

Debugging a failing heartbeat lineage

1. agent_summary { by: "role" }                     → the digest role's sessions, newest activity
2. agent_list { groupKey: "hb:digest" }              → individual runs, live first then newest
3. agent_get { id: <failed run> }                    → error + current.lastAssistantText
4. agent_tail { id, types: ["runtime.error", "runtime.retry", "item.completed"] }

A tick its gate (<name>.heartbeat.ts) said no to has no agent row at all — it is in run_list { entity: "heartbeat:<id>" } as skipped with statusReason: "gate"; a gate that failed reads failed with error.class: "gate_error" and the gate’s message.

Heartbeat/hook run rows are reaped after the retention TTL (default 24h), and the reap takes the run’s whole transcript with it: the conversation and every event in it are children of the agent row in the database, so a hard delete leaves nothing to read — not a truncated version, nothing. The one exception is an agent that holds a resource watch — it is waiting on something outside, not idle, so it is kept until the watch ends (every watch has its own expiry). A reaped row answers not_found; agent_tail would have come back empty regardless — it reads a bounded Redis ring that has usually rolled past a finished run.

A soft delete (agent_delete, or deleting from the dashboard) keeps the row, and its conversation and events with it, until the reap (agent_get’s reapAt): agent_get, agent_history and agent_export by id still read it, and agent_list { deleted: true } lists it.

So forensics on a specific run is a same-day activity, and a finding you want to keep has to be written down somewhere rather than pointed at.

Reading a run: run_trace, run_compare, run_history

The agent tools above read one session. A run — a workflow or pipeline run, a heartbeat tick, a hook fire — has three reads of its own (run:read), and they keep working after the run’s agents were reaped:

run_trace { id } answers spans: a workflow’s phases and tasks, a pipeline’s steps, the runs it started, every request it waited on (hitl_wait) and every turn its agents billed (turn, with costUsd and tokens — a sub-agent’s under the span of the agent that spawned it). Each span has a status in the run vocabulary, startedAt / endedAt (null = still running) and a ref to open (the agent’s session, a child run, the request). A task or step that was retried has one span per attempt — attempt 1, 2, …, sharing attemptOf — with a backoff span over the wait between them, and attempts counts them. A span’s cost includes its turns’; the turns add up to the run’s cost. honesty says what the trace cannot show: reapedAgents (their spans keep their timing and their cost — the spend ledger outlives an agent — but their transcript is gone), costCoverage: "partial" when the ledger no longer holds everything the run billed, and truncated when the run had more turns than a trace draws.

run_compare { id, against? } compares a run with another — by default the previous finished run of the same entity: deltas (duration, cost, tokens, status), spans aligned by their label path (Research / Collect sources), each side with its status, duration, cost and attempts, input (where the two inputs differ, leaf by leaf) and brainSha (whether they ran on different brain commits).

run_history { id, source?, agentId?, q?, cursor?, limit } is the run’s log, newest page first and each page oldest first: agent entries (a transcript row of one of the run’s agents — written while it worked for this run, so a reused or long-lived agent’s other work stays out — as the live stream carries it), pipeline entries (lifecycle events) and journal entries (Restate’s retries and pauses), in one order. nextCursor pages to older entries; source reads only those logs (["journal"]: just the retries), agentId narrows to one of the run’s agents (a sub-agent one of them spawned included), q to entries whose text or event matches.

Why did this run take so long?

1. run_trace { id }                        → the longest spans, their attempts and back-offs,
                                             the requests it waited on
2. run_compare { id }                      → what is slower or dearer than last time, and
                                             whether its input or the brain commit changed
3. run_history { id, agentId, q: "error" } → what the slow attempt's agent said