zhixing-system/.agents/skills/trellis-channel/references/progress-debugging.md

Progress And Debugging

Pretty output is for operators. Raw output is the audit log. Subcommands (forum, thread, messages, context) are the audit interface — reach for them before grepping events.jsonl by hand.

Pretty vs --raw

trellis channel messages <channel> renders a compact, human-readable view: timestamps, identities, kind, and a short body. It is meant for operators scanning a channel, not for diagnostics.

Pretty output can and will truncate:

• long progress deltas (text_delta, partial tool args)
• tool names and command lines
• multi-line status fields and structured detail blobs
• forum thread titles past the column budget

When something looks "off" — a worker appears stuck, a progress line ends mid-word, an action field shows ... — switch to --raw. Raw mode emits one JSON event per line exactly as it lives in events.jsonl, so nothing is dropped.

# Pretty (operator view)
trellis channel messages <channel> --kind done --last 10
trellis channel messages <channel> --kind error --last 10

# Raw (diagnostic view) — one JSON per line
trellis channel messages <channel> --raw --kind progress --last 20
trellis channel messages <channel> --raw --last 50

Rule of thumb: never diagnose a worker from a truncated progress line.

Rebuild Streaming Text

To reconstruct what a model actually streamed during a turn, concatenate detail.text_delta from progress events:

trellis channel messages <channel> --raw --kind progress --last 80 \
  | python3 -c 'import json,sys; [print((json.loads(l).get("detail") or {}).get("text_delta",""), end="") for l in sys.stdin if l.strip()]'

Stalled Worker Diagnosis

Symptom: trellis channel list shows the worker as running, but no new events appear in messages and wait keeps timing out.

Triage order:

1. Locate the channel files. Use list --all --all-projects if you are not sure which bucket the channel lives in.

   trellis channel list --all --all-projects
   CHAN=~/.trellis/channels/<bucket>/<channel>
   

2. Confirm the supervisor and worker PIDs are alive.

   cat "$CHAN/<worker>.pid"            # supervisor PID
   cat "$CHAN/<worker>.worker-pid"     # actual CLI subprocess PID
   ps -p "$(cat "$CHAN/<worker>.pid")"
   ps -p "$(cat "$CHAN/<worker>.worker-pid")"
   

   If the supervisor PID is gone but the channel still lists the worker, you have a ghost entry — clean it with trellis channel kill <name> --as <worker> --force.

3. Tail the worker log. This is the canonical place to see provider / MCP / tool startup output that never makes it onto the channel.

   tail -f "$CHAN/<worker>.log"
   

4. Check the last raw events. A worker that emitted progress but no message/done is usually mid-stream or blocked on a tool call:

   trellis channel messages <channel> --raw --last 50
   

Common "alive but silent" causes:

• Provider cold start before the first token (long, but eventually moves).
• A blocking MCP server during startup — visible in the worker log.
• Worker is waiting for a tool result whose subprocess hung.
• Prompt is huge / model is rate-limited; check provider-side errors in the worker log.

Progress Event Interpretation

A progress event represents an in-flight piece of work. Its shape varies by action field, but the load-bearing fields are always under detail:

• detail.text_delta — incremental model output (concatenate across events to rebuild the streamed reply).
• detail.tool_name, detail.tool_input — tool call about to run or currently running.
• detail.status — short string used by long-running actions (starting, running, flushing, done).
• detail.action — semantic label (e.g. status for thread heartbeats).

Progress events are noisy by design. wait ignores them unless you pass --include-progress. When you do want to see them, prefer:

trellis channel messages <channel> --raw --kind progress --last 80

A stream that emits progress at a steady cadence but never closes with done/error/message is the classic shape of a hung tool call — inspect the worker log for the subprocess.

Wait Semantics (Quick Reference)

channel wait watches events.jsonl from EOF and wakes on:

• message
• done
• error
• killed
• progress only with --include-progress

Useful filters:

trellis channel wait T --as main --from check --kind done --timeout 15m
trellis channel wait T --as main --from check,check-cx --kind done --all --timeout 15m
trellis channel wait T --as worker --tag interrupt --timeout 1h
trellis channel wait T --as main --thread release-note --action status --timeout 10m

Exit codes: 0 matched, 124 timeout, 1/2 errors. On wait --all timeout, stderr names the workers still missing.

Auditing events.jsonl — Use Subcommands, Not grep

Every channel persists its full history at $CHAN/events.jsonl. It is tempting to tail / grep / jq this file directly during debugging. Don't make it a habit, and never do it for forum channels.

Why subcommands first:

• messages already replays the file with filters (--kind, --from, --last, --tag, --thread, --action) and gives you --raw for the exact JSON. Anything you would write a one-liner for, messages already does.
• wait consumes the same file with EOF semantics — re-implementing that with tail -f | jq will drop events under load and misorder them under rotation.
• context materializes a worker's inbox view, including cursor state. Hand-rolled filters do not respect <worker>.inbox-cursor.

Forum channels: never parse events.jsonl directly

Forum channels multiplex many logical threads onto a single events.jsonl. Each event carries thread, action, and tag fields that the forum subcommands know how to fold together. Parsing the file by hand will:

• Mix threads together and make a thread look incoherent.
• Miss thread lifecycle events (open / status / close) that change how later events should be interpreted.
• Ignore worker inbox cursors, so you will "see" events a worker has already consumed and assume they are pending.

Use the forum-aware views instead:

# List logical threads inside the forum channel
trellis channel forum list <channel>

# Inspect one thread end-to-end
trellis channel thread show <channel> <thread>

# Replay messages for a thread (supports --raw, --kind, --last)
trellis channel messages <channel> --thread <thread> --raw --last 100

# What a specific worker still has pending
trellis channel context <channel> --as <worker>

Direct reads of events.jsonl are reserved for the case where the CLI itself is suspect — e.g. confirming an event was actually persisted, or diffing against <worker>.inbox-cursor while debugging the supervisor.

Common Failures

Symptom	Cause	Fix
trellis: command not found	CLI not installed globally	npm install -g @mindfoldhq/trellis
wait exits immediately	wrong filter or identity collision	use distinct --as, inspect raw messages
zsh errors on message text	shell interpreted punctuation	use --stdin or --text-file
progress line is cut off	pretty output truncation	use messages --raw --kind progress
worker never speaks	provider startup / prompt / MCP delay	inspect <worker>.log, ps, raw events
channel not found in another cwd	project bucket mismatch	cd to project, use --scope global, or list --all-projects
ghost worker in list	supervisor died without cleanup	trellis channel kill <name> --as <worker> --force
forum thread looks scrambled	parsed events.jsonl directly	use forum, thread, messages --thread

Storage Layout

~/.trellis/channels/
└── <bucket>/
    └── <channel-name>/
        ├── events.jsonl
        ├── <channel>.lock
        ├── <worker>.log
        ├── <worker>.pid
        ├── <worker>.worker-pid
        ├── <worker>.config
        ├── <worker>.session-id
        ├── <worker>.thread-id
        ├── <worker>.inbox-cursor
        └── <worker>.spawnlock

Agents normally use the CLI, not direct file reads. Direct file reads are for debugging when CLI views are insufficient — and even then, never on a forum channel's events.jsonl.