Dynamic Workflows: Orchestrate Subagents at Scale

A workflow script is your agent's script—the one that spawns subagents. Instead of one agent doing everything, you break it into tasks, assign each to a subagent, and orchestrate the flow.

The pattern: parent agent reads a job (e.g., 'audit the monorepo'), spawns subagents ('audit package/ui', 'audit package/api', ...), waits for results, and synthesizes output. Each subagent runs independently (possibly in parallel) with its own context, worktree, and tools.

Fan-out: dispatch N subagents in parallel (all start at once). Sequential: dispatch one, wait for it, then dispatch the next. Most workflows are mixed: fan-out for independent work, sequential for dependencies.

Fan-out is faster but requires more resources (N concurrent agents, N worktrees). Sequential is slower but simpler. Choose based on dependency graph: if task-B depends on task-A's output, they must be sequential. If independent, fan them out.

The failure mode: one subagent fails. In fan-out, do you wait for all or cancel the rest? In sequential, do you retry or abort? We'll cover recovery below.

Each subagent needs context from the parent: which package to audit, the repo state, any config. You pass this as the input string or a context dict. Subagents don't share memory; they only see what you explicitly pass.

After a subagent finishes, its output becomes input for downstream tasks. Example: subagent-A analyzes test coverage, returns a coverage report. Subagent-B takes that report as input, flags low-coverage modules, and suggests improvements.

The pattern: define a state object (dict or JSON) that flows through the workflow. At each step, one agent reads state, updates it, and passes to the next.

Subagents can fail: timeout, tool error, out of memory, or Claude returns an error. Your workflow must decide: retry, skip, or abort.

Best practice: exponential backoff (retry with delay). Retry up to 3 times; if still failing, mark as failed and continue (unless critical). For critical tasks, abort the entire workflow.

The challenge: detecting failure. A subagent might return valid JSON but with an 'error' field instead of real data. You need output validation (covered in the next section).

A subagent should return structured output (JSON). Before you use the output downstream, validate it matches the contract. Contract: field names, types, required vs optional fields.

Example contract: audit result must have {coverage: float 0-100, issues: list<string>, status: 'pass'|'fail'}. If a subagent returns {coverage: 'unknown'} (string instead of float), validation fails. You can then retry or use a fallback.

Validation prevents garbage data from propagating. It's cheap (just schema check) and catches bugs early.

A monorepo with 8 packages (ui, api, core, auth, db, mobile, web, docs). You want a complete audit: test coverage, security scan, performance analysis, documentation check. Normally, this takes hours. With parallel subagents, it takes 15 minutes.

Workflow: parent agent dispatches 8 subagents (one per package) in parallel. Each runs a suite of checks. After all complete, parent synthesizes a report: which packages pass, which need work, overall score.

If an audit fails (e.g., api package timeout), parent retries once. If it still fails, marks it as incomplete and includes a note in the final report.

When a workflow fails, you need to know why. Logs help: what was each subagent's input? what output did it produce? what validation errors occurred? The debugging pattern: save workflow state at each step, re-run from a checkpoint, or debug individual subagents in isolation.

Best practice: detailed logging. For each subagent spawn, log the input, timeout, and worktree. For each result, log the output and validation status. When something fails, you can replay from the logs and pinpoint the issue.