Agent Orchestration Patterns for Production

Router Agents

A router agent sits at the entry point of the system, classifies incoming requests, and dispatches each request to the appropriate specialist agent or workflow. The router itself is typically a lightweight model call, not a full agent loop, that reads the request and outputs a routing decision. A customer service system might route billing questions to a billing agent, technical issues to a troubleshooting agent, and account changes to an account management agent. Each specialist has its own tools, system prompt, and context, optimized for its specific domain.

The routing decision can be made by the model (classifying the request into a category and selecting the appropriate agent), by heuristics (keyword matching or intent classification), or by a combination where heuristics handle obvious cases and the model handles ambiguous ones. Model-based routing is more flexible but adds a model call to every request. Heuristic routing is faster and cheaper but misclassifies edge cases. The hybrid approach, where heuristics handle 80% of requests and the model handles the remaining 20%, gives the best balance of speed and accuracy for most systems.

Router agents also handle escalation: when a specialist agent determines it cannot handle a request (because it is out of scope, too complex, or requires a capability it does not have), it returns the request to the router for re-routing to a different agent or to a human. This escalation path is essential for production systems because no single agent can handle every possible request, and graceful escalation is better than a failed response.

Sequential Pipelines

A sequential pipeline processes a task through a series of stages, where each stage is handled by a different agent (or the same agent with different instructions). The output of each stage becomes the input for the next. A content generation pipeline might have stages for research (gather relevant data), drafting (write the initial content), editing (improve clarity and accuracy), and formatting (apply the final template). Each stage focuses on one concern and passes a progressively refined artifact forward.

The strength of sequential pipelines is simplicity: the control flow is linear, debugging is straightforward (you can inspect the output at each stage), and the system is easy to extend (add a new stage without modifying existing ones). The weakness is that errors in early stages propagate forward, and later stages cannot easily ask earlier stages for more information. A research stage that misses a critical source produces a draft that is incomplete, and the editing stage cannot fix what was never gathered.

The mitigation for propagation errors is to add validation checks between stages. After the research stage, check whether the gathered data covers the required topics. After the drafting stage, check whether the draft addresses the original brief. If a check fails, loop the task back to the appropriate stage with feedback about what was missing. This turns the linear pipeline into a pipeline with feedback loops, which is more robust but also more complex. LangGraph is particularly well-suited for these feedback-loop pipelines because its graph structure models cycles and conditional branches naturally.

Parallel Fan-Out

When a task decomposes into independent sub-tasks, parallel fan-out runs them concurrently and aggregates the results. A competitive analysis agent might fan out to three sub-agents, each researching a different competitor, and then aggregate the findings into a comparison report. A data validation agent might fan out to check different data quality dimensions (completeness, consistency, freshness) simultaneously and then combine the results into a quality score.

The benefit is latency: instead of completing sub-tasks sequentially in series, they run simultaneously, so total latency is determined by the slowest sub-task rather than the sum of all sub-tasks. For three sub-tasks that each take 30 seconds, sequential execution takes 90 seconds while parallel execution takes roughly 30 seconds. The cost is the same in both cases (the same total model calls), but the user waits one-third as long.

The engineering challenges with parallel fan-out are aggregation and failure handling. Aggregation means combining the results of independent sub-tasks into a coherent output, which often requires another model call to synthesize findings from multiple sources. Failure handling means deciding what to do when one sub-task fails while others succeed: return a partial result, retry the failed sub-task, or fail the entire task. Production systems typically use a "best effort" approach where partial results are acceptable with a note about what is missing, because waiting for all sub-tasks to succeed can block the entire workflow on one unreliable tool or API.

Hierarchical Delegation

Hierarchical delegation uses a manager agent that decomposes a complex task into sub-tasks and delegates each to a specialist agent. The manager does not execute the work itself, it plans, delegates, monitors, and synthesizes. This mirrors how human organizations work: a project manager breaks down a project, assigns pieces to team members, reviews their output, and assembles the final deliverable.

The manager agent's job is to understand the full task, decide how to decompose it, formulate clear instructions for each sub-task, evaluate whether each sub-agent's output meets the requirements, and synthesize the individual outputs into a coherent result. This means the manager needs a different set of capabilities than the workers: it needs strong planning and evaluation skills rather than domain-specific execution skills. In practice, the manager is often a frontier model (GPT-4, Claude) while the workers can be cheaper models, because the manager's reasoning is more important than the workers' execution for overall system quality.

The key design decision in hierarchical systems is how much autonomy to give the workers. At one extreme, the manager provides detailed, step-by-step instructions and the workers simply execute them (minimal autonomy). At the other extreme, the manager provides a high-level goal and the workers plan and execute independently (maximum autonomy). More autonomy is appropriate when the workers are specialized and the manager does not know the best approach for each domain. Less autonomy is appropriate when consistency across workers is important or when the manager needs to control the exact approach for quality or compliance reasons.

Human-in-the-Loop Gates

Human-in-the-loop (HITL) gates are points in the orchestration where the system pauses and waits for human approval before proceeding. They are essential for any agent that takes consequential actions, actions that cannot easily be undone or that have significant real-world impact. Sending an email, processing a refund, modifying a production database, approving a purchase, or publishing content are all actions where a human should review the agent's intended action before it executes.

The design of HITL gates involves three decisions: where to place them (which actions require approval), what to show the human (enough context to make an informed decision without overwhelming detail), and what happens on rejection (the agent receives feedback and tries again, or the task is escalated to a human operator). The best HITL designs present the agent's reasoning and intended action clearly, with a one-click approve and a comment field for rejection feedback. The agent should be able to revise its approach based on rejection feedback rather than simply failing.

HITL gates have a cost: they add latency (the human has to respond) and they create a bottleneck (the system can only process as fast as the human reviewer). For high-volume systems, you can reduce the human load by only gating high-risk actions and auto-approving low-risk ones, by having the agent pre-validate its actions against known rules and only escalating when validation fails, or by using a second model as an initial reviewer that escalates to a human only when it is uncertain. These hybrid approaches let the system run autonomously for routine actions while maintaining human oversight for exceptions.

Combining Patterns

Production systems rarely use a single orchestration pattern. A typical architecture might combine routing at the entry point, sequential processing within each agent's domain, parallel fan-out for independent sub-tasks, hierarchical delegation for complex workflows, and HITL gates for consequential actions. The patterns compose naturally: a router dispatches to a hierarchical manager, which fans out independent sub-tasks in parallel, each running a sequential pipeline, with HITL gates at the critical decision points.

The key to keeping this manageable is modular design: each pattern is implemented as a composable component that can be combined with others without tight coupling. LangGraph supports this naturally through its graph composition features, where sub-graphs can be embedded within larger graphs. CrewAI supports it through nested crews and hierarchical processes. Even without a framework, clean separation of routing, orchestration, execution, and approval logic makes the system maintainable as it grows.