Prompt Chaining: Breaking Complex Tasks into Steps

Why Single Prompts Fail on Complex Tasks

A single prompt asking a model to "analyze this dataset, identify the top trends, compare them to industry benchmarks, assess risks, and write an executive report with recommendations" will produce a mediocre result. Not because the model is incapable of any of those tasks individually, but because cramming five distinct tasks into one prompt dilutes the model's attention, forces it to juggle multiple objectives simultaneously, and produces output where each section gets less effort than it would as a standalone task.

The problem is cognitive load, even for models. When a prompt asks for five things at once, the model allocates its token budget across all of them, giving each task approximately one-fifth of the depth it would get in isolation. The analysis is shallow. The comparison is incomplete. The recommendations are generic. Each sub-task suffers because it competes with the others for the model's attention and output space.

Prompt chaining solves this by giving each sub-task its own prompt with its own instructions, its own context (only what that step needs), and the model's full attention. The analysis step gets a prompt optimized for data analysis. The comparison step gets the analysis output plus benchmark data and a prompt optimized for comparison. The report step gets the analysis and comparison outputs and a prompt optimized for executive writing. Each step produces better output because it is the only thing the model is working on.

How to Design a Prompt Chain

A prompt chain has three design decisions: what the steps are, what passes between steps, and what each step's prompt looks like.

Decomposing the task into steps is the first and most important decision. Each step should be a task the model can complete reliably in a single call. The test is: if you gave this step to the model as an isolated task, would it produce a high-quality result? If the answer is no, the step is too complex and should be broken down further. If the answer is "this is trivially simple," the step might be too granular and could be combined with an adjacent step.

Common chain patterns include: Extract then Analyze (pull data from a document, then analyze the extracted data). Analyze then Synthesize (analyze individual items, then synthesize findings into a summary). Generate then Review (produce a draft, then review and improve it). Classify then Route (classify the input, then apply category-specific processing). Transform then Format (process content, then format it for the target audience or system).

Designing the data flow between steps determines both quality and cost. The naive approach is to pass the entire output of each step to the next step, but this is wasteful and can degrade quality by overwhelming the next step with irrelevant detail. Better practice is to extract only the information the next step needs. If step 1 produces a 2,000-token analysis, but step 2 only needs the key findings, have step 1's output format include a "key_findings" section that step 2 receives, or add an intermediate extraction step.

Use structured output for inter-step communication. When each step produces JSON with defined fields, the next step receives exactly the information it needs in a predictable format. This eliminates the ambiguity of passing free-form text between steps and makes the chain easier to debug because you can inspect the structured data at each junction.

Optimizing each step's prompt means treating each step as its own prompt engineering problem. The prompt for each step should include: the role appropriate for that step's task, clear instructions for what to do with the input, the format for the output, and any constraints specific to that step. Do not use the same generic prompt for every step. A data extraction step needs different instructions than an analysis step, even if they are part of the same chain.

Linear Chains vs Branching Chains

A linear chain is the simplest: Step 1 feeds into Step 2, which feeds into Step 3, in a fixed sequence. This works for tasks where every input follows the same processing path. Most production chains are linear because most tasks have a predictable processing sequence.

A branching chain adds conditional logic: the output of one step determines which subsequent step runs. "If the document is a contract, run the contract analysis step. If it is an invoice, run the invoice extraction step. If it is neither, run the general summary step." This is more powerful but more complex to build and test, because each branch is a separate path that needs its own testing.

Parallel chains run multiple steps concurrently and combine their results. "Simultaneously extract the financial data, the legal terms, and the timeline, then combine all three extractions into a unified summary." Parallel chains are faster (steps run at the same time) but require a combination step that knows how to merge outputs from multiple sources.

The design principle is: start with a linear chain. Add branching only when different inputs genuinely need different processing. Add parallelism only when steps are truly independent and latency is a concern. Unnecessary complexity in chain design creates testing burden without improving output quality.

When Chaining Becomes an Agent

There is a continuum from prompt chaining to agentic AI, and knowing where you are on it helps you choose the right architecture. A linear prompt chain with fixed steps and no conditional logic is simple to build and does not need a framework. A branching chain with many branches and dynamic step selection starts to look like an agent's decision loop. A chain where the model decides what step to run next, based on the output of the previous step, is an agent.

The tipping point is when the chain needs to make decisions that cannot be predefined. If you can draw the entire chain as a flowchart before any input arrives, it is a chain, and you should build it as application code that calls the model at each step. If the processing path depends on what the model discovers during processing, it is an agent, and you should use an agent framework like LangGraph, CrewAI, or the OpenAI Agents SDK that handles state management, tool routing, and loop control for you.

Do not use an agent framework for tasks that are naturally chains. Agent frameworks add overhead (state management, loop detection, termination logic) that you do not need if your task follows a fixed sequence. A five-step chain implemented as five function calls with structured input/output is simpler, faster, and easier to debug than the same five steps wrapped in an agent loop.

Cost and Latency Trade-Offs

Prompt chaining involves multiple model calls, which means higher latency and potentially higher cost than a single-call approach. However, the cost comparison is not as unfavorable as it appears, because each step uses a smaller context window than a monolithic prompt that tries to do everything at once.

Consider a task that requires a 4,000-token input and produces a 2,000-token output. As a single call with detailed instructions: 5,000 input tokens (4,000 content + 1,000 instructions), 2,000 output tokens. As a three-step chain: Step 1 might use 4,500 input tokens and 500 output tokens (extraction). Step 2 might use 1,000 input tokens and 800 output tokens (analysis). Step 3 might use 1,200 input tokens and 1,500 output tokens (report). Total across steps: 6,700 input tokens and 2,800 output tokens. The chain uses about 30% more total tokens, but the output quality is substantially higher because each step is focused.

For cost optimization, you can use different models for different steps. Use a frontier model for the complex analysis step and a cheaper, faster model for the extraction and formatting steps. This mixed-model approach, sometimes called model routing, reduces cost by 40 to 60% compared to using the frontier model for every step. The routing by complexity page covers this pattern.

Latency is the more significant trade-off. Three sequential model calls take roughly three times as long as one call. For real-time applications where response time matters, this is a genuine constraint. Mitigation strategies include running independent steps in parallel, caching intermediate results for repeated inputs, and streaming the final step's output to the user so they see results as they are generated rather than waiting for the entire chain to complete.

Error Handling in Chains

Error handling in prompt chains is simpler than in agent loops because the chain is deterministic: you know which step failed and can retry that specific step. The key design principle is that each step should validate its output before passing it to the next step. If validation fails, retry the step (with the validation error included in the retry prompt). If the retry also fails, return an error to the calling code with the step that failed and the reason.

Do not let a failed step's partial output cascade through the chain. A downstream step working with bad input from an upstream failure will produce plausible-looking but wrong output, which is worse than a clear error. Validate at every step junction, and fail fast when validation fails.