Fine-tune when you need consistent behavioral changes that prompting cannot reliably deliver, when you need to reduce prompt length to save cost at scale, or when the model lacks the internal representations to perform your task. Use retrieval instead when the problem is missing factual knowledge, and use memory when you need personalization or cross-session continuity. The decision is not about which approach is best in the abstract, it is about matching the right technique to the specific problem you are solving.
The Decision Framework
Before spending time on fine-tuning, answer three questions about your application. First, is the problem behavior or knowledge? If the model gives wrong answers because it does not have access to the relevant information, fine-tuning is not the fix. You need retrieval (RAG) or a memory layer to put the right facts in context. Fine-tuning is for when the model has the information but does not use it correctly, or when you need it to adopt a specific style, format, or reasoning pattern it does not exhibit from prompting alone.
Second, have you fully explored prompting? A well-crafted system prompt with clear instructions and a few examples (few-shot prompting) handles most format and style requirements without any training. Many teams jump to fine-tuning because their first prompt attempt did not work, when a better prompt would have solved the problem. Spend at least a week iterating on prompts before concluding that fine-tuning is necessary. The bar should be: "I have written the best prompt I can, tested it on 100+ representative inputs, and it still fails on X% of cases in ways that matter for my application."
Third, do you have the data? Fine-tuning requires a minimum of 200 to 500 high-quality examples for narrow tasks and 1,000 to 5,000 for broader ones. If you do not have this data and cannot create it at sufficient quality, fine-tuning is not currently feasible for your project. Low-quality training data produces low-quality models with high confidence, which is worse than a base model that is merely adequate.
When Fine-Tuning Is the Right Choice
The model needs to follow a specific output format reliably
If your application requires JSON with exact field names, a specific markdown structure, or a consistent response template, and prompting achieves this 90% of the time but fails on edge cases, fine-tuning is the fix. A model trained on 2,000 examples of the correct format will produce it by default without needing format instructions in every prompt. This is one of the highest-ROI fine-tuning use cases because it also reduces prompt token count on every call.
Prompt length is a cost or latency problem at scale
If your system prompt contains 1,000 to 2,000 tokens of instructions, examples, and rules to get the right behavior, that overhead is paid on every API call. At 1 million requests per month, that is 1 to 2 billion extra input tokens, which costs hundreds to thousands of dollars depending on the model. Fine-tuning the behavior into the model lets you drop the prompt to a minimal instruction, saving those tokens on every call. The one-time training cost pays for itself within the first month at high volume.
The task requires domain knowledge not in the pretraining data
If your task involves proprietary formats, specialized notation, or domain-specific reasoning patterns that are not well-represented in general web text, no prompt teaches the model to handle inputs it has never seen patterns for. Fine-tuning on examples of the pattern builds the internal representations. Common examples include parsing proprietary log files, extracting structured data from industry-specific documents, and applying clinical or legal reasoning frameworks.
You want a smaller model to match a larger model on your specific task
A fine-tuned 8B model can outperform GPT-4 on the specific task it was trained for, at a fraction of the inference cost. If your application handles a focused set of tasks and you want to reduce per-request costs, fine-tuning a small model to specialize is one of the most cost-effective moves available. This is how companies serve millions of requests affordably: they use a fine-tuned specialist instead of paying for a generalist on every call.
When Fine-Tuning Is the Wrong Choice
The problem is missing or outdated factual knowledge
If the model answers incorrectly because it does not know about your latest product update, your internal policies, or recent regulatory changes, fine-tuning is not the solution. Fine-tuned knowledge is static: it reflects the training data at the time of training and cannot be updated without retraining. Retrieval (RAG) pulls current information into the context at request time and can be updated instantly by adding or modifying documents in your knowledge base. For factual grounding, retrieval is cheaper, more auditable, and more current than fine-tuning.
You need per-user personalization
Fine-tuning produces one model that behaves the same way for all users. If you need the system to remember individual user preferences, adapt to different communication styles, or maintain context across sessions, you need a
memory layer rather than fine-tuning. Memory stores and retrieves per-user information at runtime, providing personalization without training a separate model for each user, which would be impractical at any scale.
Your data changes frequently
If the information the model needs to incorporate changes weekly, daily, or in real time, fine-tuning cycles cannot keep up. Each training run takes hours and requires validation before deployment, making it impractical for rapidly changing domains. Retrieval systems and memory layers handle dynamic information by design, since they pull current data into the context window at request time rather than baking it into static weights.
You have not tried prompting properly
If your best prompt attempt is a one-paragraph system message and you have not experimented with few-shot examples, chain-of-thought instructions, or structured output formatting, start there. Modern frontier models with good prompting can handle most tasks without fine-tuning, and the effort to write a great prompt is orders of magnitude less than the effort to curate a training dataset and run a training pipeline. Fine-tuning should be the choice of last resort after prompting has been exhausted, not the first tool you reach for.
The Hybrid Approach
Most production AI systems end up using multiple techniques together, and the choice is not as binary as "fine-tune or do not fine-tune." The typical mature architecture combines three layers: a fine-tuned base model that handles behavior (format, style, domain reasoning), a retrieval system that provides current factual knowledge on each request, and a memory layer that maintains user-specific and session-specific context across interactions.
The fine-tuning handles what should be constant across all requests: the model's tone, its output structure, its domain vocabulary, and its reasoning patterns. The retrieval handles what varies by topic: the specific documents, policies, or data points relevant to each question. The memory handles what varies by user: their preferences, their history, their corrections, and their context. Each technique does what it is best at, and none tries to substitute for the others.
This is why the "when to fine-tune" question is really "what role does fine-tuning play in your system" rather than "should I fine-tune or not." The answer for most non-trivial applications is that fine-tuning handles one layer of the stack, the behavioral layer, while retrieval and memory handle the knowledge and personalization layers. The deeper comparison is covered in the existing guide on memory versus fine-tuning.
Key Takeaway
Fine-tune for behavior changes (format, style, domain capability) that prompting cannot reliably achieve. Use retrieval for factual knowledge that changes over time. Use memory for per-user personalization and cross-session continuity. Most production systems combine all three, with each handling what it does best.