Continued Pre-Training vs Fine-Tuning: When to Use Each

The Two Training Stages Explained

A pretrained language model goes through its original pretraining on a massive, general corpus (trillions of tokens of web text, books, code, and other sources). This gives it broad language understanding and general knowledge. But its knowledge of specialized domains is proportional to how much of that domain appeared in the pretraining data. A model trained on a web-scale corpus knows a lot about popular topics like programming, cooking, and history, but comparatively little about niche domains like patent law, petroleum engineering, or veterinary ophthalmology, because those topics are underrepresented in web text.

Continued pre-training addresses this knowledge gap by feeding the model a large corpus of domain-specific text (medical journals, legal filings, financial reports, engineering specifications) and training it to predict the next token, the same objective used in original pretraining. The model does not need labeled input-output pairs; it learns from the raw text by absorbing vocabulary, terminology patterns, factual relationships, and reasoning styles that are characteristic of the domain. After continued pre-training on 1 to 10 billion tokens of medical text, the model understands medical terminology, drug interactions, diagnostic criteria, and clinical reasoning patterns at a level that matches a model that had those texts in its original pretraining.

Fine-tuning, by contrast, trains on labeled examples of specific tasks: given this clinical note, extract the diagnosis; given this patient question, provide this answer in this format. Fine-tuning does not expand the model's knowledge; it teaches the model to apply whatever knowledge it has to a specific task in a specific format. If the model lacks domain knowledge, fine-tuning on task examples will produce a model that follows the right format but generates wrong or shallow content, because it does not have the underlying understanding to draw on.

This is why the sequence matters. A model that undergoes continued pre-training on medical literature and then fine-tuning on medical Q&A examples will outperform a model that only receives fine-tuning, because the continued pre-training gave it the knowledge foundation that the fine-tuning then teaches it to apply.

When Continued Pre-Training Is the Right Choice

Continued pre-training is justified in three specific situations, all of which involve a significant knowledge gap between the base model and your domain.

The first situation is domains with specialized vocabulary and concepts. Legal text uses terms like "estoppel," "res judicata," and "injunctive relief" in precise ways that general-purpose models may understand superficially but not deeply. Biomedical text contains gene names, protein interactions, drug mechanisms, and clinical terminology that form a dense network of relationships the model needs to internalize. Financial text has its own vocabulary around derivatives, risk metrics, regulatory frameworks, and accounting standards. If your domain has thousands of specialized terms that the model's pretraining barely covered, continued pre-training on domain text is the most effective way to teach them.

The second situation is domains with reasoning patterns not well-represented in general web text. Legal reasoning involves applying precedent, distinguishing cases, and constructing arguments under specific rules of evidence. Scientific reasoning involves hypothesis testing, experimental design, and statistical interpretation. Engineering reasoning involves tolerance analysis, failure mode assessment, and standards compliance. These reasoning patterns are not just vocabulary; they are structured ways of thinking that the model needs extensive exposure to before it can replicate them. A few hundred fine-tuning examples cannot teach a reasoning pattern that requires thousands of pages of exposure to absorb.

The third situation is languages or dialects underrepresented in pretraining. Most base models are trained predominantly on English text, with smaller proportions of other high-resource languages. If your application targets a low-resource language, a regional dialect, or a technical sublanguage (legal German, medical Japanese, financial Mandarin), continued pre-training on text in that language or dialect is often more effective than fine-tuning on translated examples, because it teaches the model the natural patterns of that language variety.

Data Requirements and Scale

Continued pre-training requires significantly more data than fine-tuning because the training objective (next-token prediction on raw text) learns more slowly than supervised learning on labeled pairs. The general guideline is:

Minimum viable corpus: 50 to 100 million tokens (roughly 50,000 to 100,000 pages of domain text). Below this, the model does not see enough domain text to meaningfully shift its internal representations, and the compute cost is not justified by the marginal improvement.

Strong results: 500 million to 2 billion tokens (500,000 to 2 million pages). This is the range where most published domain-adapted models operate. Bloomberg's BloombergGPT used about 363 billion tokens of financial text. Google's Med-PaLM 2 used extensive medical literature. For a practitioner without those resources, 1 billion tokens of domain text is a realistic target that produces genuine domain expertise in the model.

Practical sources: Academic papers (PubMed has 36 million biomedical abstracts). Legal filings (PACER has millions of court documents). Financial reports (SEC filings, earnings transcripts). Technical documentation (internal wikis, product manuals, standards documents). Industry publications (trade journals, conference proceedings). The data does not need to be labeled or structured; raw text is the input format.

The compute cost scales with data volume and model size. Continued pre-training of a 7B model on 1 billion tokens using standard training (not LoRA, because continued pre-training typically modifies all weights) costs roughly $500 to $2,000 on cloud GPU instances in 2026. For a 70B model, multiply by approximately 10x. This is significantly more expensive than fine-tuning, which is why continued pre-training is only justified when the knowledge gap is large enough to warrant the investment.

Technical Approach

The training setup for continued pre-training differs from fine-tuning in several important ways. The training objective is causal language modeling (next-token prediction), not supervised learning on input-output pairs. The learning rate must be low enough to avoid catastrophic forgetting of the model's existing general capabilities: typically 1e-5 to 5e-5, which is 10x to 20x lower than the learning rates used in LoRA fine-tuning. The idea is to gently incorporate new knowledge rather than aggressively overwriting existing weights.

A learning rate warmup and cosine decay schedule is standard. Start with a warmup over the first 1 to 5% of training steps, ramp up to the peak learning rate, then decay following a cosine curve to near zero by the end of training. This prevents the model from being destabilized by large weight updates at the start and ensures it settles into a stable state at the end.

Unlike fine-tuning, continued pre-training typically trains for one epoch over the data, meaning each document is seen exactly once. Training for multiple epochs on the same domain corpus risks overfitting to the specific texts rather than learning general domain knowledge, and it increases the risk of the model memorizing and regurgitating specific passages verbatim. If you have more compute budget than data, it is better to find more domain text than to train for additional epochs.

Full-weight training (not LoRA) is the standard for continued pre-training because the goal is to modify the model's internal knowledge representations, which are distributed across all layers. LoRA adapters can capture behavioral changes with a small number of parameters, but absorbing new domain knowledge requires broader weight updates. Some recent work has explored using LoRA for continued pre-training at higher ranks (128 to 256), which reduces compute requirements while capturing most of the benefit, but this remains an active research area and full-weight training is still the default recommendation.

The Full Pipeline: CPT Then SFT

The optimal pipeline for building a domain-specific model is a three-stage process:

Stage 1: Continued pre-training on domain text. This gives the model domain knowledge, domain vocabulary, and domain reasoning patterns. The output is a domain-adapted base model that understands your field but has not been taught to follow instructions or perform specific tasks. It will complete text in a domain-appropriate way but will not answer questions, follow formats, or interact like an assistant.

Stage 2: Instruction fine-tuning (supervised fine-tuning, or SFT) on labeled input-output pairs for your specific tasks. This teaches the domain-adapted model to use its new knowledge in the format your application needs: answering questions, extracting information, classifying documents, generating reports, or whatever your use case requires. This is standard supervised fine-tuning using LoRA or QLoRA.

Stage 3: Preference alignment (optional) using DPO or RLHF to teach the model which outputs are preferred over others for subjective quality dimensions like helpfulness, safety, and tone. This stage polishes the model's behavior but is optional for many applications where the SFT stage already produces acceptable outputs.

This three-stage pipeline is how models like Med-PaLM, BloombergGPT, and CodeLlama were built. The continued pre-training provides the knowledge foundation, the SFT teaches task behavior, and the preference alignment refines quality. Skipping the first stage and jumping straight to SFT works for general-purpose tasks but leaves significant performance on the table for specialized domains.

When to Skip Continued Pre-Training

Continued pre-training is expensive and complex, and it is not always necessary. Skip it when:

Your domain is already well-represented in the base model's pretraining data. Topics like software development, general business, common medical conditions, and popular science are extensively covered in web text, and a base Llama 3 or GPT-4 already has strong knowledge in these areas. Fine-tuning alone will teach the model your specific task format without needing additional domain knowledge.

Your knowledge requirements can be met by retrieval. If the model needs access to specific documents, products, or policies, RAG is more practical than training the knowledge into the model. Continued pre-training teaches broad domain understanding, not specific document retrieval, and it cannot be updated without retraining.

Your compute budget does not support it. If you have $50 to spend on model customization, that budget covers a LoRA fine-tuning run with excellent results for task-specific behavior. It does not cover continued pre-training, which requires $500 or more in compute. The marginal benefit of domain knowledge must justify the 10x cost increase, and for many applications it does not.

Your dataset is too small. Continued pre-training on fewer than 50 million tokens is unlikely to produce meaningful improvements and risks degrading the model's general capabilities through catastrophic forgetting without sufficient new knowledge to compensate. If you have limited domain text, fine-tuning on labeled examples extracted from that text is a better use of the data.