How to Fine-Tune an LLM for a Specific Industry

Audit the Domain Gap

Before committing to fine-tuning, measure exactly where the general model falls short. Create a test set of 50 to 100 representative domain tasks, the same kinds of questions, analyses, or generations the model will handle in production. Run the best available instruction-tuned model (Llama 3.1 70B Instruct or equivalent) on these tasks with a well-crafted domain prompt, and have domain experts score the outputs on correctness, terminology, format compliance, and overall quality.

This audit serves two purposes. First, it establishes a baseline that any fine-tuning must beat to justify its cost. If the base model with good prompting already handles 90% of tasks correctly, the fine-tuning investment needs to be justified by the value of the remaining 10%. Second, it identifies the specific failure modes that fine-tuning should address. Common domain gaps include: incorrect use of domain terminology (using lay terms instead of technical ones), wrong output structure (narrative paragraphs when the domain expects structured findings), missed domain-specific reasoning steps (failing to check contraindications in medical contexts), and inappropriate level of certainty (being too definitive when the domain requires hedging, or too hedging when the domain expects decisive recommendations).

Collect Domain-Specific Training Data

Domain data comes from three sources, each with distinct trade-offs. The first and best source is real production data: actual outputs produced by domain experts in the course of their work. Medical practices have clinical notes, law firms have briefs and memos, financial firms have analyst reports, and engineering teams have design documents. These represent the gold standard because they reflect how experts actually communicate in the domain. The challenge is access and compliance: production data often contains PII or confidential information that requires anonymization, and regulated industries (healthcare, finance, legal) have specific rules about how data can be used for model training.

The second source is expert creation: hiring domain experts to write model responses to representative prompts. This produces the highest-quality training examples but is expensive, typically $20 to $100 per example depending on the domain expertise required. A budget of $5,000 to $20,000 produces 200 to 500 expert-written examples, which is often enough for format and terminology adaptation but may not be enough for complex reasoning patterns.

The third source is domain corpora: publicly available domain-specific text (medical journals, legal opinions, financial filings) that can be used for continued pretraining before the fine-tuning step. Continued pretraining on domain text teaches the model domain vocabulary and reasoning patterns at a deeper level than fine-tuning alone, but it requires much more data (millions of tokens rather than thousands of examples) and more compute. The combination of continued pretraining on domain text followed by SFT on domain-specific instruction data is the most thorough approach and is used by domain-specific models like Med-PaLM (medicine) and BloombergGPT (finance).

Fine-Tune with Domain Data

For domain fine-tuning, start with an instruction-tuned base model rather than a raw pretrained model. The instruction tuning provides the conversational and instruction-following behavior your application needs, and the domain SFT adds the domain specialization on top. Use LoRA or QLoRA to preserve the base model's general capability, with a rank of 32 or 64 (higher than for simple format adaptation) because domain adaptation requires learning more complex patterns.

Two training strategies work well for domain fine-tuning. The first is pure domain SFT: train only on domain-specific examples. This is the simplest approach and works when the base model already has reasonable domain knowledge and you primarily need format and terminology adaptation. The second is mixed training: combine domain-specific examples with a smaller set of general instruction-following examples (drawn from open-source instruction datasets). The general examples prevent the model from forgetting its general instruction-following ability while learning domain specialization. A mix of 70% domain data and 30% general data is a common starting point.

Use a learning rate of 5e-5 to 1e-4 (lower than for simple SFT) and train for 2 to 3 epochs. Domain adaptation requires gentler weight updates because you are teaching the model new reasoning patterns while trying to preserve existing capability, and aggressive training is more likely to cause catastrophic forgetting.

Evaluate with Domain Experts

Automated metrics are insufficient for domain fine-tuning evaluation because they cannot assess domain-specific correctness. A response that scores well on BLEU or perplexity can still use terminology incorrectly, apply the wrong reasoning framework, or miss a critical domain-specific consideration. Domain expert evaluation is required.

Have at least two domain experts independently evaluate the fine-tuned model's outputs on your held-out test set. Use a rubric with domain-specific criteria: terminological accuracy (does the model use the right terms?), reasoning correctness (does it apply domain-specific logic correctly?), structural compliance (does the output follow industry conventions?), and safety (does it avoid giving incorrect advice in high-stakes domains?). Compare the expert scores for the fine-tuned model against the baseline scores from your initial audit. If the fine-tuned model does not show a clear improvement, revisit your training data quality before training again with more data.

For regulated industries, the evaluation guide covers additional requirements like reproducibility, audit trails, and bias testing that may be required for compliance.