How to Measure AI Summary Quality and Accuracy

The Four Dimensions of Summary Quality

Every summarization evaluation should measure at least these four dimensions, though their relative importance varies by application:

Faithfulness (factual accuracy): Does the summary contain only information present in or directly supported by the source? A faithful summary never introduces facts, statistics, or claims the source does not contain. This is the most critical dimension for applications in regulated industries, fact-checking, and any context where users trust the summary as a reliable representation of the source. Unfaithful summaries that introduce hallucinated information are actively harmful.

Completeness (coverage): Does the summary capture all important information from the source? A complete summary does not miss major topics, key conclusions, or critical details. Completeness is always relative to the target compression level: a 50-word summary cannot be complete in the same way a 500-word summary can. Evaluate completeness relative to the expected compression ratio.

Conciseness (information density): Does the summary avoid unnecessary repetition, filler, and information that does not serve the reader? A concise summary packs maximum information into minimum tokens. Summaries that are too long waste tokens (expensive in production) and user attention. Summaries that repeat the same point in different words fail conciseness even if they are technically accurate.

Coherence (readability): Does the summary read as a well-organized, logical piece of text? Coherent summaries have clear transitions between ideas, consistent terminology, proper sentence structure, and logical flow. This dimension matters most for user-facing summaries where human readers consume the output directly.

Automated Metrics: Fast but Imperfect

ROUGE Scores

ROUGE measures n-gram overlap between the generated summary and one or more reference summaries. ROUGE-1 counts unigram (individual word) matches. ROUGE-2 counts bigram (two-word phrase) matches. ROUGE-L measures the longest common subsequence. Each produces precision, recall, and F1 scores.

ROUGE is free to compute, requires no API calls, and runs in milliseconds. It correlates moderately with human judgment for extractive summaries (where good summaries naturally share words with the source). For abstractive summaries, ROUGE correlates poorly because good abstractive summaries use different words than the original while conveying the same meaning, scoring low on word overlap despite being excellent summaries.

Use ROUGE as a regression detection tool rather than an absolute quality measure. If your average ROUGE-2 F1 suddenly drops 15% after a prompt change, something likely degraded. But do not optimize for ROUGE scores directly, because maximizing word overlap with a reference pushes output toward extractive-style text rather than fluent abstractive summaries.

BERTScore

BERTScore computes token-level semantic similarity between generated and reference summaries using contextual embeddings from a BERT-class model. Instead of matching exact words, it matches semantically equivalent words and phrases. "Revenue increased" scores well against "Income grew" because the embeddings are close despite zero word overlap.

BERTScore correlates more strongly with human judgment than ROUGE for abstractive summarization. It requires a small model inference (not an LLM API call, just a BERT-size encoder) and runs in 100-500ms per evaluation. Use it as your primary automated metric when evaluating abstractive summaries. A BERTScore F1 above 0.88 typically indicates good quality; below 0.82 suggests significant quality issues.

Length and Compression Metrics

Simple but informative: measure output length in tokens, compression ratio (output/input), and length adherence (how close the output is to your target length). If you specified "summarize in 200 words" and consistently get 400-word outputs, your prompt needs strengthening. Track these as basic health checks. Alert when compression ratio deviates more than 30% from your target.

LLM-as-Judge Evaluation

The most practical method for production quality monitoring uses a separate LLM call to evaluate each summary. Present the source document and the generated summary to a judge model, then ask it to score on specific dimensions.

A typical judge prompt structure:

"You are evaluating a summary. Score each dimension 1-5. Source document: [source]. Summary to evaluate: [summary]. Score faithfulness (does the summary only contain information from the source?): Score completeness (does the summary cover all major points?): Score conciseness (is the summary free of unnecessary repetition?): Score coherence (does the summary read well and flow logically?): For any score below 4, explain what specific issue you identified."

Cost per evaluation: $0.01-$0.05 depending on source length and judge model. At 5% sampling rate on 10,000 daily summaries, you evaluate 500 summaries per day at $5-$25 daily cost. This provides statistically significant quality metrics with daily granularity.

Judge model selection matters. Use a model at least as capable as the model that generated the summary. If summarization uses Haiku, evaluation can use Sonnet. If summarization uses Sonnet, evaluation should use Sonnet or Opus. Using a weaker model as judge produces unreliable scores because it may not detect subtle faithfulness violations that a stronger model would catch.

Calibrate your judge by running it on 50-100 summaries that humans have manually scored. Compare LLM scores to human scores. If correlation is below 0.7, adjust the judge prompt to better align with your quality standards. Good LLM judges achieve 0.75-0.85 correlation with human evaluators, comparable to inter-annotator agreement between humans.

Hallucination Detection

Faithfulness violations (hallucinations) deserve special attention because they are the most harmful failure mode. A coherent, well-written summary that contains false information is worse than a clunky summary that is factually accurate.

Claim decomposition + entailment checking: Break the summary into individual claims (each factual statement). For each claim, check whether it is entailed by (logically follows from) the source document. Claims that fail entailment are potential hallucinations. You can implement this with an NLI (natural language inference) model or with an LLM prompt: "Does the following source text support or contradict this claim? Claim: [claim]. Source: [relevant passage]. Answer: supported / contradicted / not mentioned."

Entity and number verification: Extract all named entities (people, companies, products) and numbers (statistics, dates, amounts) from the summary. Verify each appears in the source with the same value. "Revenue grew 15%" in the summary is a hallucination if the source says "12%" or does not mention revenue growth at all. This targeted check catches the most common and most harmful hallucination types without requiring full entailment analysis.

Frequency-based detection: Track which source sections the summary draws from. If the summary makes claims about topics not represented in any source section, those claims are likely hallucinated. This works particularly well for map-reduce summarization where you can trace which chunks contributed to which parts of the final summary.

Building a Production Quality Monitoring System

A complete monitoring system combines fast automated metrics (every summary) with deeper LLM evaluation (sampled) and human review (periodic).

Tier 1 (every summary, real-time): Length check, compression ratio, basic format validation. Cost: zero (simple computation). Catches: runaway outputs, empty responses, format violations. Alert threshold: any single failure.

Tier 2 (5-10% sample, async): LLM-as-judge scoring on faithfulness, completeness, conciseness, coherence. Cost: $5-$50 per day depending on volume. Catches: quality regressions, hallucination rate increases, prompt degradation. Alert threshold: average faithfulness score drops below 4.0 or any dimension averages below 3.5.

Tier 3 (weekly, manual): Human review of 20-50 randomly sampled summaries, plus all summaries that received low Tier 2 scores. Catches: systematic issues that LLM judges miss, calibration drift in the judge model, emerging failure patterns. This feeds back into judge prompt improvements and summarization prompt iteration.

Store all evaluation data (scores, sources, summaries, timestamps) in a queryable format. Build dashboards showing daily average scores per dimension, hallucination rate trend over time, score distribution histograms, and correlation between summary characteristics (length, source type, model used) and quality scores. This data drives systematic improvement: you can identify which source types produce the lowest quality summaries and create targeted prompts for those cases.