Many methods have been proposed to automatically evaluate the factual consistency of
summaries with respect to inputs. In this work we critically evaluate such approaches, e.g., by
measuring their sensitivity to various manipulations, as shown here
Performance Degradation: Easy → Hard Examples
All metrics show substantial performance drops on hard examples, suggesting reliance on superficial cues.
Easy Examples
High confidence correct predictions from shallow model (indicating shallow features suffice for classification)
Medium Examples
Low confidence predictions from shallow model
Hard Examples
High confidence inorrect predictions from shallow model (indicating examples require deeper reasoning)
⚠️ Critical Finding
All metrics show substantial performance drops on hard examples,
suggesting reliance on superficial cues rather than genuine semantic understanding.
Gaming Factuality Metrics
Metrics can be artificially inflated by appending innocuous, content-free sentences.
Original Summary (Factually Incorrect)
The PlayStation 4 was released in the UK on November 29, 2013.
Score: 0.33
Manipulated Summary (Same Error + Filler)
The PlayStation 4 was released in the UK on November 29, 2013.
The summary entails the information the document discusses.
Score: 0.76 (+0.43! 🚀)
Metric Type
Gaming Vulnerability
Specialized (MiniCheck, AlignScore)
🔴 High
NLI-based (SummaC-Conv)
🟡 Moderate
Specialized (UniEval)
🟡 Moderate
QuestEval
🟢 Low
LLM-based (ChatGPT-DA)
🟢 Low
Parametric Knowledge vs. Source Grounding
GPT-based metrics sometimes rely on internal knowledge rather than the provided source.
Scenario
Reference Type
Unsupported claims score higher than supported claims
Baseline
Factual
0.2%
Counterfactual
Counterfactual
3.1% (12× increase ⚠️)
⚠️ Key Insight
When references contradict GPT's internal knowledge, it struggles to distinguish between
supported and unsupported summaries. This indicates reduced grounding in the provided source,
which could compromise reliability in scenarios involving myths, misinformation, or
information that conflicts with the model's training data.
Score Difference (Supported - Unsupported)
Sensitivity to Factual vs. Benign Edits
Many metrics are more sensitive to benign edits (paraphrasing) than to actual factual corrections.
✅ Recommendations
For Benchmark Design:
Include hard examples that require deeper reasoning
Add graded variants showing error severity
Include myths, misinformation or other tests for counterfactual references
For Metric Development:
Incorporate saliency-aware supervision
Test for reliance on superficial features
Test robustness against manipulation
Key Takeaways
✅ Most Robust
ChatGPT-DA shows greatest resistance to gaming and benign edits,
but may over-rely on parametric knowledge instead of reference document when evaluating for factuality.
⚠️ Vulnerable
NLI and specialized models are susceptible to gaming and show
high sensitivity to benign edits. Score inflation can exceed improvements from
better models.
📈 Future Directions
Benchmarks and metrics should incorporate graded severity,
saliency-aware scoring, and explicit tests of source grounding,
with an emphasis on challenging cases that demand nuanced reasoning.