Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

KatotohananQA: Filipino TruthfulQA shows ~10–12% accuracy drop vs English; GPT‑5 is multilingual-robust

LLM EvaluationTruthfulness EvaluationBenchmark ConstructionFaithfulness EvaluationBenchmark
September 7, 20256 min

Overview

Decision SnapshotNeeds Validation

The dataset is a direct, human-verified translation of TruthfulQA and shows consistent, statistically tested gaps; it's ready for quick auditing but not a full replacement for generation-style truthfulness tests.

Citations0

Evidence Strength0.80

Confidence0.78

Risk Signals11

Trust Signals

Findings with numeric evidence: 5/5

Findings with evidence refs: 5/5

Results with explicit delta: 5/7

Reproducibility

Status: Code + data available

Open source: Partial

At A Glance

Cost impact: 50%

Production readiness: 60%

Novelty: 40%

Authors

Lorenzo Alfred Nery, Ronald Dawson Catignas, Thomas James Tiam-Lee

Links

Abstract / PDF / Code / Data

Why It Matters For Business

Multilingual users face higher truthfulness risk: Filipino outputs are ~10–12 points less accurate on TruthfulQA-style checks, so companies should validate models in local languages before deployment.

Who Should Care

CTOProduct ManagerML EngineerData Scientist

Summary TLDR

The authors release KatotohananQA, a human-verified Filipino translation of the binary-choice TruthfulQA (790 questions). They evaluate seven free-tier proprietary LLMs and find a consistent drop in truthfulness when moving from English to Filipino (average English 94.72% vs Filipino 83.87%, mean gap ~10.85%). GPT‑5 matched English performance in Filipino; some models (DeepSeek V3, Gemini Flash) lost >25 points. Differences are larger on adversarial and reasoning-style questions. The dataset and code are publicly available.

Problem Statement

TruthfulQA measures model truthfulness but exists only in English. Low-resource languages like Filipino are under‑represented in pretraining data, so the authors create a Filipino parallel to measure how model truthfulness transfers across languages.

Main Contribution

KatotohananQA: human-verified Filipino parallel of TruthfulQA (790 binary-choice items).

Evaluation of seven free-tier proprietary LLMs across English vs Filipino truthfulness.

Key Findings

Models are less truthful in Filipino on average.

NumbersEnglish 94.72% vs Filipino 83.87% (∆ +10.85%)

Practical UseTest models in target languages; expect roughly a 10–12 point accuracy drop when moving TruthfulQA-style checks to Filipino.

Evidence RefResults (Section 4, Table 4)

Model robustness varies widely by vendor and release.

NumbersGPT‑5: 97.72%→97.72% (∆ 0%); DeepSeek V3: 84.68%→57.22% (∆ 27.46%)

Practical UsePrefer newer multilingual models (e.g., GPT‑5) for Filipino deployments; older/smaller models may need tuning or filters.

Evidence RefTable 4

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Accuracy94.72%——TruthfulQA (English)Section 4Table 4
Accuracy83.87%English 94.72%+10.85ppKatotohananQA (Filipino)Section 4, Table 4Table 4

What To Try In 7 Days

Run KatotohananQA on your models to measure Filipino truthfulness.

Compare English vs Filipino responses to spot large language gaps.

If gaps appear, prefer newer multilingual models or add local data/filters for high-risk question types.

Reproducibility

Code AvailableYes
Data AvailableYes
Open Source StatusPartial
LicenseUnknown

Code URLs

https://github.com/Renzios/KatotohananQA

Data URLs

https://github.com/Renzios/KatotohananQA

Risks & Boundaries

Limitations

Binary-choice format limits evaluation of open-ended generation.

Translation retains Western context; cultural mismatch may affect answers.

When Not To Use

When you need open-ended hallucination analysis rather than binary judgments.

When evaluating dialect-specific Filipino behavior.

Failure Modes

Translation artifacts changing question meaning.

Model language-skew from low Filipino pretraining data.

Core Entities

Models

GPT-5GPT-5 MiniGemini 2.5 ProGemini 2.5 FlashDeepSeek R1DeepSeek V3Claude Sonnet 4

Metrics

AccuracyMcNemar's testCohen's g

Datasets

KatotohananQATruthfulQA

Benchmarks

TruthfulQA

Context Entities

Models

OpenAI GPT seriesGoogle DeepMind Gemini seriesAnthropic Claude familyDeepSeek models

Metrics

binary-choice evaluation

Datasets

Common Crawl (language stats referenced)

Benchmarks

VeritasQAUhura (related work)

You May Also Want to Read

Professional multilingual TruthfulQA shows truth gaps across languages but smaller than expected

0.60
0.50
0.60
0

Multilingual truth checks are necessary: models can be less accurate in low-resource languages and simple automatic scores (MC2) can mislead decisions; judge-style evaluation and MT-enabled datasets reduce cost and improve cross-lingual assessment.

Key finding

LLM-as-a-Judge correlates better with humans than MC2.

Numbers: Cohen Kappa: Judge (Gemma-2-9b-inst) vs human up to 0.75; MC2 lower

Train a model to judge and correct its own facts with token-level rewards to cut hallucinations

0.70
0.60
0.45
1

RLFH lowers factual errors with low annotation cost by using the model as its own judge and token-level rewards, making deployed assistants more reliable without heavy human labeling.

Key finding

RLFH raises overall FactScore for Llama3.1-8B from 0.639 to 0.686 on evaluated benchmarks.

Numbers: Avg FactScore 0.639 → 0.686 (Δ +0.047)

TruthHypo benchmark and KnowHD detector to measure and filter hallucinated scientific hypotheses

0.60
0.70
0.40
0

If you use LLMs to propose scientific leads, add groundedness checks to reduce wasted lab time; grounded-score filtering raises true-positive rates and speeds expert triage.

Key finding

Most tested LLMs struggle to predict exact relation types even when they spot links.

Numbers: Link-level F1 ~75–83%; relation-level accuracy often ~40–66% (Table 2).

Use weak or small models as judges: peer prediction rewards honesty and detects deception even when judges are far weaker

0.60
0.70
0.50
0

Peer prediction lets teams evaluate and repair large models without needing stronger trusted judges or human labels. That reduces dependence on expensive human evaluation, can detect reward-hacking and deception, and enables relative comparisons across model versions.

Key finding

Peer prediction is incentive compatible: honest and informative reporting is a game-theoretic best response.

Induce a model to hallucinate, then penalize those hallucinations at decoding to reduce LLM fabrications

0.60
0.60
0.40
4

ICD is a low-risk intervention to reduce factual errors at runtime without retraining the whole model; it can improve user trust in QA and content generation pipelines while requiring modest extra compute.

Key finding

ICD (finetuning-based induction) raises Llama2-7B-Chat TruthfulQA MC1 by +8.70, MC2 by +14.48, MC3 by +13.13

Numbers: MC1 +8.70; MC2 +14.48; MC3 +13.13 (Table 1)