Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
Retrieval Augmented Generation (RAG)Context SelectionInstruction Design

Use LLMs to generate extra context that improves specialized entity linking models

July 4, 20247 min

Overview

Production Readiness

0.75

Novelty Score

0.6

Cost Impact Score

0.65

Citation Count

4

Authors

Amy Xin, Yunjia Qi, Zijun Yao, Fangwei Zhu, Kaisheng Zeng, Xu Bin, Lei Hou, Juanzi Li

Links

Abstract / PDF

Why It Matters For Business

LLMaEL boosts entity-linking accuracy—especially for rare entities—without costly LLM fine-tuning, so teams can improve downstream QA, search, and recommendation systems with modest LLM usage.

Summary TLDR

LLMaEL prompts off-the-shelf LLMs to generate short, mention-centered descriptions and appends them to the original context before feeding into specialized entity linking (EL) models. This ‘context augmentation’ is tuning-free for the LLM and preserves specialized EL models for final linking. On six standard EL benchmarks, LLMaEL improves average disambiguation accuracy and—after lightweight fine-tuning of the EL model—sets new state-of-the-art results. Gains are largest on mid- and long-tail entities. Code and augmented datasets are released.

Problem Statement

Specialized EL models follow strict output formats but lack coverage for rare entities. Large LMs know many rare entities but fail at exact KB-name outputs and task specs. How can we cheaply combine LLM knowledge with specialized EL models to improve disambiguation, especially for long-tail entities?

Main Contribution

Introduce LLMaEL: a plug-and-play framework that uses tuning-free LLMs to generate mention descriptions and augment EL contexts.

Show multiple data-fusion strategies (five joining orders, ensemble, and optional lightweight fine-tuning of EL models).

Demonstrate consistent accuracy gains across six standard EL benchmarks and provide code and datasets.

Key Findings

Lightweight fine-tuning with LLM-augmented data yields measurable gains over the original EL model.

NumbersReFinED avg acc 85.46% → LLMaEL × ReFinEDFT 86.67% (+1.21%)

LLMaEL outperforms tuning-free prompting baselines that use LLMs as direct entity linkers.

NumbersChatEL avg 77.78% → LLMaEL × ReFinEDFT 86.67% (+8.89%)

Choice of backbone LLM matters little within tested LLMs.

NumbersMax avg diff ≤ 0.38% (vanilla), ≤ 0.22% (fine-tuned)

LLM augmentation helps mid- and long-tail entities more than head entities.

NumbersAccuracy improvements concentrated in PageRank buckets ~10^-6–10^-2 and 10^-7–10^-6

LLM-based re-ranking of candidate lists performs worse than context augmentation.

NumbersRe-rank-100 / Re-rank-10 underperform BLINK and LLMaEL on test splits (Table 5)

Results

Accuracy

ValueLLM only 74.18%, ChatEL 77.78%, BLINK 81.61%, GENRE 82.06%, ReFinED 85.46%

Accuracy

ValueLLMaEL × ReFinED 85.76% (avg)

BaselineReFinED 85.46%

LLMaEL with EL fine-tuning (best)

ValueLLMaEL × ReFinEDFT 86.67% (avg)

BaselineReFinED 85.46%

Improvement over ChatEL (best config)

ValueLLMaEL × ReFinEDFT 86.67% vs ChatEL 77.78%

BaselineChatEL 77.78%

Who Should Care

Ml EngineerProduct ManagerCtoData ScientistFounder

What To Try In 7 Days

Run your EL dev set through an open LLM (e.g., Llama-3-70b) to generate short mention descriptions.

Append generated descriptions after original context (strategy 4) and eval existing EL model for quick lift.

If useful, fine-tune the EL model on the augmented training set for another small accuracy gain.

Optimization Features

Training Optimization

  • Lightweight EL fine-tuning on LLM-augmented data

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • LLM-generated descriptions can misinterpret mentions or drift topic and mislead EL (manual error analysis shows both types).
  • Fine-tuning can overfit specific datasets (small drop on AIDA observed after fine-tuning).
  • Augmentation introduces 1.19% regression cases in an evaluated dataset; requires monitoring.
  • LLM calls add latency and cost; not ideal for strict real-time pipelines without caching.

When Not To Use

  • When extremely low-latency or offline budgets forbid external LLM calls.
  • When re-ranking many similar candidates is the intended integration — LLM re-ranking performed worse.
  • If you cannot validate or filter generated contexts; some augmentations introduce errors.

Failure Modes

  • LLM misinterpretation of the mention leading to wrong descriptions (paper: 14% of sampled regressions).
  • Topic drift where LLM describes a related but different entity (14% of sampled regressions).
  • EL model fails to use correct augmented context due to distribution mismatch (72% of sampled regressions).

Core Entities

Models

  • Llama-3-70b-Instruct
  • GPT-3.5-Turbo-Instruct
  • GLM-4
  • ReFinED
  • GENRE
  • BLINK

Metrics

  • Accuracy
  • Exact Match (entity name)

Datasets

  • AIDA-YAGO2
  • MSNBC
  • AQUAINT
  • ACE2004
  • WNED-CWEB
  • WNED-WIKI

Benchmarks

  • AIDA-YAGO2
  • MSNBC
  • AQUAINT
  • ACE2004
  • WNED-CWEB
  • WNED-WIKI

Context Entities

Models

  • Llama-3-70b-Instruct
  • GPT-3.5-Turbo-Instruct
  • GLM-4

Metrics

  • PageRank-based frequency buckets for entity frequency analysis

Datasets

  • AIDA-YAGO2 dev used for joining-strategy selection