Overview
The method shows clear metric gains on public benchmarks and trains quickly with LoRA, but it depends on an indexed KB and LLM availability; further engineering is needed for latency and domain shifts.
Citations2
Evidence Strength0.80
Confidence0.85
Risk Signals10
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 4/4
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 70%
Production readiness: 60%
Novelty: 60%
Why It Matters For Business
You can replace multiple modality-specific retrievers with one LLM-based generative retriever that scales to millions of documents, improves precision, and needs only light fine-tuning, lowering engineering and data costs.
Who Should Care
Summary TLDR
GeMKR is an end-to-end generative retriever for multi-modal queries. It fine-tunes a frozen LLaMA LLM (LoRA) plus a visual front-end (CLIP + object-aware prefix tuning) to generate short, document-unique text snippets called "knowledge clues." Those clues are constrained during decoding via an FM-Index so each clue maps to exactly one KB document. On three benchmarks (KB sizes 112K–21M) GeMKR improves retrieval metrics by 3.0–14.6% over strong baselines (e.g., P@5 49.1 vs 41.7). The system is trainable with ~20K instruction samples, updates ~14M parameters, and trains in ~3 hours on a single A6000 48GB GPU.
Problem Statement
Multi-modal retrieval usually stitches together separate text and image retrievers. That is data-hungry and weak at cross-modal interactions. We need a single, efficient retriever that handles text+image queries, generalizes to large KBs, and keeps training costs low.
Main Contribution
A generative retrieval pipeline (GeMKR) that produces short text "knowledge clues" which are then looked up in an indexed KB.
Object-aware prefix-tuning and a dual-flow attention trick to align multi-grained image features with an LLM.
Key Findings
GeMKR raises P@5 on OKVQA-GS112K to 49.1, beating ReViz-ICT (41.7).
On a 21M-document KB, GeMKR improves P@5 by 14.6% and R@5 by 8.9% over baselines.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| P@5 (OKVQA-GS112K) | 49.1 | ReViz-ICT 41.7 | +7.4 | OKVQA-GS112K | Table 1 shows GeMKR P@5 49.1 vs ReViz-ICT 41.7 | Table 1 |
| P@5 (OKVQA-WK21M, 21M KB) | 46.0 | ReViz-ICT 31.4 | +14.6 | OKVQA-WK21M | Table 1 reports P@5 46.0 for GeMKR vs 31.4 for ReViz-ICT | Table 1; Main Results |
What To Try In 7 Days
Index a small document corpus with an FM-Index and try constraint decoding to map generated substrings to docs.
Fine-tune a frozen LLaMA-7B with LoRA and feed CLIP image embeddings via a projection layer to generate short knowledge clues.
A/B test clue-based retrieval vs your existing pipeline on a subset of multimodal queries to measure precision and recall.
Agent Features
Memory
Tool Use
Architectures
Optimization Features
Token Efficiency
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Requires the KB to be indexed (FM-Index) so substring lookups work.
Clues that appear in multiple documents are discarded, which may lower recall on noisy corpora.
When Not To Use
You lack a document corpus that can be indexed by substring (no text or non-text KB).
You cannot run or fine-tune an LLM (resource or policy constraints).
Failure Modes
Generated clue maps to several docs — the model drops the result and misses retrieval.
Insufficient visual features or bad object detection degrades cross-modal clues.