Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
Training Data and MethodsDataset ConstructionBenchmark ConstructionMultilingual LLMs

Survey of 84 recent papers mapping models, datasets, benchmarks and gaps for Indic languages

June 13, 20248 min

Overview

Production Readiness

0.6

Novelty Score

0.45

Cost Impact Score

0.65

Citation Count

2

Authors

Sankalp KJ, Vinija Jain, Sreyoshi Bhaduri, Tamoghna Roy, Aman Chadha

Links

Abstract / PDF

Why It Matters For Business

Indic languages cover ~1.5–2 billion speakers; focused datasets and compact models enable local-language products with much lower compute and cost than retraining huge universal models.

Summary TLDR

This is a focused literature review of 84 recent papers on large language model (LLM) work for Indic languages. It catalogs models, corpora, benchmarks, tools, and methods; highlights uptake of small efficient models, large parallel corpora, and many new benchmarks; and lists recurring gaps: data scarcity, script and dialect diversity, code-mixing, and inconsistent evaluation. The paper is a practical map for building or evaluating Indic NLP systems and for prioritizing dataset and benchmark work.

Problem Statement

Indic languages cover huge linguistic diversity but are under-served by LLM research. Progress is patchy due to limited high-quality data, many scripts and dialects, frequent code-mixing, and few standardized benchmarks. This review aims to summarize what exists, surface gaps, and offer a taxonomy to guide researchers and practitioners.

Main Contribution

Systematic review and curated bibliography of 84 recent papers on Indic LLM research.

A taxonomy that groups work into five areas: LLMs, Corpora, Benchmarks & Evaluation, Techniques, and Tools & Applications.

Concise summaries of representative works, data sources, benchmarks, and recurring challenges for practitioners.

Key Findings

Number of papers reviewed

Numbers84 papers screened and summarized

Small efficient Indic models can match or beat larger general models on local-language generation

NumbersParamanu family: 13.3M–367.5M params; claimed 20–66× smaller yet outperform GPT-3.5 and others on Bangla/Hindi/Sanskrit

Large parallel corpora materially improve MT and related tasks

NumbersSamanantar: 49.7M English–Indic sentence pairs (plus 83.4M Indic–Indic pairs via pivot)

ASR for Indian languages benefited from fine-tuning on diverse training sets

NumbersIndicWhisper achieved lowest WER in 39/59 Vistaar benchmarks; average WER reduced by 4.1 points

Parameter-efficient and prompting strategies can yield large multilingual gains

NumbersLEAP reports 15–20% average improvement in multilingual tasks; FLix and LoRA-style adapters improve zero-shot transfer

Benchmark and evaluation coverage is expanding but uneven

NumbersNew suites cover 20–137 languages (IndicXTREME, MULTIQ, MEGAVERSE) but gaps remain for many low-resource Indic scripts

Results

papers reviewed

Value84

Paramanu model sizes

Value13.29M–367.5M params; context 1024

Samanantar parallel pairs

Value49.7M English–Indic pairs

IndicWhisper WER wins

Valuelowest WER in 39/59 Vistaar benchmarks; avg WER −4.1 points

Baselinepublic/commercial ASR baselines

Who Should Care

Product ManagerMl EngineerFounderData ScientistCtoEngineering Lead

What To Try In 7 Days

Inventory which Indic languages and scripts your product needs and map to Samanantar, Vistaar, or language-specific corpora.

Test a small, language-specific model (e.g., Paramanu or instruction-fine-tuned Airavata/Tamil-LLaMA) before defaulting to large general LLMs.

Prototype PEFT adapters (LoRA/FLix) on an open model to measure gains with minimal compute.

Optimization Features

Token Efficiency

  • script-aware tokenizers mentioned (advanced Indic tokenizer in Paramanu)

Model Optimization

  • LoRA

Training Optimization

  • typology-guided pretraining and comparable corpora (to avoid multilinguality curse)

Reproducibility

Data Urls

  • Samanantar and many referenced corpora were released by original authors (see their papers)

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Survey depends on included 84 papers; rapidly evolving field may have new work beyond their cutoff.
  • Heterogeneous reporting in source papers prevents uniform quantitative comparison.
  • Many claims (e.g., model wins) rely on individual paper evaluations and differing metrics or human-judged setups.

When Not To Use

  • When you need certified high-stakes accuracy in under-represented dialects without further validation.
  • When a specific Indic script or dialect was not covered by the referenced datasets or models.

Failure Modes

  • Tokenization inequality: some scripts use many more tokens, raising cost and shrinking context.
  • Cross-lingual contamination or benchmark leakage may inflate reported scores.
  • Models fine-tuned on synthetic or translated instructions can hallucinate cultural or factual errors.

Core Entities

Models

  • Paramanu (Gyan AI Paramanu)
  • BLOOM
  • Glot500
  • XGLM
  • IndicBART
  • mT5
  • XLM-R
  • LLaMA / Llama2
  • GPT-3.5 / GPT-4
  • Airavata
  • Tamil-LLaMA
  • IndicTrans2
  • IndicWhisper
  • BanglaT5

Metrics

  • WER
  • BLEU
  • spBLEU
  • ROUGE
  • F1 / micro-F1
  • ROUGE-L
  • Human evaluation ratings

Datasets

  • Samanantar
  • LMSYS-Chat-1M
  • Bactrian-X
  • CulturaX
  • Vistaar
  • MASSIVE
  • XL-Sum
  • JW300
  • OdiEnCorp 2.0
  • Samanantar (Indic–Indic pairs)

Benchmarks

  • MEGAVERSE
  • MEGA
  • MULTIQ
  • IndicXTREME
  • FLORES-101
  • INDICXNLI
  • IndicNLG
  • BenLLM-Eval
  • Vistaar (ASR benchmarks)