Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
LLM EvaluationBenchmark ConstructionPrompt TuningBenchmark Leakage

PromptCBLUE: convert CBLUE into a prompt-format, multi-task Chinese medical benchmark and report baselines

October 22, 20237 min

Overview

Production Readiness

0.6

Novelty Score

0.45

Cost Impact Score

0.6

Citation Count

4

Authors

Wei Zhu, Xiaoling Wang, Huanran Zheng, Mosha Chen, Buzhou Tang

Links

Abstract / PDF

Why It Matters For Business

PromptCBLUE gives a practical, Chinese-language testbed for medical LLM products. It shows that inexpensive PEFT fine-tuning of open 13B models can beat few-shot API use, so companies can invest in targeted fine-tuning to improve medical features.

Summary TLDR

The authors convert the Chinese CBLUE medical NLP benchmark into PromptCBLUE: a prompt/instruction-format, multi-task testbed for Chinese medical LLMs. PromptCBLUE covers entity extraction, classification, inference, dialogue symptom/status tasks and generation. They provide human‑validated prompt templates, limit per-task splits for LLM training, host an online leaderboard, release code, and report baselines: few-shot API results (GPT-4 ≈0.518 overall) and many PEFT fine-tuning runs where open-source 13B models (Baichuan-13B with LoRA) reach ≈0.71 overall. Key takeaways: prompt format + PEFT fine-tuning matter; LoRA is a strong PEFT choice; medical-only pretraining alone did not clearly out

Problem Statement

Existing biomedical benchmarks are mostly English, not in instruction/prompt form, and some test sets leaked into LLM pretraining. That makes it hard to evaluate Chinese LLMs on practical multi-task medical text processing and to compare prompt-based/fine-tuning methods.

Main Contribution

Built PromptCBLUE: a prompt-format, multi-task Chinese medical benchmark derived from CBLUE.

Created and expert-validated prompt template pool (manual + ChatGPT paraphrases).

Provided dataset splits tuned for LLM training and an online evaluation platform/leaderboard.

Systematic baselines: few-shot API runs and PEFT fine-tuning (LoRA, AdaLoRA, Adapter, P‑tuning/v2).

Open-sourced code and prompt templates to reproduce evaluation and use as a toolkit.

Key Findings

Fine-tuned open-source 13B models outperform few-shot commercial APIs on PromptCBLUE.

NumbersBaichuan-13B (LoRA fine-tuned) overall 0.71 vs GPT-4 few-shot 0.518

Among PEFT methods tested, LoRA gave the highest average score under large-scale fine-tuning.

NumbersOverall scores: LoRA 0.71, AdaLoRA 0.708, Adapter 0.705, P-tuning-v2 0.69, P-tuning 0.685

Few-shot fine-tuning using the small demonstration set yields much lower performance than large-scale fine-tuning.

NumbersFew-shot fine-tuned overall ≈0.26–0.31 vs large-scale LoRA fine-tuned ≈0.71

Further pretraining on medical text did not clearly improve performance over open-domain LLMs in this benchmark.

NumbersChatMed and Ziya-medical‑13B did not significantly beat Baichuan-13B after fine-tuning

Data and prompt quality checks show low label noise.

NumbersRandom check found average 0.9% mislabeling rate

Results

Overall score (few-shot)

ValueGPT-4 0.518

Overall score (PEFT fine-tuned)

ValueBaichuan-13B (LoRA) 0.71

BaselineGPT-4 few-shot 0.518

PEFT method comparison (overall)

ValueLoRA 0.71 / AdaLoRA 0.708 / Adapter 0.705 / P-tuning-v2 0.69 / P-tuning 0.685

Information extraction F1 (example)

ValueCMeEE-V2 instance-level strict micro-F1: Baichuan-13B LoRA 0.749

Who Should Care

Product ManagerMl EngineerEngineering LeadCto

What To Try In 7 Days

Run LoRA fine-tuning on a 13B open model (Baichuan-13B or LLaMA-2-13B) using a small task subset from PromptCBLUE.

Adopt the paper's prompt templates and test both plain prompts and formats with explicit output specs.

Add 50–200 labeled examples per target task and compare few-shot vs PEFT fine-tuned performance using the benchmark scripts in the repo.

Optimization Features

Model Optimization

  • LoRA

Training Optimization

  • PEFT methods to limit tunable params to <1% of model

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Benchmark focuses on Chinese and CBLUE-derived tasks; not a full clinical QA or decision-support test.
  • Some tasks require structured post-processing (regex extraction) which may bias results.
  • Performance reflects the specific splits and prompt templates chosen; alternative prompts or more data may change rankings.

When Not To Use

  • Not suitable as a clinical safety or regulatory validation dataset without extra clinical audit.
  • Not a replacement for English or multi-lingual medical benchmarks when cross-lingual claims are needed.

Failure Modes

  • LLMs omit entities or predict wrong entity spans in extraction tasks (paper reports omissions and boundary errors).
  • Models misclassify 'not-in-list' / 'other' labels frequently.
  • Generation summaries can miss key exam/findings in long dialogues (ROUGE-L low examples).

Core Entities

Models

  • GPT-4
  • ChatGPT
  • ChatGLM
  • Baichuan-13B
  • Ziya-13B-medical
  • ChatMed
  • Chinese-LlaMA-2-7B
  • Chinese-LlaMA-2-13B
  • BLOOMZ-7.1B-mt
  • Randeng-T5-784M
  • GPT-2 Chinese

Metrics

  • instance-level strict micro-F1
  • micro-F1
  • macro-F1
  • ROUGE-L

Datasets

  • PromptCBLUE
  • CBLUE

Benchmarks

  • PromptCBLUE
  • CBLUE