Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

Psy-LLM: fine-tuned Chinese LLM for scalable online mental-health Q&A

LLM EvaluationHuman EvaluationAutomatic EvaluationSystem
July 22, 20237 min

Overview

Decision SnapshotNeeds Validation

The system is a functioning prototype with deployable UI and human evaluation, but limited by data quality, compute resources and ethical constraints; suitable as an assistive tool under supervision.

Citations34

Evidence Strength0.60

Confidence0.70

Risk Signals13

Trust Signals

Findings with numeric evidence: 5/5

Findings with evidence refs: 5/5

Results with explicit delta: 5/5

Reproducibility

Status: No open assets linked

Open source: No

At A Glance

Cost impact: 60%

Production readiness: 40%

Novelty: 35%

Authors

Tin Lai, Yukun Shi, Zicong Du, Jiajie Wu, Ken Fu, Yichao Dou, Ziqi Wang

Links

Abstract / PDF

Why It Matters For Business

An LLM-based Chinese Q&A assistant can reduce counsellor load, provide fast triage and scale service availability cheaply, but should be used under human supervision.

Who Should Care

Product ManagerML EngineerCTOFounderData Scientist

Summary TLDR

Psy-LLM is a prototype system that fine-tunes Chinese LLMs (PanGu 350M and WenZhong variants) on domain Q&A and large web-crawled psychological text to provide online mental-health question-answering. The team assembled ~400k crawled samples plus PsyQA (22k questions, 56k answers), fine-tuned the models, deployed a web front end and collected human ratings (6 psychology students, 200 Q-A). PanGu 350M outperformed WenZhong on perplexity (34.56 vs 38.40), ROUGE-L (28.18 vs 23.56) and human scores. The system is a practical assistive tool for screening and immediate support but is not a replacement for trained counsellors and faces data, ethical, and deployment limits.

Problem Statement

There are far too few Chinese-speaking mental-health professionals for growing demand. The paper aims to build an AI-assisted Q&A system that gives timely, domain-aware responses and supports counsellors or users when human help is unavailable.

Main Contribution

Defined Psy-LLM: a pipeline to fine-tune Chinese LLMs for psychological Q&A and deploy a web front end

Collected and cleaned a domain corpus (~400k samples) and used PsyQA (22k questions, 56k answers) for fine-tuning

Key Findings

PanGu 350M produced lower perplexity than WenZhong on the evaluation data.

NumbersPerplexity: PanGu 34.56 vs WenZhong 38.40

Practical UsePrefer PanGu 350M for Chinese psych Q&A when compute allows; it predicts tokens better on the authors' data.

Evidence RefTable 5 (Intrinsic evaluation)

Generated answers from PanGu were judged better by humans across four quality axes.

NumbersHuman ratings (Helpfulness): PanGu 3.87 vs WenZhong 3.56 (scale 1–5)

Practical UseFine-tuning PanGu improved perceived usefulness and fluency versus WenZhong; use PanGu-style models for initial deployment.

Evidence RefTable 6 (Human evaluation)

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
PerplexityPanGu 34.56; WenZhong 38.40—PanGu −3.84 vs WenZhongEvaluation set from PsyQA / crawled data (authors)Table 5 intrinsic evaluationTable 5
ROUGE-LPanGu 28.18; WenZhong 23.56—PanGu +4.62Evaluation set from PsyQA / crawled data (authors)Table 5 intrinsic evaluationTable 5

What To Try In 7 Days

Fine-tune a mid-size Chinese LLM on a curated subset of PsyQA to validate domain answers

Deploy a minimal web front end (React + Flask on EC2) and collect user ratings

Run a small human-evaluation (20–50 Q-A) with domain experts to detect major failure modes

Optimization Features

Infra Optimization
Used single V100 (PanGu) and RTX3060 (WenZhong) due to resource limits
System Optimization
Distributed crawler used for data collection
Training Optimization
Two-stage training: general crawl pretrain then PsyQA fine-tuneEarly stopping used to avoid overfittingBatch size 8; trained ~100k iterations (PanGu)
Inference Optimization
Deployed as Flask API on EC2 for low-latency responsesFrontend asynchronous calls to mask model latency

Reproducibility

Code AvailableNo
Data AvailableNo
Open Source StatusNo
LicenseUnknown

Risks & Boundaries

Limitations

Data quality: web-crawled data is noisy and only partially reviewed by experts

No nonverbal cues: text-only models miss key counselling signals

When Not To Use

As sole responder for suicidal or high-risk emergencies

As a clinical diagnostic or legal decision tool without clinician oversight

Failure Modes

Hallucinations or irrelevant outputs that fail to answer the question

Overfitting to noisy web data leading to incoherent logic

Core Entities

Models

PanGu 350MPanGu (other sizes mentioned)WenZhong 3.5BWenZhong-GPT2-110M

Metrics

PerplexityROUGE-LDistinct-1Distinct-2Human ratings: Helpfulness, Fluency, Relevance, Logic

Datasets

PsyQA (22k Q, 56k A)Crawled corpus from TianyaCrawled corpus from ZhihuCrawled corpus from Yixinli

You May Also Want to Read

Hallucinations in LLMs are diverse, theoretically inevitable, and must be managed with grounding and human oversight

0.40
0.60
0.70
0

Hallucinations can damage trust, cause legal/financial harm, and break workflows. Because some hallucination is inevitable, companies must design systems that detect, ground, and escalate risky outputs rather than assuming perfect model truthfulness.

Key finding

Hallucination is provably unavoidable for computable LLMs.

ThaiSafetyBench: 1,954 Thai malicious prompts reveal cultural blind spots in LLM safety

0.60
0.50
0.60
0

If you deploy Thai-language LLM features, culture-specific attacks raise failure rates; use ThaiSafetyBench and the classifier to find gaps fast and prioritize alignment or safer model choices.

Key finding

Dataset size and composition

Numbers: 1,954 samples total; public subset 1,889 samples

SciIG: a benchmark that asks LLMs to draft research-paper introductions from title, abstract, and related work

0.60
0.45
0.35
0

LLMs can produce high-quality introduction drafts quickly, saving researcher time, but outputs still need expert checking for citation and factual precision.

Key finding

LLaMA4-Maverick scores highest on many semantic and faithfulness metrics.

Numbers: ROUGE-1 0.4402, ROUGE-2 0.1509, Faithfulness (LLM-judge) 0.8157 (Table 2)

PersonaLens: a large benchmark and LLM-based user+judge agents to measure personalization in task-oriented assistants

0.60
0.65
0.50
0

PersonaLens gives a practical, large-scale way to measure personalization separate from task success; use it to benchmark assistants, prioritize memory/retrieval investments, and avoid overestimating personalization from high task completion.

Key finding

Large, multi-domain benchmark: 122,133 user-task scenarios from 1,500 profiles and 111 tasks across 20 domains.

Numbers: 122,133 scenarios; 1,500 profiles; 111 tasks; 20 domains

Use simple entropy-based reweighting to make cheap model judges match human preferences.

0.70
0.60
0.80
0

SLMEval gives more human-aligned automated evaluations for subjective product content at a small fraction of GPT-4 evaluation cost, enabling affordable, frequent model comparisons in production.

Key finding

SLMEval achieves substantially higher Spearman correlation with humans on two production tasks than many baselines.

Numbers: Peptalk ρ=0.48; Recommendation ρ=0.57 (Table 1)