Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

Automating IEEE BioCompute Object creation from papers using RAG and LLMs

Retrieval Augmented Generation (RAG)Iterative RAGFaithfulness EvaluationSystem
September 23, 20247 min

Overview

Decision SnapshotNeeds Validation

The prototype is functional and open-source, but relies on external LLM APIs, lacks large-model testing, and needs human review for critical fields.

Citations2

Evidence Strength0.60

Confidence0.85

Risk Signals12

Trust Signals

Findings with numeric evidence: 0/3

Findings with evidence refs: 3/3

Results with explicit delta: 0/2

Reproducibility

Status: Partial assets available

Open source: Yes

At A Glance

Cost impact: 60%

Production readiness: 60%

Novelty: 50%

Authors

Sean Kim, Raja Mazumder

Links

Abstract / PDF / Code

Why It Matters For Business

Automating BCO creation cuts manual work for documenting legacy bioinformatics workflows and speeds evaluation, handoff, and regulatory review when human verification is applied.

Who Should Care

Product ManagerML EngineerData ScientistCTOFounder

Summary TLDR

This paper builds a proof-of-concept tool (BCO assistant) that uses Retrieval-Augmented Generation (RAG) plus LLMs to auto-generate IEEE BioCompute Objects (BCOs) from scientific papers and optional GitHub repos. Key engineering choices: per-domain prompts, chunk+embed+vector-store retrieval, two-pass retrieval with a cross-encoder re-ranker, optional repo ingestion, and integrated automated and human evaluation. Code and docs are open on GitHub. The tool lowers manual work for retroactive documentation but still needs human review for missing repo-level details and to catch hallucinations.

Problem Statement

Creating standard-compliant BioCompute Objects for past bioinformatics studies is time-consuming. Papers often omit workflow details that live in external code repos. Manual BCO creation is a barrier to reproducibility and adoption of the IEEE BCO standard.

Main Contribution

A working BCO assistant that ingests a paper (PDF) and optional GitHub repo to auto-generate per-domain BCO JSON.

A RAG pipeline with chunking, embeddings, top-k retrieval, and a two-pass re-ranking (cross-encoder) to improve relevance.

Key Findings

RAG plus LLMs can produce domain-specific BCO text from papers and repos.

Practical UseUse a RAG pipeline to extract paper content and format it into BCO domains, then have a human verify outputs.

Evidence RefAbstract; Implementation; Tool Overview

Two-pass retrieval with a cross-encoder reranker improved output quality compared to only embedding-based retrieval.

Practical UseImplement a cheap first-pass embedding search followed by a focused cross-encoder re-rank for better relevance.

Evidence RefImprovements section describing reranking and Figure 3

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Answer relevancy (automated)Evaluated using DeepEval; no numeric score reported in paper——Generated BCO domainsImprovements and Extensibility sections describing DeepEval relevancy metric—
Faithfulness (automated)Evaluated using DeepEval; no numeric score reported in paper——Generated BCO domains vs retrieved nodesImprovements and Extensibility sections describing DeepEval faithfulness metric—

What To Try In 7 Days

Clone the repo and run BCO assistant on one paper using default settings.

Index a paper plus its public GitHub repo to see how repo ingestion changes outputs.

Run the provided evaluation UI to compare generated vs human-curated domains for one workflow.

Optimization Features

Token Efficiency
Per-domain generation reduces effective context needs
Infra Optimization
Planned microservices architecture for scaling and experimentation
System Optimization
Two-pass retrieval with cross-encoder rerankerSplit retrieval embedding and LLM prompt to avoid polluting similarity scores

Reproducibility

Code AvailableYes
Data AvailableNo
Open Source StatusYes
LicenseUnknown

Code URLs

https://github.com/biocomputeobjects/bco-rag

Risks & Boundaries

Limitations

Could not test largest frontier open-source LLMs due to compute and cost limits.

Tool depends on external libraries (e.g., LlamaIndex) which may hide low-level behavior.

When Not To Use

If the paper has no linked public code or data and precise run parameters are required.

For final regulatory submissions without human verification.

Failure Modes

LLM hallucination leading to incorrect or fabricated parameter values.

Missing parametric or file-location details when repos are not indexed.

Core Entities

Models

OpenAI API (unspecified LLMs used via API)Llama 3.1 (discussed but not used due to compute limits)

Metrics

answer relevancyfaithfulness

You May Also Want to Read

MTRAG: a human-made benchmark of multi-turn RAG conversations that stresses retrieval, unanswerables, and later-turn context.

0.60
0.50
0.40
2

Multi-turn customer or assistant flows need better retrievers and grounded generators; MTRAG shows current systems miss later-turn context and unanswerables, causing wrong or misleading answers that harm trust.

Key finding

Benchmark scale and structure

Numbers: 110 conversations; 842 tasks; avg 7.7 turns; 16.9 unique passages/conversation

Atomic fact-checking for medical RAG LLMs boosts factuality and traceability

0.60
0.60
0.50
0

Per-claim fact checking makes medical LLM outputs more trustworthy and traceable, reducing clinical risk and improving compliance with guideline-driven standards while enabling on-prem deployment with smaller models.

Key finding

Atomic fact-checking improved final answer quality in the hardest (tumor-board) set by 40%.

Numbers: Overall improved answers: validation 20%, test 12%, tumor-board 40%

Build query-specific evidence graphs on the fly to fix missing links and filter distractor facts

0.60
0.70
0.60
0

Relink reduces multi-hop QA errors and increases robustness by building only the facts a query needs, cutting wrong reasoning and making answers easier to verify.

Key finding

Relink yields consistent accuracy gains over leading GraphRAG baselines on five multi-hop QA datasets.

Numbers: avg +5.4% EM; avg +5.2% F1 across five benchmarks

RAGLAB — an open, modular toolkit to reproduce, compare and develop RAG algorithms fairly

0.70
0.60
0.60
0

RAGLAB speeds RAG development and fair benchmarking, helping teams pick the right RAG variant and avoid wasted engineering time when reproducing papers.

Key finding

Self-RAG outperforms other reproduced RAG algorithms when paired with a 70B fine-tuned generator.

Numbers: PopQA ACC 48.8 (Self-RAG adaptive) vs 39.6 (NaiveRag) on Llama3-70B

InsQABench: a Chinese insurance QA benchmark plus SQL-ReAct and RAG-ReAct methods

0.60
0.60
0.70
1

Insurance products are described across short FAQs, structured product databases, and long legal clauses. A unified benchmark plus task-specific pipelines (SQL-ReAct, RAG-ReAct) cut mistakes in automated answers and speed up building customer-facing QA tools.

Key finding

Supervised fine-tuning (LoRA) raises commonsense QA accuracy for GLM4-9B from 64.40 to 70.26.

Numbers: ACC +5.86 (64.40 → 70.26)