Overview
The system is implemented end-to-end and integrated into Slack; numeric evidence shows sizable recall gains on small in-domain test sets, but the data and code are internal and evaluations are limited to 100-item subsets.
Citations1
Evidence Strength0.75
Confidence0.85
Risk Signals11
Trust Signals
Findings with numeric evidence: 3/4
Findings with evidence refs: 4/4
Results with explicit delta: 0/5
Reproducibility
Status: No open assets linked
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 70%
Novelty: 50%
Why It Matters For Business
A hybrid RAG layer plus a small abbreviation lookup can cut wrong answers and boost recall on internal technical queries, speeding engineering work and reducing time spent hunting docs.
Who Should Care
Summary TLDR
Ask-EDA is a domain chat assistant for chip design that pairs an LLM with a hybrid RAG retrieval layer (dense + sparse + reciprocal rank fusion) and an abbreviation de-hallucination module. Evaluated on three 100-item, in-domain test sets, hybrid RAG improved recall vs no-RAG (40%+ on q2a-100, 60%+ on cmds-100) and abbreviation lookup (ADH) improved recall on abbr-100 by >70%. The system runs over Slack and returns sources for user review. Key limits: a small tailored knowledge base (≈400 MB, IBM-specific), 249 abbreviations, and remaining LLM recall/hallucination issues.
Problem Statement
Design engineers struggle to find correct, up-to-date technical guidance and command syntax across scattered internal docs and Slack. Off-the-shelf LLMs hallucinate or lack current/institutional knowledge. The goal is a 24/7 assistant that returns accurate, sourced answers and reduces hallucinated abbreviation expansions.
Main Contribution
Built Ask-EDA: a chat assistant for electronic design that combines an LLM, hybrid retrieval (dense + sparse), and abbreviation de-hallucination.
Implemented a hybrid search pipeline using sentence-transformer dense vectors, BM25 sparse index, and reciprocal rank fusion (RRF).
Key Findings
Hybrid RAG substantially increases answer recall versus no retrieval.
Abbreviation de-hallucination (ADH) greatly reduces wrong expansions.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| q2a-100 Recall improvement (hybrid vs none) | >40% relative increase | no RAG | — | q2a-100 | Abstract; III.C Results (Fig.3) | Abstract; III.C |
| cmds-100 Recall improvement (hybrid vs none) | >60% relative increase | no RAG (Recall=0) | — | cmds-100 | Abstract; III.C Results (Fig.3) and text | III.C |
What To Try In 7 Days
Build a small hybrid index (dense + BM25) over your most-used internal docs and test recall on 50 common queries.
Add a curated abbreviation dictionary and inject exact matches into prompts for abbreviation-heavy domains.
Expose retrieval sources in the UI so engineers can verify answers quickly.
Agent Features
Memory
Tool Use
Frameworks
Is Agentic
Yes
Architectures
Collaboration
Optimization Features
Token Efficiency
System Optimization
Reproducibility
Risks & Boundaries
Limitations
Knowledge base is IBM-specific and ~400MB; results may not generalize to other orgs.
Abbreviation dictionary has 249 entries; only ~25% are general industry terms.
When Not To Use
When you need perfect recall on open-ended, up-to-the-minute sources not ingested into the index.
When handling highly sensitive or confidential data unless retrieval and access controls are hardened.
Failure Modes
LLM ignores injected abbreviation info and hallucinates expansions despite ADH.
Hybrid context overwhelms the LLM leading to lower F1 even with higher recall.