Overview
The method shows consistent gains across many public tasks and LLM backends and reduces API/token cost; main risks are remaining API call counts and single‑objective focus.
Citations1
Evidence Strength0.80
Confidence0.85
Risk Signals9
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 3/3
Reproducibility
Status: Partial assets available
Open source: Unknown
At A Glance
Cost impact: 80%
Production readiness: 60%
Novelty: 70%
Why It Matters For Business
SEE finds stronger prompts with far fewer API calls and tokens, so teams can improve LLM task accuracy while cutting prompt optimization cost and speeding experimentation.
Who Should Care
Summary TLDR
SEE is a prompt‑search system that treats a full prompt (instruction + few‑shot examples) as one optimization target. It runs a four‑phase loop that alternates focused local operators (fast local fixes) with global fusion operators (search across candidates). On 35 public tasks SEE often finds stronger prompts than recent baselines while using far fewer LLM API calls and tokens. The method is model‑agnostic, uses five LLM operators (Lamarckian, EDA, Crossover, Feedback, Semantic), and adds two practical tweaks: performance‑based vectors with Hamming distance and adaptive phase stop rules.
Problem Statement
Current automatic prompt search usually optimizes instruction text or example selection separately. That splits the prompt and misses interactions between instruction and examples. Jointly optimizing both is combinatorial, expensive, and hard to converge. The problem: how to search this high‑dimensional discrete space efficiently and reliably so prompts work better while keeping API/token cost reasonable.
Main Contribution
Formulate cohesive prompt optimization: jointly search instruction + examples to find prompts that work together.
Design SEE, a quad‑phased metaheuristic that alternates exploration (global operators) and exploitation (local operators) and adaptively picks LLM operators.
Key Findings
On hard BBH tasks SEE improves final test accuracy vs prior SOTA by double‑digit points.
SEE cuts prompt optimization compute (API calls and tokens) substantially versus evolutionary/metaheuristic baselines.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | avg +13.94 pp | SOTA (averaged over compared methods) | +13.94 pp | BBH (8 tasks) | Abstract; Sec 4 (BBH results Table 2) | Table 2 |
| compute (API calls / tokens) | −58.67% | SOTA methods (aggregate comparison) | −58.67% | Reported benchmarks (BBH comparisons) | Abstract; Sec 4.4 (Computational Cost) and Fig.6 | Fig.6 |
What To Try In 7 Days
Run SEE (GPT‑3.5) on one hard task you care about and compare final dev/test accuracy vs your current prompts.
Swap cosine similarity for performance vectors + Hamming distance when combining prompts and measure search speed.
Test operator tolerances: let Feedback run briefly but give EDA/Crossover more iterations; measure API calls saved.
Agent Features
Planning
Tool Use
Architectures
Optimization Features
Token Efficiency
System Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Still requires nontrivial compute: authors report ~12 iterations and ~4,000 API calls in some runs.
Single‑objective optimization: SEE focuses on accuracy and cost, not multi‑objective tradeoffs like fairness or interpretability.
When Not To Use
If you need ultra‑low latency online re‑prompting (SEE needs thousands of calls during search).
If you require multi‑objective tuning (accuracy plus other constraints) out of the box.
Failure Modes
Search stalls if the initial pool lacks diversity; SEE relies on good initialization (Lamarckian or human examples).
Operator prompts or LLM failures (API errors) reduce effective evaluation and can bias selection.