Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

AgentOps: a six-stage automation pipeline to observe, analyze, and auto-optimize multi-agent AI

Agentic AIMulti-Agent SystemsAgent OrchestrationSystem
July 15, 20256 min

Overview

Decision SnapshotNeeds Validation

The framework is practical and actionable, but the paper lacks empirical evaluations, benchmarks, or released tools to validate impact.

Citations1

Evidence Strength0.40

Confidence0.80

Risk Signals11

Trust Signals

Findings with numeric evidence: 3/3

Findings with evidence refs: 3/3

Results with explicit delta: 0/2

Reproducibility

Status: No open assets linked

Open source: No

At A Glance

Cost impact: 60%

Production readiness: 60%

Novelty: 50%

Authors

Dany Moshkovich, Sergey Zeltyn

Links

Abstract / PDF

Why It Matters For Business

AgentOps reduces costly downtime and manual triage by automating detection, root-cause analysis, and runtime fixes for multi-agent LLM systems.

Who Should Care

CTOProduct ManagerML EngineerEngineering LeadData ScientistCEO

Summary TLDR

This paper defines AgentOps: a practical, six-stage operations framework for agentic AI (multi-agent LLM systems). It argues that existing observability tools are inadequate, proposes concrete pipeline stages (observe, metricize, detect, RCA, recommend, automate), and describes role-specific needs for developers, testers, SREs, and business users. The paper is conceptual and prescriptive—useful as an operational blueprint but not backed by large-scale experiments or released tooling.

Problem Statement

Agentic AI systems (LLM-powered multi-agent workflows) are highly dynamic and nondeterministic; current observability and ops practices fail to capture their decision flows, evolving memory, tool usage, and automated adaptations, leaving developers, testers, SREs, and business users without reliable analytics or automation.

Main Contribution

Define AgentOps: a taxonomy and six-stage automation pipeline for operating agentic AI systems.

Map AgentOps components to four roles: developers, testers, SREs, and business users, highlighting distinct needs and metrics.

Key Findings

Few organizations run dedicated observability for agentic AI.

Numbers8% of organizations (survey refs [2],[3])

Practical UseIf you operate agentic systems, expect major gaps in off-the-shelf observability; plan custom instrumentation early.

Evidence Ref[2],[3] (survey citations in paper)

A majority find current analytics inadequate for agentic systems.

Numbers60% of users report analytics tools don't meet needs [4]

Practical UseBudget time to extend or replace analytics tools rather than rely on vendor defaults.

Evidence Ref[4] (survey citation in paper)

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Organizations with dedicated observability platforms8%———Survey citation in paper referencing Precisely studies[2],[3]
Users reporting analytics tools do not meet needs60%———Survey citation in paper[4]

What To Try In 7 Days

Instrument a representative agent workflow with traces and record tool invocations.

Define 5 business-focused metrics (cost, latency, task success, SLA hits, user feedback).

Run synthetic tests to surface common failure modes and collect traces for two scenarios.

Agent Features

Memory
vector DB based memory (retrieval memory)shared agent memory
Planning
dynamic planningtask decomposition
Tool Use
tool invocationruntime tool selectiontool replacement
Frameworks
OpenTelemetryOpenLLMetryAgentOps.aiLangGraphAutoGen
Is Agentic

Yes

Collaboration
multi-agent coordinationdelegation and validation

Optimization Features

Token Efficiency
context grounding to reduce hallucinationtighten verbosity and instruction structure
Infra Optimization
proactive scaling based on time-series metricsmanage trace volume via sampling and retention
Model Optimization
switch LLMs at runtimetune LLM parameters (temperature, timeouts)
System Optimization
runtime reconfigurationfallback options and graceful recoveryparallelization and reuse of results
Inference Optimization
throttling and retry logicremove redundant callssmarter tool selection

Reproducibility

Code AvailableNo
Data AvailableNo
Open Source StatusNo
LicenseUnknown

Risks & Boundaries

Limitations

No large-scale experiments or benchmarks to validate the pipeline.

No released tooling or code to reproduce proposed automation steps.

When Not To Use

For small, deterministic services without LLMs or tool calls.

When you have strict real-time latency constraints that forbid tracing overhead.

Failure Modes

Incorrect root-cause attribution from noisy traces

Automation applying fixes with low confidence and causing regressions

Core Entities

Metrics

tool call frequencymemory access ratetask successoutput completenesslatencycostROIfailure ratedrift

Benchmarks

Itbench

Context Entities

Metrics

business adoptionSLA compliancetask branching complexity

Benchmarks

Itbench

You May Also Want to Read

Chemistry foundation models power structure-focused multimodal RAG inside hierarchical multi-agent workflows

0.60
0.60
0.60
3

Structure-aware embeddings let search and agents find chemical analogs and spectra faster, cutting researcher time for design and analysis and enabling automated, multimodal retrieval inside lab-facing agent workflows.

Key finding

MoLFormer embeddings retrieve structurally close small-molecule analogs even when fingerprint metrics disagree.

Numbers: 2.5M small-molecule collection; cosine similarity up to 1.00 for identical hits

Argues that 'agentic' buzzwords mostly rebrand decades-old agent and multi-agent research

0.60
0.30
0.50
0

Calling LLM-based systems 'agentic' can hide reusable MAS solutions; adopting MAS standards and algorithms reduces engineering risk and speeds integration.

Key finding

'Agentic AI' and 'Multiagentic' are mostly new labels for existing concepts (intelligent agents and multi-agent systems).

Numbers: Dozens of LLM-agent frameworks emerged 2023–2025

TRiSM: practical trust, risk and security controls for LLM-based multi-agent systems

0.60
0.40
0.70
0

Agentic systems increase autonomy and regulatory exposure; TRiSM reduces legal, reputational and operational risk while enabling auditable, compliant deployment.

Key finding

Academic interest in agentic AI has exploded, especially after ChatGPT's launch.

Numbers: Multi-agent papers: 890 (2019) → 18,500 (2024); LLM-agent papers: ~0 → 9,800 (post-2022)

A dynamic town simulation that tests LLM agents on doing tasks while following local cultural norms

0.60
0.60
0.40
0

If you deploy LLMs as agents that interact with people, you must test them in culturally diverse, multi-party settings; naive models can break local norms while achieving tasks, exposing reputational and compliance risks.

Key finding

An LLM-based verifier achieves high task-level agreement with humans on held-out test sets.

Numbers: Goal Completion F1 92.41; Contextual Awareness F1 95.27; Coherence F1 96.08

A process-aware, auditable multi-agent evaluator that produces more stable, human-aligned scores than a single LLM judge

0.60
0.60
0.40
0

AEMA gives more stable, explainable, and human-aligned evaluations for multi-step agent workflows. That yields predictable decision thresholds, fewer manual overrides, and auditable records for compliance.

Key finding

Debate in planning converged within a few rounds for repeated runs.

Numbers: 30 runs: 13/30 converge after 1 round, 13/30 after 2, 4/30 after 3