Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
Agentic AIAgent OrchestrationMulti-Agent Systems

MAO: multi‑agent LLM pipeline that generates and repairs BPMN process models

August 4, 20247 min

Overview

Production Readiness

0.6

Novelty Score

0.6

Cost Impact Score

0.7

Citation Count

1

Authors

Leilei Lin, Yumeng Jin, Yingming Zhou, Wenlong Chen, Chen Qian

Links

Abstract / PDF

Why It Matters For Business

MAO automates BPMN drafting, reducing time and per‑model cost while producing models closer to reference designs than many human modelers on tested cases.

Summary TLDR

MAO is a four‑phase framework that uses multiple LLM agents to turn textual requirements into BPMN process models. Phases: Generation (create initial BPMN text), Refinement (split activities/add gateways), Reviewing (fix semantic hallucinations via reviewer agents), and Testing (use external tools to fix format errors). Experiments with GPT‑4 on two public datasets (FG‑C, CG‑O) show MAO produces higher‑quality models than ProMoAI and outperforms most human modelers on the fine‑grained dataset. Code is publicly available.

Problem Statement

Manual BPMN modeling is slow and costly. LLMs can automate model creation but make logical (semantic) and formatting (syntax) errors. Existing LLM approaches either produce only simple sequences or require human fixes. The paper asks: can multi‑agent LLM orchestration generate usable BPMN models and automatically detect and repair hallucinations?

Main Contribution

MAO: a practical four‑phase multi‑agent pipeline (Generation, Refinement, Reviewing, Testing) to produce BPMN text from process requirements.

Prompt strategy that mixes knowledge injection (concise BPMN text format), few‑shot examples, and Chain‑of‑Thought to improve agent outputs.

Automatic handling of two hallucination types: semantic (logic/order errors) fixed by reviewer agents and format (syntax/attribute errors) fixed via external API tools.

Empirical evaluation on two public datasets showing MAO beats an existing LLM baseline (ProMoAI) and often outperforms average human modelers; code released.

Key Findings

MAO outperformed manual modelers on four FG‑C cases.

NumbersMAO surpassed 89%, 61%, 52%, 75% of human models (datasets 1–4)

MAO is faster and cheaper than the compared LLM baseline (ProMoAI).

NumbersMAO: ~10 min, $0.12; ProMoAI: ~15 min, $0.40

Both Reviewing and Testing phases materially reduce model distance to the standard answers.

NumbersExample (dataset 4): Testing removed ~45–50% of distance vs no testing (141→77)

MAO produces richer, less duplicated models than ProMoAI on open, coarse requirements.

NumbersQualitative comparison on CG‑O (Ordering & delivering pizza) in Section 4.3

Results

Percent of manual models surpassed (FG‑C)

Value89%, 61%, 52%, 75% (datasets 1–4)

BaselineHuman-Mean

Average generation time and cost

ValueMAO: ~10 min, $0.12 per model; ProMoAI: ~15 min, $0.40

BaselineProMoAI

Model distance to standard (MAO)

Value32.5, 58.5, 73.5, 77 (datasets 1–4)

Who Should Care

CtoProduct ManagerMl EngineerEngineering Lead

What To Try In 7 Days

Run the MAO repo on one simple process spec using GPT‑4 to see end‑to‑end BPMN text output.

Convert a few internal process descriptions to the paper's BPMN text format and compare outputs with existing diagrams.

Enable the Reviewing and Testing steps and measure how format checks reduce manual rework times.

Agent Features

Memory

  • few-shot in-context examples
  • prompted knowledge injection

Planning

  • multi-round refinement dialogues
  • chain-of-thought prompting

Tool Use

  • external API format checker
  • BPMN text encoder/decoder

Frameworks

  • MAO

Is Agentic

true

Architectures

  • LLM-based agents
  • role-based leader/reviewer/expert

Collaboration

  • multi-agent orchestration
  • instructor-assistant prompting

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Supports only common BPMN elements; data and message flows are not fully handled.
  • Relies on GPT‑4 (closed model) and API costs can scale with usage.
  • Evaluation uses BPMNDiffViz structural distance; real business utility beyond structural closeness is not measured.
  • CG‑O open‑ended comparisons are qualitative rather than strict benchmarks.

When Not To Use

  • When full BPMN features (rich message/data flows) are required.
  • When offline or closed‑network operation is mandatory (requires GPT‑4 APIs).
  • When legal/regulatory audits forbid AI‑generated designs without human signoff.

Failure Modes

  • Semantic hallucinations: wrong activity order or irrelevant activities.
  • Format hallucinations: missing attributes or malformed BPMN text.
  • Poor handling of message flow and other advanced BPMN elements.
  • Quality degrades if reviewers or external checks are skipped.

Core Entities

Models

  • GPT-4

Metrics

  • BPMN model distance (BPMNDiffViz)
  • percent of human models surpassed
  • generation time and API cost

Datasets

  • FG-C (Camunda fine-grained)
  • CG-O (OMG coarse-grained)

Benchmarks

  • BPMNDiffViz Greedy
  • BPMNDiffViz TabuSearch
  • BPMNDiffViz Ants
  • BPMNDiffViz SimulatedAnnealing