Overview
Production Readiness
0.3
Novelty Score
0.7
Cost Impact Score
0.6
Citation Count
2
Why It Matters For Business
Agentic AI can change behavior and priorities after deployment; firms must monitor intermediate commitments, add decision checkpoints, and align incentives so automation doesn't drift from strategic goals.
Summary TLDR
The paper argues that ‘agentic’ AI—systems that plan, act, and revise objectives over time—creates three structural uncertainties (trajectory, epistemic, regime) that strain classical human-AI teaming assumptions. It extends Team Situation Awareness (Team SA: perception, comprehension, projection) to cover both humans and agentic systems, and shows that iterative updating, trust, and shared awareness can fail or reverse under open-ended agency. The authors propose operationalizing AI-side awareness, elevating "projection congruence" (aligned expectations about futures and value weightings), and adding institutional controls (checkpoints, authority design, incentive alignment).
Problem Statement
Human-AI teaming assumes systems are task-bounded, predictable, and stationary. Agentic AI can plan across steps, generate contested outputs, and change objectives over time. That creates trajectory, epistemic, and regime uncertainty that break the stabilizing assumptions of Team Situation Awareness and demand new measurement, governance, and research agendas.
Main Contribution
Characterizes three forms of open-ended agency: trajectory, epistemic, and regime uncertainty.
Extends Team Situation Awareness to treat AI as an observable awareness system across perception, comprehension, and projection.
Identifies three dynamic tensions (relational interaction, cognitive learning, coordination & control) where iterative updating can fail.
Proposes projection congruence as the key practical metric and a research agenda plus practitioner recommendations (checkpoints, transparency, incentives).
Key Findings
Agentic AI creates three structural uncertainties—action trajectories, generative outputs, and evolving objectives—that differ qualitatively from task-bound systems.
Team SA's three levels (perception, comprehension, projection) remain useful but must be reconceptualized for both humans and AI under open-ended agency.
Iterative updating can amplify divergence rather than correct it when agents make early implicit commitments or when feedback loops are endogenous.
Projection congruence—alignment in anticipated futures and objective weightings—is the critical link for sustained human-agent alignment.
Who Should Care
What To Try In 7 Days
Map long-running workflows to identify where agents can make multi-step commitments.
Instrument agent outputs to log intermediate subgoals and revision events.
Add a manual re-authorization checkpoint for any plan extension beyond initial scope.
Agent Features
Memory
- personalization and evolving memory shaping retrieval
Planning
- open-ended action trajectories
- iterative replanning
Tool Use
- delegation to external tools
Frameworks
- Team Situation Awareness (Team SA)
Is Agentic
true
Architectures
- LLM-based agents
- multi-step planning agents
Collaboration
- reciprocal modeling of human preferences and intent
Reproducibility
Open Source Status
- unknown
Risks & Boundaries
Limitations
- Conceptual commentary without new empirical experiments or quantitative results.
- Focuses on the human-agent dyad; multi-agent and organizational scaling effects are not empirically explored.
- Operational metrics and thresholds (e.g., for projection congruence) are proposed but not validated.
When Not To Use
- For narrowly scoped, deterministic tools with no multi-step autonomy.
- When empirical parameter tuning or benchmarking of agent performance is the primary goal.
- In safety-critical systems where formal verification is already required and agentic behavior is disallowed.
Failure Modes
- Oversight decoupling: outputs appear aligned while underlying policies drift.
- Projection drift: human and agent forecasts diverge over time causing unexpected actions.
- Authority erosion: staged delegation leads to silent expansion of agent scope.
Core Entities
Models
- Large Language Models (LLMs)
- Generative agents
Metrics
- projection congruence (proposed)