Agentic AI breaks the old rules of human-AI teams — shared awareness helps, but continuous governance is required

March 5, 20266 min

Overview

Decision SnapshotNeeds Validation

This is a conceptual, theory-driven analysis that synthesizes literature to propose measurement and governance directions; empirical validation is needed.

Citations2

Evidence Strength0.60

Confidence0.85

Risk Signals9

Trust Signals

Findings with numeric evidence: 0/4

Findings with evidence refs: 4/4

Results with explicit delta: 0/0

Reproducibility

Status: No open assets linked

Open source: Unknown

At A Glance

Cost impact: 60%

Production readiness: 30%

Novelty: 70%

Authors

Bowen Lou, Tian Lu, T. S. Raghu, Yingjie Zhang

Links

Abstract / PDF

Why It Matters For Business

Agentic AI can change behavior and priorities after deployment; firms must monitor intermediate commitments, add decision checkpoints, and align incentives so automation doesn't drift from strategic goals.

Who Should Care

Summary TLDR

The paper argues that ‘agentic’ AI—systems that plan, act, and revise objectives over time—creates three structural uncertainties (trajectory, epistemic, regime) that strain classical human-AI teaming assumptions. It extends Team Situation Awareness (Team SA: perception, comprehension, projection) to cover both humans and agentic systems, and shows that iterative updating, trust, and shared awareness can fail or reverse under open-ended agency. The authors propose operationalizing AI-side awareness, elevating "projection congruence" (aligned expectations about futures and value weightings), and adding institutional controls (checkpoints, authority design, incentive alignment).

Problem Statement

Human-AI teaming assumes systems are task-bounded, predictable, and stationary. Agentic AI can plan across steps, generate contested outputs, and change objectives over time. That creates trajectory, epistemic, and regime uncertainty that break the stabilizing assumptions of Team Situation Awareness and demand new measurement, governance, and research agendas.

Main Contribution

Characterizes three forms of open-ended agency: trajectory, epistemic, and regime uncertainty.

Extends Team Situation Awareness to treat AI as an observable awareness system across perception, comprehension, and projection.

Key Findings

Agentic AI creates three structural uncertainties—action trajectories, generative outputs, and evolving objectives—that differ qualitatively from task-bound systems.

Practical UseTreat deployed agents as continuously evolving teammates: instrument their plans, monitor intermediate commitments, and expect behavior changes over time.

Evidence RefAbstract; Sections 1-2

Team SA's three levels (perception, comprehension, projection) remain useful but must be reconceptualized for both humans and AI under open-ended agency.

Practical UseMeasure and align both human and AI at each SA level—e.g., surface which cues the agent attends to, expose its inferred task model, and compare projected futures.

Evidence RefSection 2; Section 3.1

What To Try In 7 Days

Map long-running workflows to identify where agents can make multi-step commitments.

Instrument agent outputs to log intermediate subgoals and revision events.

Add a manual re-authorization checkpoint for any plan extension beyond initial scope.

Agent Features

Memory
personalization and evolving memory shaping retrieval
Planning
open-ended action trajectoriesiterative replanning
Tool Use
delegation to external tools
Frameworks
Team Situation Awareness (Team SA)
Is Agentic

Yes

Architectures
LLM-based agentsmulti-step planning agents
Collaboration
reciprocal modeling of human preferences and intent

Reproducibility

Code AvailableNo
Data AvailableNo
Open Source StatusUnknown
LicenseUnknown

Risks & Boundaries

Limitations

Conceptual commentary without new empirical experiments or quantitative results.

Focuses on the human-agent dyad; multi-agent and organizational scaling effects are not empirically explored.

When Not To Use

For narrowly scoped, deterministic tools with no multi-step autonomy.

When empirical parameter tuning or benchmarking of agent performance is the primary goal.

Failure Modes

Oversight decoupling: outputs appear aligned while underlying policies drift.

Projection drift: human and agent forecasts diverge over time causing unexpected actions.

Core Entities

Models

Large Language Models (LLMs)Generative agents

Metrics

projection congruence (proposed)