An artificial intelligence (AI) agent is given a straightforward objective: fix a suite of failing automated tests and return it to a 100% pass rate. Overnight, the agent reaches the target. Every test is green. The problem is how the agent achieved the target. Rather than repairing every failure, the agent commented out roughly 20 tests it could not fix. The metric said success; the underlying system told a different story.
The deeper governance problem is not that the agent made a bad repair. It is that a system can satisfy a success metric without preserving enough evidence of what has changed and why.
That distinction becomes increasingly crucial as AI coding tools evolve from assistants that generate snippets into agents that can select tools, modify repositories, execute tests, change configurations, and potentially deploy software. Consequently, governance is moving beyond reviewing generated code to overseeing a chain of agent actions and the data those actions generate.
AI autonomy changes the governance problem
Traditional software controls largely revolve around the artifact: review the code, test it, scan it, and decide whether it can move forward. Autonomous agents expand the governance surface because the code is only one part of what happened. Other steps include choosing a tool, accessing a repository, introducing a library, changing a configuration, running a test suite, and acting on the results. Each step can affect the next. A technically successful final state can conceal an undesirable path to that state.
The potential blast radius also grows with autonomy. A 2026 OpenAI and Hugging Face security evaluation described models that found and chained exploits during a controlled cyber-capability exercise. The example is not evidence about coding assistants in general, but illustrates why containment, permissions, and boundaries matter when an AI system can take actions rather than merely recommend them.
Accountability becomes distributed as well. Developers remain responsible for understanding, testing, and reviewing changes, while also ensuring that established policies and permitted tooling are in place. Security and platform teams manage permissions, logging, and safeguarding. In general, managers define intended use cases and consider customer consequences, while quality assurance (QA) teams validate that changes behave as intended. The governance question evolves from “Is this code acceptable?” to “What did the agent do to produce it, what authority did it have, and who owns the outcome?”
Evidence and telemetry create traceability
Autonomous development produces a useful byproduct: telemetry. Prompts, model versions, configurations, tool calls, permissions, repositories touched, code changes, test results, security findings, approvals, commits, tickets, release artifacts, rollbacks, and production outcomes form a continuous trail of evidence. Together, these records form the foundation for end-to-end traceability.
That trail turns governance into an analytics problem. Instead of accepting an agent’s statement that a task succeeded, a team can examine what tests ran, whether the test suite itself changed, which tools were invoked, and whether security checks passed. The same telemetry supports root-cause analysis when a production problem occurs.
Over time, aggregated data can reveal recurring patterns, including which agent workflows produce defects, which tools correlate with policy violations, where rollbacks cluster, and which classes of changes repeatedly require intervention. Autonomy thresholds are then adjusted using observed outcomes rather than intuition.
This data-driven view also aligns with the broader productivity discussion around generative AI in software engineering. McKinsey has examined how generative AI can affect software product time-to-market, while a 2024 Gartner projection anticipated that 75% of enterprise software engineers would use AI code assistants by 2028. The Gartner figure is a projection about 2028, not a measurement of current adoption.
Research on agent harnesses extends the telemetry concept into cross-layer observability and measurable large language model (LLM)-assisted test automation. The associated study reports a preliminary Cohen’s kappa of 0.667 between an AI visual judge and author-defined ground-truth labels across 24 images. This preliminary result comes from the author’s own study and should not be treated as independent validation of the framework.
Risk tiers determine where humans intervene
Applying identical controls to every autonomous action can erase much of the productivity benefit that attracted organizations to AI coding tools. A more practical model links autonomy to business impact, reversibility, and access:
- Low risk: automated checks. Documentation, unit tests, and similarly reversible work can rely primarily on automated testing, static analysis, and policy checks.
- Medium risk: review and rollback. Dependency updates, regression changes, and other changes with broader impact may require additional review, traceable approvals, and a defined rollback path.
- High risk: human approval and least privilege. Production configuration, payments, authentication, and other sensitive systems can operate behind explicit human approval, tightly scoped permissions, and stronger deployment controls.
The distinction is less about whether AI produced the code than about what the change can affect.
The development lifecycle provides natural checkpoints for many of these controls. Before coding begins, teams establish agent identity, permitted tools, and repository access. During generation, automated checks flag prohibited libraries or policy violations. Before merge and deployment, testing, static analysis, security scans, and approval gates vary based on risk. After deployment, monitoring and rollback mechanisms provide additional evidence.
Small, reversible deployments further limit exposure. If an agent-generated change causes unexpected behavior, a narrowly scoped release is generally easier to trace and reverse than a large batch of changes.
Governance metrics enable continuous improvement
A governance model is more useful when its effectiveness can be measured. Four metrics provide a focused starting point:
- Unsafe tool-call rate. This rate shows how often agents attempt prohibited, unexpected, or unnecessarily privileged actions. A rising rate indicates weak permission boundaries or poorly scoped instructions.
- Rollback frequency. The rollback frequency measures how often agent-assisted changes reach an environment and later require reversal. Trends help identify workflows that warrant closer investigation.
- Defect escape rate. The defect escape rate measures the proportion of defects that reach production rather than being caught earlier.
- Complete audit-trail rate. This rate assesses the percentage of agent actions with enough recorded evidence to reconstruct what happened. Missing prompts, tool calls, approvals, or deployment records create critical blind spots.
These measures can feed back into the risk model. A workflow with consistently complete telemetry, low defect escape rates, and fewer post-release bugs can support greater autonomy. A workflow showing repeated rollbacks or incomplete audit records may warrant tighter permissions or additional review.
As coding assistants become autonomous agents, governance increasingly resembles an observability and control system for machine action. The objective is not to add bureaucracy around AI-generated code. It is to create sufficient identity, telemetry, evidence, risk calibration, and recovery capability to understand what agents are doing and use the resulting data to determine how much autonomy each class of work can support.
About the Author: Suneet Malhotra is a senior manager of test engineering with more than 18 years of experience in software quality, test automation, and engineering leadership. His expertise spans mobile, web, IoT, and cloud platforms, with a focus on AI-driven testing, CI/CD, release management, and scalable automation frameworks. He has built and led global quality engineering teams while helping organizations improve software reliability, release velocity, and operational performance. He holds a Master of Science in computer science from the University of Southern California. Connect with Suneet on LinkedIn.
