# Beyond Code Review # Building Change Confidence in an AI-Native SDLC An engineer can give Claude, Cursor, Devin, Codex, or another coding agent a task and increasingly get back not just code, but tests, documentation, and a pull request. That changes an assumption our engineering processes have been built around for decades: **Generating the change is no longer necessarily the expensive part. Establishing confidence in the change is.** I recently came across framing of engineers as **"Indexers" and "Instrumenters."** An Indexer relies heavily on engineers maintaining a deep mental model of the codebase. An Instrumenter externalizes more of that knowledge into systems that provide evidence about whether software behaves correctly. I find myself leaning toward the Instrumenter side. But I think there's a harder question underneath it: > **What exactly should we instrument for?** * * * ## Passing Tests Doesn't Mean Safe to Ship Tests are an obvious answer. Did the unit tests pass? Did integration tests pass? Did the build succeed? Did static analysis find anything? All useful signals. But consider two changes. **PR A** modifies an internal Tier-4 service with few dependencies and limited customer impact. **PR B** modifies a Tier-1 service sitting on a critical customer path, has twelve downstream consumers, and touches an area associated with two previous P1 incidents. Both PRs could have: * all tests passing, * high test coverage, * successful builds, * clean AI reviews. Yet they clearly don't carry the same risk. The difference isn't necessarily visible in the code. It's in the **context surrounding the change**. And that's where I think code review needs to evolve. * * * ## The Unit of Review Should Be the Change Most code-review systems are fundamentally diff-oriented. ```text Pull Request | +-- Files changed +-- Lines changed +-- Tests +-- Reviewer comments ``` This model made sense when humans wrote most of the code and reviewers could reasonably build context by reading it. That becomes harder when agents can generate substantial changes in minutes. Instead of asking only: > What changed in the code? we should also be able to ask: > What does this change mean to the system? That requires enriching the PR with context outside the repository. ```text Developer / AI Agent | v Intent (Jira) | v Pull Request | v Services Changed | +-- Service Tier +-- Ownership +-- Dependency Graph +-- Blast Radius +-- Incident History +-- Test Evidence +-- Deployment Strategy +-- Runtime Health ``` Now the review isn't just about a diff. It's about a **change entering a running system**. * * * ## What Would That Look Like? Imagine opening a PR and seeing: ```text Change Context Services: checkout-api Tier 1 pricing-service Tier 2 Dependency impact: 3 upstream services 11 downstream services Operational history: 2 P1 incidents in last 12 months Change characteristics: API contract modification Database schema change Required evidence: Contract tests Integration tests Canary deployment SLO validation during rollout ``` Notice what this isn't. It isn't an AI saying: > "Risk score: 87. Do not merge." I'm skeptical of reducing complex engineering risk to an opaque number. Instead, the system should **assemble evidence** and use deterministic policies where appropriate. That gives both humans and AI reviewers substantially better context for making decisions. * * * ## Risk Should Influence the Path Once we understand the context of a change, the SDLC itself can become adaptive. Today, many organizations effectively have: ```text Change ↓ Review ↓ Test ↓ Deploy ``` with roughly the same controls applied to every PR. But why should every change take the same path? A low-risk change might require: ```text Review → Tests → Deploy ``` A higher-risk change might require: ```text Deep Review ↓ Integration / Contract Tests ↓ Ephemeral Environment ↓ Canary ↓ SLO Validation ↓ Progressive Rollout ``` The important part is that this decision shouldn't be driven primarily by something simplistic like **lines changed**. A two-line configuration change affecting traffic routing for a Tier-1 service can be far more dangerous than a 500-line isolated refactor. The better signals are things like: **service criticality + dependency impact + type of change + operational history + available evidence** * * * ## So Where Does Code Review Fit? Code review remains extremely important. But I don't think its job should be to provide all of our production confidence. Review is good at answering: * Does the implementation match the intent? * Does this violate an architectural boundary? * Is an API contract changing? * Are there concurrency or failure-mode concerns? * Are security or data assumptions changing? * Is the implementation unnecessarily complex? Some of those questions are excellent candidates for AI assistance. But even the best reviewer can't completely determine how a distributed system will behave in production by inspecting a diff. So confidence should accumulate across the lifecycle. ### Code Review Validate **intent, implementation, architecture and obvious risk**. ### Pre-production Validate **actual behavior** through integration tests, contract tests, ephemeral environments, performance testing, or whatever evidence the change warrants. ### Deployment Control **exposure** through feature flags, canaries, progressive delivery, health gates and automated rollback. ### Production Validate **reality** through SLOs, errors, latency, telemetry and customer-impact signals. This gives us something closer to: ```text Intent ↓ Change ↓ Review Evidence ↓ Pre-production Evidence ↓ Deployment Evidence ↓ Production Evidence ↓ Learning ``` No individual stage needs to prove that a change is safe. **Confidence accumulates.** * * * ## Incident History Is an Underused Engineering Signal This is the part I find particularly interesting. Most incident data lives far away from the pull request. PagerDuty knows which services caused incidents. Datadog knows what happened at runtime. Postmortems describe failure modes. The service catalog knows criticality and ownership. The dependency graph knows blast radius. GitHub knows what is changing. Jira knows why. But the engineer reviewing the next change often sees only a fraction of this information. Imagine instead that a reviewer sees: > This PR modifies the retry mechanism in `checkout-api`. The same component was involved in two previous P1 incidents caused by retry amplification. That immediately changes the review conversation. More importantly, it might automatically change the required evidence: ```text Previous incident pattern detected ↓ Failure-mode integration test ↓ Canary required ↓ Stricter SLO gate ``` Now incident management is no longer purely retrospective. **Production history becomes an input into future engineering decisions.** * * * ## Close the Loop This leads to what I think is the more interesting architecture. ```text AI Agent / Developer | v Intent | v Pull Request | v Change Context | v Required Evidence | v Deploy | v Production Signals | v Incident / Success | +--------------------+ | v Future Changes ``` Suppose a particular type of database migration repeatedly causes incidents. Future changes matching that pattern should automatically require stronger evidence. Conversely, suppose thousands of changes of a particular type deploy successfully with the same controls. Perhaps some unnecessary friction can eventually be removed. The system starts learning not from whether code was generated by AI, but from **what actually happened when changes reached production**. That is a much more useful feedback loop. * * * ## AI Makes This More Necessary — and More Possible None of the individual pieces here are new. We already have: * GitHub for changes, * Jira for intent, * service catalogs for ownership, * dependency graphs for topology, * CI systems for test evidence, * PagerDuty for incidents, * observability platforms for runtime health, * deployment platforms for progressive delivery. The problem is that they mostly operate as separate systems. AI makes this interesting for two reasons. First, AI increases the **throughput of changes**, making human attention increasingly scarce. Second, AI is very good at **synthesizing context across systems**. The opportunity isn't another dashboard where engineers need to manually correlate ten tools. It's bringing the relevant evidence to the point where the decision is being made: **the change itself.** * * * ## Human Review Becomes Scarcer — and More Valuable This doesn't mean removing engineers from code review. Quite the opposite. If AI-generated changes increase significantly, human attention becomes one of the most expensive resources in the SDLC. We should spend it deliberately. An engineer probably shouldn't spend twenty minutes verifying boilerplate that deterministic systems can validate. But that same engineer absolutely should spend twenty minutes questioning an architectural change affecting a critical customer path with a history of incidents. The goal isn't: > AI replaces human review. It's: > **Use instrumentation to determine where human judgment matters most.** * * * ## From Code Confidence to Change Confidence For years, engineering systems have optimized heavily around validating code. But what enters production isn't just code. It's a **change to a running system** with dependencies, customers, historical failure modes and operational consequences. As AI makes generating that change increasingly inexpensive, I think the scarce resource shifts. It becomes **evidence that the change can be trusted**. So perhaps the question we ask before merging shouldn't only be: > Has someone reviewed this code? It should increasingly become: > **What evidence do we have that this change is safe to ship?** That shift—from **code review to change confidence**—may become one of the more important changes in how we design developer platforms for an AI-native SDLC.