HAAM Index / AI systems · July 6, 2026 · 14 min read

Every Coding Agent Needs a Regression Agent

AI can produce the next version of a product in minutes. Teams now need agents that remember what the previous version promised and detect when code, interfaces, performance, accessibility, data, or user journeys quietly move backward.

AI AgentsRegression TestingProduct QualityObservabilityAccessibility

A regression agent watches the whole promise of the product, then turns differences into evidence a human can understand and act on.

AI has made producing a change radically cheaper. Remembering every promise that change might break has become the expensive part.

Geoffrey Litt describes understanding as the new bottleneck. Agents can write more code than a person can comfortably read, yet the person still needs enough understanding to participate in the next creative loop. Litt's response is to improve the explanation layer through literate diffs, quizzes, interactive micro-worlds, and shared spaces.

That argument has a direct operational consequence. Teams need better ways to understand changes, and they also need software that continuously checks whether those changes damaged anything people already depended on. Human attention should be spent on consequential judgment, not on rediscovering every possible failure surface by hand.

GitLab's June 2026 AI Accountability Report makes the pressure visible. In its survey of 1,528 developers and technology buyers, 85% agreed that AI had shifted the bottleneck from writing code to reviewing and validating it. Faster generation moves unfinished work downstream when the quality system remains unchanged.

A regression agent is product memory with tools.

The definition must expand

A regression is any broken promise.

Software teams often treat regressions as code that used to pass and now fails. That is one important category, but products make many promises outside the test suite. A page can compile while becoming unreadable. A workflow can complete while losing keyboard access. An API can return 200 while changing the meaning of a field. A feature can ship successfully while increasing abandonment, support requests, or privacy risk.

The useful unit is the promise: a behaviour, visual relationship, speed threshold, permission boundary, semantic structure, data contract, or user outcome that the product had previously earned the right to treat as dependable.

Seven surfaces of regression

The test suite sees code. The product exists across a larger field.

A useful agent patrols several kinds of evidence and understands that a release can improve one surface while damaging another.

Code + behaviour

Existing functions, workflows, APIs, permissions, and edge cases still behave as expected.

Evidence: Unit, integration, end-to-end, contract, mutation, fuzz, and replay tests.

Example: Checkout still succeeds, but discount stacking now produces the wrong total.

Visual + interaction

The interface still looks, moves, responds, and adapts as designed across states and viewports.

Evidence: Screenshots, DOM and ARIA snapshots, design tokens, motion states, responsive layouts, and interaction traces.

Example: The button works, but its label wraps under the icon on a narrow screen.

Accessibility

People can still perceive, operate, understand, and use the product with different bodies, senses, devices, and assistive technologies.

Evidence: Automated checks, keyboard journeys, focus order, screen-reader semantics, contrast, zoom, target size, and human review.

Example: A modal looks correct while trapping keyboard focus behind the overlay.

Performance + reliability

The product remains fast, stable, and efficient under realistic conditions.

Evidence: Core Web Vitals, traces, logs, metrics, profiles, resource budgets, error rates, and load tests.

Example: A new personalization call adds 900 ms to the first useful interaction.

Data + integrations

Schemas, events, APIs, analytics, migrations, and third-party services still agree about what data means.

Evidence: Contract tests, schema diffs, event validation, migration checks, reconciliation, and sandbox transactions.

Example: The app renders normally while a renamed event silently breaks the revenue dashboard.

Security + permissions

A change does not widen access, expose data, weaken controls, or introduce a vulnerable dependency.

Evidence: Static and dynamic analysis, dependency scans, authorization matrices, secret detection, and attack-path review.

Example: A helpful agent action can now read records outside the user's workspace.

Product + meaning

The release still helps people complete the intended job without creating new confusion, exclusion, or business damage.

Evidence: Golden journeys, support signals, search quality, conversion, retention, abandonment, content checks, and qualitative feedback.

Example: Every test passes while a clearer onboarding message disappears and activation falls.

Why conventional CI reaches a ceiling

Pipelines execute checks. Agents can investigate change.

Traditional continuous integration remains essential. Fast deterministic tests are one of the most valuable forms of engineering leverage. Playwright can compare screenshots against accepted references. Pact can verify whether providers still honour consumer contracts. Accessibility scanners can catch concrete failures. OpenTelemetry can expose changes through logs, metrics, traces, and profiles.

Each system watches a slice. The difficult work begins when the screenshot changed because the copy changed, a trace slowed because an upstream service changed, an accessibility warning appears only after a modal opens, or an analytics event vanished without causing a visible error. Someone must connect the signals to the intent of the release.

An agent can inspect the diff, choose relevant journeys, run tools, open the product, compare states, query telemetry, search incident history, reproduce the anomaly, and explain the likely consequence. The value comes from orchestration and interpretation around the checks, not from replacing dependable checks with a probabilistic opinion.

The regression-agent job description

Watch, connect, reproduce, explain, remember.

Remember the promises

Maintain a versioned map of specifications, accepted screenshots, design-system rules, API contracts, accessibility expectations, performance budgets, incidents, and critical user journeys.

Predict the blast radius

Read the diff, dependency graph, ownership map, runtime traces, and previous failures to decide which surfaces and journeys deserve deeper testing.

Operate the product

Use browsers, devices, APIs, test environments, feature flags, seeded accounts, and observability tools to experience the change rather than merely inspect the files.

Investigate anomalies

Compare before and after states, reproduce failures, isolate likely causes, distinguish flaky noise from consistent change, and gather the smallest useful evidence packet.

Explain the consequence

Report what changed, which promise may be broken, who is affected, how confident the diagnosis is, which commit or dependency is implicated, and whether to block, warn, canary, or observe.

Turn memory into infrastructure

Convert every confirmed regression into a durable test, contract, baseline, alert, runbook, or product rule so the same failure becomes harder to repeat.

A practical operating loop

From diff to decision packet.

01A change appearsA pull request, dependency update, prompt edit, model swap, design-token change, CMS update, feature flag, or production anomaly enters the system.
02The agent maps impactIt connects the change to components, services, routes, journeys, contracts, audiences, and historical incidents.
03It selects evidenceCheap deterministic checks run first. Broader browser, visual, accessibility, performance, security, and production-trace checks follow where risk justifies them.
04It investigates differencesThe agent compares expected and observed behaviour, attempts reproduction, checks related signals, and estimates severity and confidence.
05A human receives a decision packetThe reviewer sees the changed promise, affected users, evidence, likely cause, suggested action, and any uncertainty that still requires judgment.
06The system learnsConfirmed regressions become tests. Accepted changes update baselines through explicit approval. False alarms improve the agent without erasing the underlying evidence.

Independence matters

The checker needs permission to disagree with the builder.

A coding agent that immediately reviews its own output often shares the assumptions that produced the mistake. Stronger systems create separation through a different context window, model, prompt, tool set, evidence budget, or explicit adversarial role. The point is to introduce a second perspective with its own success criterion: preserve the product's accepted promises while judging the proposed change.

This separation should remain pragmatic. Deterministic tests do not need personality. Ambiguous changes do need challenge, especially when a visual difference might be intentional, an accessibility issue may require human interpretation, or a performance cost might be an acceptable trade for a valuable feature.

The agent should never confuse “different” with “broken,” or “approved” with “safe to forget.”

Guardrails for the quality system

Automation needs boundaries around its memory.

01Never let a visual baseline update itself simply because the new screenshot exists.
02Keep the builder and checker roles separate enough to create real disagreement.
03Prefer deterministic assertions for known rules and agentic investigation for ambiguous evidence.
04Record the reason a change was accepted, not only the final pass or fail state.
05Treat flaky tests as defects in the quality system, not background weather.
06Use production observation to complement pre-release testing, with privacy and permission boundaries.
07Escalate high-impact uncertainty instead of manufacturing confidence.

Early evidence

Context helps agents test the right things.

A 2026 preprint called TDAD tested graph-based code and test impact analysis as a skill for coding agents. In its evaluated setup, test-level regressions fell from 6.08% to 1.82%, while resolution improved from 24% to 32%. The study is limited and should not be treated as a universal benchmark, but its direction is useful: agents improved when they received contextual information about which tests a change might affect.

AgentAssay approaches a related problem for non-deterministic agent workflows. Its central idea is that prompts, tools, models, and orchestration can regress even when a binary pass/fail test cannot capture the behavioural shift. That matters because modern products increasingly contain both conventional software and probabilistic agents. The regression system must watch both.

Human understanding remains central

The agent protects attention so people can participate.

Litt's strongest point is that people understand systems to participate in their evolution. Verification alone produces a gatekeeper. Understanding produces a collaborator who can notice a new possibility, question an assumption, or decide that a technically valid change points in the wrong direction.

The regression agent should support that participation. Its output should resemble a well-made explainer: the previous promise, the proposed change, the observed difference, affected journeys, supporting traces or screenshots, confidence, likely causes, and the decision that remains human.

This turns review from a pile of files and failing jobs into a shared model of what the product is becoming. The machine searches the space of possible damage. The person decides which promises still matter, which trade-offs are justified, and what the product should learn from the event.

What HAAM would build

A product-quality memory that follows every change.

The first useful version does not need to test everything. Start with one critical journey and five accepted promises: functional behaviour, a visual baseline, keyboard and semantic access, a performance budget, and the analytics event that proves completion. Connect the repository, preview deployment, browser automation, observability, and incident history. Let the agent prepare evidence before every merge and after every production release.

Over time, every confirmed defect expands the memory. Every accepted change records why the baseline moved. Every incident improves impact analysis. The result becomes a living account of how the product is supposed to work, what has hurt it before, and where the next change deserves suspicion.

Build an AI product with HAAM Add accessibility regression checks

Sources + further reading