AI Agents for QA Teams: Triage Flaky Suites, Catch Regressions First

QA teams don't have a testing problem. They have a triage problem. The checkout regression suite runs 1,240 tests and takes three hours and forty minutes on CI, which sounds tolerable until you notice the team merges to main four times a day and every deploy queues behind the last green run. Add the 34 tests marked @skip because someone tagged them flaky eighteen months ago and nobody has owned fixing the root cause since. Add a p95 latency regression on the payment endpoint that shipped on a Tuesday and got caught by a support ticket the following Sunday, five days after QA had already signed off. This is the real search behind AI agents for QA teams: not "we need more tests written," but "we need someone to triage the tests we already have, fast, before the sprint closes out." That is a different job than scaffolding a new spec file, and it is the one most AI tooling quietly ignores.

Why ChatGPT and Cursor don't fix a flaky suite

Paste one flaky test into ChatGPT or Claude.ai and you get a plausible-sounding answer about race conditions and async waits. What you don't get is the thing that actually matters: correlation across the other 33 flaky tests in the suite, a read of the last 40 CI runs to see which ones fail together, or a classification of which failures share a root cause versus which are genuinely independent. A generalist chatbot answers the question you typed about the file you pasted. It has no visibility into your CI history, no memory of which tests were quarantined last quarter, and no way to tell you that six of your flaky tests share the same underlying problem: a shared fixture that leaks state between parallel workers. It will happily rewrite the one test you show it and leave the other 33 exactly where they were.

Cursor and GitHub Copilot solve an adjacent but different problem. They are excellent at scaffolding a new Playwright spec while you are typing it, autocompleting selectors, suggesting assertions, filling in boilerplate. But an autocomplete layer has no opinion about the 1,240 tests you already have. It doesn't audit existing suite health, doesn't flag that your checkout suite's runtime has crept from ninety minutes to three hours and forty minutes over two quarters, and doesn't compare this release's API latency against the last one to catch a regression before it merges. When you ask Cursor to help with a flaky test, you get a suggestion for the file open in your editor. When the actual problem is suite-wide triage or release-over-release performance comparison, an inline completion tool is answering a question nobody asked.

The mismatch is structural. Both tools are built to respond to what's in front of them, one prompt, one file, one question at a time. QA at any real scale is a systems problem: flaky tests compound because nobody owns the audit, regression suites bloat because nothing prunes them, and performance regressions reach production because nobody is diffing latency baselines release over release. Fixing that requires an agent built for test strategy and suite health specifically, not a chat window that happens to know what Playwright is.

Proof, Bench, and Chaos: AI agents for QA teams, built for Claude Code

Proof is tonone's QA and testing engineer: test strategy, E2E suites, integration testing, and flaky test triage. Where a generalist tool reacts to whatever test you paste in, Proof works at the suite level, treating the whole regression suite as the unit of work rather than any single file. Bench sits next to Proof as the API performance specialist, and Chaos rounds out the bench for teams whose QA scope extends into resilience testing. Together the three cover the three failure modes that actually reach customers before QA does: functional regressions nobody triaged, performance regressions nobody diffed, and failure modes nobody rehearsed.

Triaging flaky tests without guessing

proof-audit inventories an entire test suite and surfaces flaky tests, slow tests, coverage gaps, and testing anti-patterns as a structured report, not a one-off fix. It reads CI history across recent runs rather than a single execution, which is the only way to tell a genuinely flaky test (fails under specific, reproducible conditions) from a test that failed once because a shared CI runner was under load. The output groups failures by shared cause: a leaking fixture, an unbounded async wait, a timezone-dependent assertion, so a team fixes the underlying pattern once instead of patching thirty symptoms.

Planning coverage before the next sprint, not after an incident

proof-strategy produces a test strategy for a project or feature: a risk map of what's most likely to break and how badly, decisions about which layer to test at (unit, integration, E2E) for each risk, coverage targets that are actually achievable, and the CI configuration to enforce them. This is the step most QA teams skip under sprint pressure, they add tests reactively, after a bug ships, instead of deciding up front what the payment flow's failure modes are and testing for them deliberately. proof-recon is the prerequisite step for teams inheriting a suite they didn't build: it inventories frameworks, coverage, and CI integration so proof-strategy has real data to work from instead of assumptions. And proof-e2e builds the actual end-to-end specs for critical user journeys, Playwright or Cypress, page objects, setup and teardown included, once the strategy has identified which journeys matter most.

Tonone's Proof audits an entire test suite for flaky, slow, and anti-pattern tests as a single structured report, not a one-off fix to whatever test you paste in.

Catching regressions before they reach a customer

Bench owns the performance half of the QA problem: latency profiling, throughput testing, and regression detection. bench-recon audits existing performance testing to find missing benchmarks, stale baselines, and CI gaps, most teams have a benchmark for one or two hot paths and nothing for the rest. bench-profile designs the actual benchmark, test scenarios, metrics, tooling, for an API endpoint that has none yet. And bench-compare is the one that matters most for QA teams specifically: it compares API performance across versions, release over release, and does root cause analysis down to the responsible commit. That is the difference between a customer reporting a slow checkout page five days after release and catching the same regression in CI before the merge is even approved.

Rehearsing failure before it happens in production

Chaos rounds out the bench for teams whose QA problem extends past functional and performance testing into resilience. chaos-recon audits existing resilience posture and identifies untested failure modes, chaos-design builds a specific chaos engineering experiment with a hypothesis, blast radius, and abort conditions, and chaos-game designs a full game day, simulated failure scenario, runbook, and post-event review. A QA team that has triaged its flaky suite and closed its performance regression gap is usually the same team ready to ask "what happens when the payment provider times out mid-checkout," and that's Chaos's job, not Proof's or Bench's. Bringing all three onto the same QA bench means functional correctness, performance, and resilience get the same level of rigor instead of resilience being the thing nobody gets to.

A worked example: triage day at Lockstep

Lockstep is a mid-size billing SaaS with a four-person QA team and a regression suite that has quietly grown to 1,240 tests over two years. The suite covers checkout, invoicing, and account management, and it takes three hours and forty minutes to run in CI. Merges to main happen four to six times a day. QA's actual complaint isn't the runtime, it's that 34 tests are marked @skip for flakiness, each one added by a different engineer under sprint pressure, none of them ever revisited. Meanwhile, two weeks ago, a refactor to the invoice PDF generator shipped Tuesday and quietly pushed p95 response time on POST /invoices/:id/finalize from 220ms to 890ms. Nobody caught it in review. A customer opened a support ticket the following Sunday complaining that finalizing an invoice "hangs for a few seconds." QA found out about their own regression from a support ticket.

The team runs /proof-audit against the full suite. Instead of rewriting one flaky test at a time, Proof pulls the last 40 CI runs, clusters the 34 skipped tests by failure signature, and returns a triage report grouped by root cause rather than by file. In parallel, the team runs /bench-compare against the finalize endpoint, comparing the current release branch against the last three tagged releases to find exactly where the latency crept in.

Proof Audit, Regression Suite Triage (checkout/invoicing/accounts)
Scope: 1,240 tests, 40 CI runs analyzed, 34 tests currently @skip

-----------------------------------------------------------------
Cluster A, Shared fixture state leak (19 tests)
  Root cause: seedInvoiceFixture() writes to a shared org_id across parallel
  workers. Tests assert on row counts that drift under concurrency.
  Fix: scope fixture to worker-local org_id. Not a timing issue, don't add waits.
  Estimated fix: ~3h for all 19, single PR.

Cluster B, Unbounded async wait (9 tests)
  Root cause: tests assert on a webhook callback with a fixed 2s timeout;
  callback p95 is 2.4s under CI load.
  Fix: replace fixed timeout with polling assertion, 10s ceiling.
  Estimated fix: ~1.5h.

Cluster C, Genuinely flaky, no shared cause (6 tests)
  Recommend: keep @skip, file individually, low priority.
-----------------------------------------------------------------
28 of 34 skipped tests fixable in ~4.5h combined. Suite runtime after fix +
de-dup: est. 3h40m -> 2h55m (removes redundant fixture re-seeding per test).

Bench Compare, POST /invoices/:id/finalize
  v2.14.0 (3 releases back):  p50 84ms   p95 218ms
  v2.15.0 (2 releases back):  p50 86ms   p95 224ms
  v2.16.0 (last release):     p50 91ms   p95 231ms
  current branch:             p50 302ms  p95 891ms
  Regression introduced: commit a3f9e21, "refactor PDF generation to sync render"
  Root cause: PDF render moved from queued job to inline request handling.
  Recommendation: revert to async render + webhook callback, or block merge.

That's the whole shift in what QA teams can do with AI agents for QA teams built for the job. Proof didn't tell Lockstep to rewrite 34 tests one at a time, it clustered them by root cause and pointed to a single fixture fix that resolves 19 of the 34 in one PR. Bench didn't wait for a customer ticket, it caught the finalize endpoint regression on the release branch, before merge, with the exact commit that introduced it. The four-person team spends the sprint fixing two root causes instead of triaging 34 symptoms, and the performance regression never reaches a customer. That's the actual leverage: not more tests, faster triage of the tests already written.

Tonone's Bench compares API performance release over release and identifies the exact commit responsible for a regression, before it merges.

None of this is a one-time cleanup. Lockstep's QA lead now runs /bench-compare against the last three tagged releases as a standing step before every release cut, not just this one, and re-runs /proof-audit at the start of each sprint rather than waiting for the skip count to climb back into the thirties. The point isn't a single triage day, it's making suite health and performance regression checks a repeatable cadence instead of a fire drill that only happens after a customer notices first.

Proof and Bench vs the alternatives

The comparison below is specific to what QA teams actually need day to day: triaging an existing suite, not just writing new tests; catching performance regressions release over release, not just profiling once; and understanding suite health well enough to take over an inherited codebase's tests without guessing.

Tonone's Proof and Bench turn QA triage from a per-test conversation into a suite-level and release-level analysis, catching the regression before a customer does.

Install and try

Tonone is free and MIT-licensed. Install it once and Proof, Bench, Chaos, and the rest of the agent roster are available directly inside your Claude Code session. You pay only for Claude Code token usage during the work itself.