AI Agents for Pharma: Audit Trails and Data Validation

At a mid-size biotech running a Phase 2 adverse-event trial, the retrained sensitivity model came back three points more accurate this quarter, and nobody on the data team could say why. The scientist who last touched the training set had rotated onto a different program. The last regression report was a spreadsheet from four months earlier, hand-updated, half the columns stale. When regulatory affairs asked for the audit trail ahead of the interim submission, the answer was a shrug and a promise to reconstruct it from Slack history. This is the actual failure mode that AI agents for pharma need to fix, not by writing a slicker Python script, but by treating validation and audit trail generation as a designed, versioned, repeatable pipeline instead of whatever the last person on the project remembered to save before they left. Pharma data teams carry three chronic pains that compound each other: an audit trail requirement attached to every model change that nobody owns end to end, clinical data quality problems (duplicate patient records across sites, unit mismatches between lab systems, missing patient-reported outcomes) that surface three weeks before a submission deadline instead of at ingestion, and 21 CFR Part 11 compliance that gets treated as a checkbox until an FDA inspector asks for the electronic signature history on a specific record.

Why ChatGPT and Cursor both fail pharma data teams

Ask Claude.ai or ChatGPT to explain 21 CFR Part 11 and you get a competent summary of electronic records, electronic signatures, and audit trail requirements. Ask it to produce the actual audit trail for the adverse-event model your team retrained six times this quarter, and it has nothing. It has no memory of your training pipeline, no record of what changed between model version 4 and version 5, no golden dataset it can compare a new model against, and no mechanism for writing that comparison to a versioned artifact an FDA inspector could review during a data integrity audit. A generalist chatbot can describe the regulation. It cannot produce the evidence that you are complying with it, because evidence requires a pipeline that runs the same way every time, not a conversation that starts fresh every session.

Cursor and GitHub Copilot solve a narrower and different problem. They are fast, context-aware autocomplete inside the editor, and if you ask either one to write a deduplication function for patient records, you will get a reasonable pandas snippet. What you will not get is a systematic validation pipeline that checks every incoming site export against a schema, flags range violations in lab values before they reach the analysis dataset, or a regression suite that runs a labeled golden set of adverse-event cases against every retrained model and fails the build if sensitivity drops below a clinical threshold. Autocomplete tools respond to the file you have open. They do not design the pipeline, and they have no concept of a compliance gap that spans your access control model, your audit logging, and your data validation layer all at once. In pharma, that gap is exactly where warning letters come from.

Evals, Clean, and Warden: the audit trail stack for pharma model changes

Tonone's answer for pharma and biotech data teams is not one agent trying to cover clinical data quality, model validation, and access compliance simultaneously. It is three specialists that each own a piece of the problem and hand off cleanly. Evals owns the audit trail for every model iteration: designing the eval harness, building the regression suite against a golden set, and analyzing what changed between versions. Clean owns the data quality layer underneath the model: validating trial data pipelines, catching duplicate patient records and unit mismatches before they poison an analysis dataset. Warden owns the access control and audit logging inventory that 21 CFR Part 11 actually requires, restricted system access, tamper-evident logs, and a documented compliance gap analysis nobody has to reconstruct from memory during an inspection.

Tonone's Evals builds the automated regression suite that becomes the audit trail for every model change in a pharma pipeline, golden set, threshold, and version diff, all in one place.

Designing the eval harness before the model changes again

The evals-harness skill is where this starts. Before the next retraining cycle, Evals designs the task schema for the adverse-event detection model: what counts as a true positive, what the dataset versioning scheme looks like, what metrics get tracked (sensitivity, specificity, false negative rate on the rare-event classes that matter most clinically), and how the eval is wired as code rather than a manual spreadsheet someone updates when they remember to. This is the step generalist tools skip entirely, they will happily discuss what a good eval should measure, but they will not produce the versioned harness that runs identically every time a data scientist retrains the model.

Building the regression suite that is the audit trail

Once the harness exists, evals-regress builds the automated regression suite: a golden set of labeled adverse-event cases, a CI hook that runs the suite every time a model artifact changes, and threshold alerting that fails the pipeline if a new model regresses below the clinical bar the team set. This is the artifact that answers the audit trail question directly, every model version has a timestamped run against the same golden set, with a pass or fail decision and the metric delta from the prior version. When regulatory affairs asks what changed between version 4 and version 5 of the adverse-event model, the answer is a report, not a reconstruction.

Validating the trial data before it reaches the model

None of the eval work matters if the data feeding the model is compromised. clean-validate designs the validation pipeline that sits between site data exports and the analysis dataset: schema checks against the expected trial data model, range validation on lab values (catching an mmol/L value that got exported as mg/dL from one site's LIMS), and quality metrics tracked over time so a data manager can see completeness trending down before it becomes a protocol deviation. Clean does not replace the biostatistics team's SAP-driven validation, it is the automated first layer that catches the mechanical errors, duplicate patient IDs across site exports, malformed dates, out-of-range values, before they ever reach a statistician's desk.

Tonone's Clean validates trial data pipelines at the schema and range level, catching duplicate patient records and unit mismatches before they reach the analysis dataset.

Closing the Part 11 gap nobody wants to own

The third pain point, 21 CFR Part 11 compliance, is fundamentally an access control and audit logging problem, and that is Warden's domain. warden-recon produces a full inventory of secrets management, IAM, encryption, audit logging, and compliance gaps against the current system, exactly the categories a Part 11 predicate rule assessment needs: who can access electronic records, whether that access is logged immutably, whether encryption at rest meets the bar, and where the gaps are before an inspector finds them first. This does not replace a validated QMS or a formal Part 11 assessment performed by regulatory specialists, but it gives the engineering and data team a concrete, current gap list instead of an assumption that everything is fine because nobody has been cited yet.

A worked example: Aldrin Therapeutics' adverse-event model

Aldrin Therapeutics is running a Phase 2 trial across 41 sites with 1,240 patients enrolled, using a predictive model that flags adverse events from a combination of EHR vitals and patient-reported outcome (PRO) diaries. The data team inherited the pipeline from a contractor eight months ago and has retrained the model six times as new site data has come in. Nobody on the current team wrote the original eval harness, and the last audit trail entry is a comment in a Jupyter notebook from March.

First, Clean runs clean-recon against the existing ingestion pipeline and finds what the team suspected but had not measured: 3.8% of patient records across the 41 site exports are duplicates from re-submitted CRFs, and two sites have been exporting lab glucose values in mmol/L against the trial's mg/dL standard, silently skewing 4% of the analysis dataset for those sites. clean-validate is then used to design the fix: a schema check against the trial's data dictionary, a range validator that flags glucose values outside the expected mg/dL band for manual review, and a deduplication rule keyed on patient ID plus visit date plus site ID.

With clean data flowing in, Evals builds the audit trail the team never had. evals-design defines the task: sensitivity on a labeled golden set of 200 confirmed adverse-event cases must stay at or above 0.92, specificity at or above 0.85, and any model that regresses on either metric fails CI before it is allowed into the trial's decision pipeline. evals-regress wires this as an automated suite that runs on every retraining event, and evals-analyze produces the version-over-version comparison. Here is what that comparison looks like after the seventh retraining, folded in after the data fixes:

Aldrin AE Model, Regression Report v7 (post data-cleaning fixes)
Golden set: 200 labeled adverse-event cases, v3, frozen 2026-04-02

────────────────────────────────────────────────────────────
Metric              v6 (pre-fix)   v7 (post-fix)   Threshold   Result
Sensitivity          0.89           0.94            >= 0.92     PASS
Specificity          0.83           0.91            >= 0.85     PASS
False negative rate  0.11           0.06             <= 0.08    PASS
Duplicate patient    3.8%           0.1%             <= 0.5%    PASS
  records in train
Unit mismatch rows   4.0% (2 sites) 0.0%             0%         PASS
────────────────────────────────────────────────────────────
Delta v6 -> v7: sensitivity +0.05, specificity +0.08
Attribution: data-cleaning fixes (clean-validate), not model change.
Audit trail: model artifact SHA, golden set version, threshold config,
  and this report are stored together, timestamped, immutable.
Next: Warden recon flagged audit-log retention at 90 days;
  Part 11 recommends indefinite retention for trial-critical records.

That last line is Warden's contribution. warden-recon inventoried the audit logging configuration across the data pipeline and found the retention window was set to a default 90 days, adequate for a typical SaaS product, inadequate for records that need to survive an FDA inspection years after database migration. The data team extended retention and flagged the finding for the regulatory affairs team's formal Part 11 assessment. None of this required a new hire or a six-week compliance consulting engagement. It required three specialists doing three specific jobs, in sequence, with the output of each becoming the input to the next.

AI agents for pharma vs the alternatives

The comparison below is specific to what a pharma or biotech data team actually needs from an AI tool: not general coding help, but audit trail generation, trial data validation, and compliance gap inventory that a generalist chatbot and an autocomplete tool have no mechanism for producing.

Tonone's Warden inventories the access control and audit logging gaps that 21 CFR Part 11 requires, before an FDA inspector finds them during a data integrity review.

Install and try

Tonone is free and MIT-licensed. Install it once and Evals, Clean, Warden, and the rest of the specialist agents are available in your Claude Code session. You pay only for the Claude Code token usage during the work itself.