Acceptance criteria and independent review caught the computational faults of an autonomously built engineering model, but not its modeling errors
An account of the controls used while AI coding agents built a steam and cogeneration model, what those controls detected and what they did not.
Keywords: AI agents; scientific software; model governance; independent review; change control; verification
Abstract
AI agents can now build substantial engineering models with limited human intervention, which raises the question of what controls make their results trustworthy. We report the governance of a modeled steam and cogeneration system built in 8 phases by AI coding agents. Controls comprised 55 acceptance criteria specified before each phase was built, designated and cited input files for all numeric inputs, independent review by models from other developers, a change log requiring human approval for every deviation (42 entries, 40 approved), and 223 logged decisions. The controls detected every fault class they were designed for, including a multistart spread above its threshold, a schedule that was infeasible as returned by the solver, and unreproducible outputs. Two of 12 independent reviews returned a failing verdict; both concerned deviations already approved by the supervising engineer and were reconciled. The controls did not detect errors of modeling judgment: a regional price mismatch, a validation check that cannot fail and unrealistic low-pressure superheat were recorded as disclosures rather than findings. We conclude that modeling judgments must themselves be written as acceptance criteria if they are to be enforced.
1. Introduction
Autonomous coding agents can write, run and revise scientific software. Their output can be wrong in ways that are hard to see, and reproducibility alone does not guarantee correctness [1]. Engineering practice offers established controls: requirements fixed in advance, independent review, change control and traceable inputs. Their effectiveness when the developer is an AI agent has rarely been documented.
This report describes the controls used to build a modeled steam and cogeneration system, what they detected, and what they did not.
2. Methods
2.1 Pre-specified acceptance criteria
Before each phase was built, its acceptance criteria were written into the specification, each as a measurable quantity with a threshold: 55 criteria across 8 phases. A criterion is satisfied only by a measured value, and each has a negative test that deliberately breaks the physics and must cause it to fail, in the manner of mutation testing [2].
2.2 Input management
Numeric inputs for the first phases are held in a single inputs module, each with its citation and verification status; later phases added three separately cited input files whose contents are checked against recorded hashes before use. A scanner checks source files for numeric literals outside the designated inputs.
2.3 Independent review
A development agent wrote each phase. Two reviewing models from other developers, DeepSeek and Meta, replayed each phase from a read-only snapshot with their own scripts and returned a verdict; a coordinating agent replayed the reviewers' scripts before accepting their numbers. A separate review model assessed routine decisions, and decisions reserved to a human were taken by the supervising engineer.
2.4 Change control
Any deviation from the specification required an entry in an append-only change log, approved in the supervising engineer's recorded words before taking effect. Failed and superseded values remain recorded beside the values that replaced them.
3. Results
3.1 Change log
The change log holds 42 entries; 40 were approved and 2 remain proposals. Changes arrived in two bursts, during remediation of the first phase and during the autonomous construction of phases three to eight (Figure 1).
3.2 Faults detected
The first phase required 9 remediation passes, and 19 independent review reports from 6 model developers are retained for it, covering the environment, the property layer and data ingestion. Acceptance waited for a review whose evidence reconciled; several were rejected for lacking it.
In the dispatch phase, the first formulation's twenty starts spread by 1.8 percent, above the half-percent threshold; the failure led to an exact treatment of the boiler split. In the coupling phase, a reported schedule satisfied its balance criteria under a re-solve convention but, evaluated directly on the solver's returned state, had a residual of 1.1 × 10-8 and 168 hours above the feasibility tolerance. A reviewer detected this, and the schedule was replaced by a certified one. The same phase's specified multistart spread 1.9 percent; that failure is recorded with the approved deviation that replaced it.
3.3 Review findings
Across the reviews of phases three to eight, reviewers raised 5 blocking findings, 15 moderate and 32 minor (Figure 2). Two of 12 reviews returned a failing verdict. In both, every blocking finding concerned a deviation the supervising engineer had already approved; the findings were reconciled and the phase accepted with the failing verdict retained in its record. A failing review can thus be overruled, and here it was twice.
The phase records hold 223 decision records: 198 settled by quorum and 16 reserved to the supervising engineer. The review model concurred 90 times and objected 8 times; 13 records carry the engineer's outcome and 97 carry no final outcome in the exported records.
3.4 Final state
The final phase measured 0 differing files between two consecutive full runs, 0 reported numbers without a traced source, 0 unresolved data-source entries and 0 failing tests. At the first phase's acceptance, the literal scan found 0 numeric literals outside the inputs module across 31 files.
4. Discussion
The controls made the model's numbers checkable and difficult to misstate. They did not make the modeled scenario realistic. Several problems were recorded only as disclosures, because no criterion measured them:
- The site is placed on the California grid but priced on US-average industrial gas, and California industrial gas prices lay above the entire tested range in every recent year.
- The chlor-alkali cell's energy-intensity check passes for every sensitivity case and therefore cannot discriminate between them.
- The low-pressure header carries nearly two hundred degrees of superheat that a real site would remove, while every balance closes.
- Time-varying prices were specified but not implemented, so the multi-hour schedule repeats a single hour.
- The citation for the grid emission factor refers to a source file version that was not retained; the value was checked against the primary source.
The pattern is consistent. Acceptance criteria measure whether the model computes what it was specified to compute, and the surrounding controls make that measurement hard to falsify. Whether the specification was right is a judgment that no criterion encoded, and the reviewers, replaying the criteria, did not make it. Part of this gap can be closed by writing modeling judgments as criteria: a regional price check, a discriminating validation band and a realism bound on header temperature.
5. Limitations
- This is a single case study of one build process and one model.
- Counts are taken from the project's own records; decisions without a recorded outcome are not interpreted.
- The effectiveness of the controls against faults outside the classes observed here is not established.
6. Conclusions
Pre-specified acceptance criteria, independent review and human-approved change control detected the computational and process faults of an autonomously built engineering model, but not its errors of modeling judgment. Such judgments should be specified as measurable criteria if they are to be enforced.
References
- Peng RD. Reproducible research in computational science. Science, 334(6060):1226-1227, 2011.
- Jia Y, Harman M. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649-678, 2011.