A 501(c)(3) non-profit organization Applied AI research for public benefit
CheAI Research Inc.

Published PCA fault-detection rates for the Tennessee Eastman process hold with held-out control limits, at 0.719 against a printed 0.720

We re-scored eight published monitoring methods on the extended Tennessee Eastman data under a pre-registered protocol that keeps every control limit away from the test runs.

CheAI Research Inc. | Research article | July 2026

Abstract

Published fault-detection rates on the Tennessee Eastman process come from protocols that differ in where control limits are set. We asked whether the rates hold when every method learns from normal runs only, every limit comes from held-out normal runs, and unseen runs are scored forward in time. After a frozen pre-registration, we scored eight published methods on the Rieth et al. data, split by simulation run. A second, independent implementation re-derived the principal component analysis (PCA) and dynamic PCA results. For PCA, the mean detection rate over 17 faults was 0.719 for Q and 0.629 for T2, against 0.720 and 0.628 printed by Heo and Lee, and both gap intervals include zero. The false-alarm rate of the eight checked cells was 0.0101-0.0111 for a target of 0.01. For two nonlinear PCA networks, the fair-minus-printed gaps ranged from -0.1348 to +0.0099. Four dynamic PCA values depend on the order of scaling and lag stacking.

1. Introduction

The Tennessee Eastman process (TEP) is a widely used benchmark for data-driven process monitoring [1]. Its extended data set holds 500 simulation runs per fault [2]. Published detection rates on TEP are hard to compare. Studies drop different faults, and several set limits on the normal test data [3], [4]. Some tune network stopping while they watch monitoring performance [4]. Machine learning calls such choices leakage, and the reported skill can then exceed the skill on new data [5], [6].

A direct test needs models fitted on normal runs, limits from separate normal runs, and forward scoring within each run. We asked whether published TEP detection rates survive such a test. For linear PCA they did, and two nonlinear PCA networks fell short of their printed rates.

2. Methods

We split the Rieth runs by run, never by row (Figure 1). Fault-free training runs 1-400 formed the fit set F and runs 401-500 the calibration set C. The 500 fault-free test runs formed the normal set N, and the 500 runs per fault the fault set E. Each run has 52 variables at 3 min intervals, with fault onset after sample 160. Classic Braatz-group files served only to re-score papers that used them.

Protocol schematic with four run sets.
Figure 1. Fair evaluation protocol on the Rieth data. Models and scaling are fitted on F, the limit is the 0.99 quantile of the statistic on C, and runs in N and E are scored forward in time. In the as-published variant (dashed), the limit comes from N.

A pre-registration [7] froze methods, settings, splits, metrics and six predictions before any result existed. The methods were PCA with the settings of three papers [3], [4], [8], dynamic PCA and kernel PCA. They also included a serial and a parallel nonlinear PCA (NLPCA) network [4], two deep autoencoders [9] and independent component analysis (ICA) [10]. PCA monitors Hotelling's T2 and the residual statistic Q. No hyperparameter was tuned on C, N or E. The NLPCA networks trained for 1000 epochs without early stopping.

The limit was the 0.99 quantile of the statistic over C, for a target false-alarm rate (FAR) of 0.01. The detection rate of a fault is the mean over its runs of the fraction of post-onset samples in alarm. Aggregates average the 17 faults that Heo and Lee printed [4]. Each method was also scored under its paper's protocol. Intervals came from a bootstrap over run IDs that drew the same IDs for N and E, although the seed mapping between files is undocumented. Aggregates use Bonferroni intervals at level 0.99643 for the 14 headline cells.

A second implementation, written without the primary code, re-scored PCA and dynamic PCA and compared every value (tolerance 0.002 for rates, 10-6 relative for limits). For the NLPCA networks, it recomputed the rates from the stored alarms. An external audit checked the code paths that fit models and set limits.

3. Results

3.1 Held-out limits keep false alarms near target

For the eight independently checked cells, FAR on the 500 unseen normal runs was 0.0101-0.0111 (Figure 2). Of all 15 Rieth cells, 14 had 95 % intervals inside the pre-registered range of 0.005-0.02. The exception was the secondary ICA joint logic, at 0.0309.

False-alarm rate by method.
Figure 2. False-alarm rate on the 500 normal test runs under the fair protocol, with 95 % run-bootstrap intervals. The band is 0.005-0.02 and the dashed line the target. Filled markers were reproduced independently (for the networks, from stored alarms). SPE, squared prediction error. LSTM, long short-term memory.

3.2 PCA matches the printed rates, the networks do not

The fair PCA rates matched those printed by Heo and Lee fault by fault (Figure 3a). The 17-fault mean was 0.719 for Q and 0.629 for T2, against 0.720 and 0.628. The gaps were -0.0004 (interval -0.0012 to 0.0003) and +0.0014 (-0.00004 to 0.0026). All 17 per-fault rates lay within 0.05 of the printed values. Over all 20 faults the means fell to 0.613 and 0.537, because faults 3, 9 and 15 stay at the false-alarm level.

Three of the four NLPCA network rates fell below their printed values, and every Bonferroni interval excludes zero (Figure 3b). The serial network gained 0.0099 for Q and lost 0.0424 for T2. The parallel network lost 0.0355 for Q and 0.1348 for T2, with an upper bound of -0.1337. For the parallel T2, the fault-1 rate was 0.707 against a printed 0.989. The best network cell beat PCA Q by only 0.019.

Fair against printed per-fault rates.
Figure 3. Per-fault fair detection rates against the rates printed by Heo and Lee, for (a) PCA and (b) the two NLPCA networks (10-seed means). The band is ±0.05 around the identity line.

3.3 Independent re-derivation

The independent implementation reproduced 538 of the 542 values it could compare. All four disagreements are dynamic PCA quantities, the two control limits and the two fault-16 median delays. Every dynamic PCA rate and FAR agreed. The primary code scaled the lag-stacked rows, and the independent code scaled the 52 variables before stacking. Switching only this order turned all four rows to agreement.

3.4 Results not verified independently

Only the primary implementation computed these values. Under the published protocols, eight comparisons reproduced, one partly and five did not. The two autoencoders reached 17-fault means of 0.980 and 0.909, a measured result, not a ranking of methods. With fixed models, moving the limit from N to C changed the 17-fault means by at most 0.0036. Of six predictions, P1, P4 and P6 were falsified, P2 and P3 held, and P5 is unresolved.

4. Discussion

For PCA, the checked result is that held-out limits leave the printed rates intact. The unchecked fixed-model contrast suggests why. The small difference is expected when calibration and normal test runs are exchangeable simulations of the same operation.

That contrast explains little of the NLPCA gaps. The pre-registration replaced performance-guided stopping with 1000 fixed epochs and used 400 instead of 500 runs, and the parallel topology was our reading of the text. The parallel T2 seed means spanned 0.348-0.482, more than the largest protocol effect. We did not test which difference drives the gap.

No method on the Rieth data detected faults 3, 9 or 15 above 0.049, in line with earlier reports [3], [10]. Since the 20-fault means are about 0.1 lower, an aggregate should name its fault set. Leakage [5], [6] cost PCA no skill here. If published TEP rates carry optimism, these results point to sources such as performance-guided stopping. The NLPCA comparison cannot separate that possibility from reproduction differences.

The ICA joint logic raises an alarm when any of three statistics exceeds its own limit. If the alarms rarely coincide, their rates of about 1 % each add to about 3 %, so a combined alarm needs its own limit.

5. Limitations

The independent re-derivation disagreed on four dynamic PCA values. They depend on whether data are scaled before or after lag stacking, which the frozen text does not fix. The independent order was sealed first, so the disagreement stands. The values are outside the headline, and the criterion that every headline value agree is not met. For the networks, the independent check used stored alarms and limits, so it verifies scoring and aggregation, not network training or control limits. Kernel PCA, ICA, the autoencoders, all classic-data results, all reproduction and prediction verdicts and the calibration contrast were not re-derived independently. Two earlier independent runs were voided on procedural grounds before producing results, with minor procedural deviations in the comparison, and none exposed primary values. The audit did not read every code module.

The computing budget was extended from 72 h to about 90 h, a deviation from the pre-registration, but no seed, job, fault or split was cut. The Bonferroni family grew from 13 to 14 cells before any deep-model output existed. Intervals describe evaluation-run variability with models and limits fixed. They exclude fitting variability, which is large for some network and ICA cells. P5 has an unresolved scope: the stored verdict covers the 14 family cells, while the registered wording covers every method, and the ICA joint-logic fault-15 lower bound of 0.048 exceeds 0.03. Some ICA components did not converge within the iteration cap, components beyond about the third are not identifiable, and per-seed convergence counts are not summarised. ICA also used 33 variables instead of 52.

Models were fitted on 400 runs, not 500, which may contribute to the NLPCA misses. The autoencoders fitted weights on 300 of them and chose the stopping epoch on 100. Settings carried from classic-data papers have no Rieth source, and the dynamic PCA gaps on the Rieth data use classic-data constants, so they are not like-for-like. The classic files have unclear redistribution rights and were used locally only, and the Peng et al. comparison concerns arXiv version 1 only. Finally, TEP is a simulation, and the results hold for this benchmark only.

6. Conclusions

On the extended Tennessee Eastman data, a pre-registered fair evaluation reproduced the published PCA detection rates within their run-bootstrap intervals. Two nonlinear PCA networks missed their printed rates, and an unchecked contrast attributes little of that miss to the limit source. An independent implementation reproduced the headline values, except four dynamic PCA quantities that depend on an unfixed scaling order. Fair TEP comparisons should fix the fault set, limit data and training details, and report agreement with an independent implementation.

References

  1. Downs JJ, Vogel EF. A plant-wide industrial process control problem. Computers and Chemical Engineering 17(3), 245-255, 1993. DOI 10.1016/0098-1354(93)80018-I
  2. Rieth CA, Amsel BD, Tran R, Cook MB. Additional Tennessee Eastman Process Simulation Data for Anomaly Detection Evaluation. Harvard Dataverse, version 1.0, 2017. DOI 10.7910/DVN/6C3JR1
  3. Russell EL, Chiang LH, Braatz RD. Fault detection in industrial processes using canonical variate analysis and dynamic principal component analysis. Chemometrics and Intelligent Laboratory Systems 51(1), 81-93, 2000. DOI 10.1016/S0169-7439(00)00058-7
  4. Heo S, Lee JH. Statistical process monitoring of the Tennessee Eastman process using parallel autoassociative neural networks and a large dataset. Processes 7(7), 411, 2019. DOI 10.3390/pr7070411
  5. Kaufman S, Rosset S, Perlich C, Stitelman O. Leakage in data mining. Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data 6(4), 1-21, 2012. DOI 10.1145/2382577.2382579
  6. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4(9), 100804, 2023. DOI 10.1016/j.patter.2023.100804
  7. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences 115(11), 2600-2606, 2018. DOI 10.1073/pnas.1708274114
  8. Peng J, Zhao H, Hu Z. Second-order component analysis for fault detection. arXiv preprint 2103.07303, version 1, 2021.
  9. Lyu N, Botcha S, Kulkarni E, Pagaria S, Alves V, Sunshine EM, Kitchin JR. Benchmarking machine learning fault detection methods on the Tennessee Eastman process dataset. ChemRxiv preprint, version 1, 2026. DOI 10.26434/chemrxiv.10001628/v1
  10. Yin S, Ding SX, Haghani A, Hao H, Zhang P. A comparison study of basic data-driven fault diagnosis and process monitoring methods on the benchmark Tennessee Eastman process. Journal of Process Control 22(9), 1567-1581, 2012. DOI 10.1016/j.jprocont.2012.06.009

Related studies