Seven published C-MAPSS methods show no demonstrated RMSE gain over tuned gradient boosting in 28 comparisons

CheAI Research Inc. | Research article | May 2026

Abstract

1. Introduction

RUL prediction estimates the operating cycles a degrading component has left. Most RUL papers test on the simulated NASA C-MAPSS turbofan data [1], [2]. Published numbers are hard to compare, because papers differ in the RUL cap, the test-label convention and data preparation [2]. Of the ten papers selected here, four state or show tuning on the official test set, and six compare only with numbers copied from earlier papers. Test-set tuning is a form of leakage, which inflates reported skill in many fields [3].

We asked how much of the published RMSE gain over simple baselines remains under one leakage-free protocol, target and tuning budget, with a single test read. No re-run method was demonstrated better than a tuned gradient-boosting baseline in any of 28 method-subset comparisons.

2. Methods

Of ten papers from a targeted search, seven met the re-run criterion of open-licensed code or a full specification in the text [5]-[11].

In each of the four subsets, FD001 to FD004, 20 % of the training engines were held out for validation. Normalisation was fitted on training engines only, with per-regime z-scores on the six-condition subsets FD002 and FD004. All models used 14 sensors and the training target

y(t) = min(T - t, C), with C = 125 cycles,

where T is the failure cycle. The primary baseline B2 is a histogram gradient-boosting regressor [12] on 84 window features, the last value, mean, standard deviation, minimum, maximum and slope of each sensor. B1 is a ridge regression on the same features and B3 a similarity method. Each tuned model had a budget of 10 trials per subset, and RULCLIPPER enumerated its eight configurations. The selected configuration was trained with 10 seeds.

One guarded module opened the test files, after all 888 trained models were hashed and committed. The primary comparison was

Δ = mean RMSE_method - mean RMSE_B2,

with means over seeds and uncapped official labels. Intervals used paired resampling of test engines and independent resampling of each model's seeds, with 50,000 replicates, at the Bonferroni level 1 - 0.05/28 = 0.9982. A pre-registration froze four predictions. H1 stated that the median ratio of fair to published gain over B2 is below 0.5 on each subset, using papers with a published gain of at least 1 cycle. H3 stated that at least one method per subset is not demonstrated better than B2. An independent checker, without the production code, re-derived the metrics from the saved predictions and sealed its result first. Only results it reproduced are headline results.

3. Results

3.1 No method was demonstrated better than B2

B2 had the lowest uncapped RMSE of the five baselines on FD001, FD003 and FD004, at 13.68, 15.23 and 27.37 cycles. On FD002, B3 was lower, at 24.56 against 26.22 cycles. Of the 28 method-subset cells, 26 showed no demonstrated gain, 2 favoured the baseline and none showed a surviving gain (Figure 1). B2 was better for da Costa et al. and Laredo et al. on FD002, with intervals of 1.06 to 5.34 and 1.02 to 4.65 cycles. Five point estimates favoured the method, by 0.05 to 0.71 cycles, and every one of their intervals included zero.

The intervals do not establish equivalence. The Li et al. intervals on FD001 and FD004 ran from -1.66 to 14.33 and from -2.16 to 13.28 cycles. H3 was supported on all four subsets.

The checker agreed with every primary and hypothesis verdict. Of 6,090 result cells, 1,743 agreed and 22 disagreed, none in the headline. Another 1,104 compared values under different definitions, and 3,221 had no independent counterpart.

3.2 Most published values do not beat the fair baseline

Seventeen paper-subset cells have a published RMSE in cycles, compared here with B2 under each paper's label convention (Figure 2). In 10 of them, the published RMSE was higher than B2. Three more had a gain below the registered 1-cycle floor, at 0.42, 0.94 and less than 0.01 cycles. No subset had the three stable gains that the median needs, so H1 and the related subset comparison H2 were not assessable.

3.3 As-published reproductions are consistent with the papers

H4 predicted that the two papers flagged for test-set tuning [10], [11] would score worse once those steps were replaced by training-only equivalents. It was not supported. All five published values lay inside the 95 % intervals of the as-published reproductions (Figure 3). The intervals were 3.4 to 5.0 cycles wide and could contain a material change. The reproductions kept hyperparameters chosen on test, so this check does not isolate the effect of test-set tuning.

3.4 Results not verified independently

Four cells kept a published gain of at least 1 cycle, but the checker did not confirm their eligibility labels. In each, the fair-protocol gain was negative or near zero. Against B1, 12 of 28 secondary cells favoured the method at the unadjusted 95 % level, and against B3, 11 of 28. The secondary grid has 308 unadjusted tests, so some of these cells are expected by chance.

4. Discussion

Except for RULCLIPPER on FD003, at 7.72 cycles above B2, every method mean lay within 3.31 cycles of this tuned tree model. Re-runs in recommender systems found the same, with simple tuned baselines matching most new neural methods [4]. The larger finding is that 10 of 17 published values were not lower than B2 in the first place. Copied comparators cannot show whether a new model beats a simple model trained the same way.

The baseline choice matters. B2 had a mean RMSE 2.5 to 3.6 cycles below B1 with the same features. The label convention matters too. On FD002, the uncapped RMSE of B2 was nearly twice its capped RMSE, at 26.22 against 13.41 cycles. The Robinson values on FD002 and FD004 lie near our capped-label reproduction, but the convention that paper used is not established here.

A new RUL method on C-MAPSS should be compared with tuned gradient boosting on window features, trained on the same split and target, with several seeds and an interval on the paired difference.

5. Limitations

The main design strength is the single test read, after every model was frozen. The paper set is a convenience sample, and the results hold for these implementations on simulated C-MAPSS data only. They say nothing about real engines. NASA states no licence for the C-MAPSS data, so the redistribution rights are unclear, and the raw files are not redistributed. The code and results sit in a private repository.

All runs used local ports or re-implementations of point-prediction models, three of them from text. Laredo et al. ran without its unlicensed training wrapper, RULCLIPPER without its ensemble and Fu et al. as Dual-Mixer-F only. Robinson's conformal intervals were not run, and its label-free per-engine rescaling on early cycles was kept. Two as-published runs were not made, and three specifications came from preprints not compared with the versions of record. A failure to beat B2 therefore does not show that a paper is wrong, and the reproduction check does not isolate test-set tuning. Two outlying Li et al. seeds stay in every mean, since no non-convergence rule was registered.

The protocol also departed from its registered form in places. The compute cap was raised from 72 to 168 hours, and the registered reductions cut tuning to 10 trials, as-published runs to 5 seeds and the cap sweep to 1 seed. The report first omitted the four published-only papers, and adding their published numbers met the requirement to report every selected paper. The requirement of paired intervals across seeds is met under an interpretation fixed before the outcomes, but the literal wording is not met for the one-seed cap sweep. Timing tests stayed incomplete because a free-disk requirement was not met, and the registered one-command rebuild of the reports is absent.

Verification has gaps as well. The checker did not confirm the H1 eligibility count, the secondary grid, the seed intervals or the uncapped cap sweep. It compared intervals by verdict, not endpoints, and did not retrain the methods. Two B2 score values on FD002 differ from the checker's re-implementation for an unknown reason. Non-determinism of the training hardware was disclosed but not measured. Selected configurations are not reported, because their files were not saved with the results, and hashes cannot recover them. Regime sizes are not reported in a checked form.

6. Conclusions

A pre-registered, leakage-free re-run of seven published RUL methods on C-MAPSS found no method demonstrated better than a tuned gradient-boosting baseline in any of 28 comparisons. Most published RMSE values were not lower than that baseline, so the surviving share of published gains could not be assessed. All five published values for the two flagged papers lay within their reproduction intervals, a check that does not isolate test-set tuning.

References

1. Saxena A, Goebel K. Turbofan Engine Degradation Simulation Data Set. NASA Prognostics Data Repository, NASA Ames Research Center, 2008.
2. Vollert S, Theissler A. Challenges of machine learning-based RUL prognosis. A review on NASA's C-MAPSS data set. 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 1-8, 2021. DOI 10.1109/ETFA45728.2021.9613682
3. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4(9), 100804, 2023. DOI 10.1016/j.patter.2023.100804
4. Ferrari Dacrema M, Cremonesi P, Jannach D. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. Proceedings of the 13th ACM Conference on Recommender Systems, 101-109, 2019. DOI 10.1145/3298689.3347058
5. Robinson CD. Remaining useful life estimation for aircraft engines with risk-aware prediction intervals via conformalized quantile regression. International Journal of Prognostics and Health Management 17(1), 2026. DOI 10.36001/IJPHM.2026.v17i1.4724
6. Ramasso E. Investigating computational geometry for failure prognostics. International Journal of Prognostics and Health Management 5(1), 2014. DOI 10.36001/ijphm.2014.v5i1.2205
7. Liu Z, Zheng X, Xue A, Ge M. Multi-scale temporal-spatial feature-based hybrid deep neural network for remaining useful life prediction of aero-engine. ACS Omega 9, 47410-47427, 2024. DOI 10.1021/acsomega.4c03873
8. Laredo D, Chen Z, Schütze O, Sun JQ. A neural network-evolutionary computational framework for remaining useful life estimation of mechanical systems. Neural Networks 116, 178-187, 2019. DOI 10.1016/j.neunet.2019.04.016
9. Fu E, Hu Y, Peng K, Chu Y. Supervised contrastive learning based dual-mixer model for remaining useful life prediction. Reliability Engineering and System Safety 251, 110398, 2024. DOI 10.1016/j.ress.2024.110398
10. Li X, Ding Q, Sun JQ. Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering and System Safety 172, 1-11, 2018. DOI 10.1016/j.ress.2017.11.021
11. da Costa PRO, Akcay A, Zhang Y, Kaymak U. Attention and long short-term memory network for remaining useful lifetime predictions of turbofan engine degradation. International Journal of Prognostics and Health Management 10(4), 2019. DOI 10.36001/ijphm.2019.v10i4.2623
12. Friedman JH. Greedy function approximation. A gradient boosting machine. The Annals of Statistics 29(5), 1189-1232, 2001. DOI 10.1214/aos/1013203451

Related studies

• Tennessee Eastman fault detection, fair test. Published process-monitoring rates re-scored under a pre-registered protocol with held-out control limits