At the earliest deciding horizon, a nominal 90 % lower bound on early-predicted battery life covers only 17 of 40 held-out cells
We tested the lower bound of a pre-registered cycle-life model on a held-out batch of public cell data.
Abstract
Early cycle-life prediction is usually judged by point error, but qualification rests on a lower bound. We used public data from 124 lithium iron phosphate (LFP)/graphite cells in three batches, under a pre-registered plan. An elastic net trained on batches 1 and 2 was tested on the 40 cells of batch 3, with a one-sided lower bound from out-of-fold residuals at horizons of 25-300 cycles. The reproduction of published errors was partial. At cycle 100 the bound covered 38 of 40 held-out cells. Most cells were first decided against an 800-cycle criterion at cycle 125, but there the bound covered only 17 of 40 cells (0.425). No tested horizon met the pre-registered false-pass rule. Early-cycle bounds should be checked for coverage on an unseen batch at the horizon of use.
1. Introduction
Severson et al. predicted the life of fast-charged LFP/graphite cells from the first 100 cycles with a mean percentage error near 9 % [1], and Attia et al. re-analysed these data [3]. Accepting a production lot against an 800-cycle criterion needs a lower bound that holds with a stated probability. Split-conformal prediction gives such a bound when test and calibration cells are exchangeable [6], [7]. A shift between batches can break the guarantee [8].
We found no study that reports the coverage of such a bound on a held-out batch against the number of cycles used. We asked how early a nominal 90 % lower bound decides an 800-cycle criterion for most cells of a held-out batch, and whether it keeps its coverage there. The answer is cycle 125, but there the bound covered only 17 of the 40 held-out cells.
2. Methods
The primary data are 124 commercial LFP/graphite cells fast-charged with varied protocols [1], [2]. Batches 1, 2 and 3 hold 41, 43 and 40 cells. A second data set of 77 cells with other discharge protocols served a transfer test [4], [5]. Cycle life L was the first cycle below 0.88 Ah, or the last cycle plus one for cells the cycler stopped there.
At each of nine horizons N from 25 to 300 cycles, a model used only cycles 2 to N. The main model was an elastic net on the published discharge features, generalised to cycle N. It was trained on batches 1 and 2 (84 cells, fewer later as cells failed) and tested on batch 3. Gradient-boosted trees and a Gaussian process (GP) served for comparison.
The model predicts ŷ = log10 L, and the one-sided 90 % lower bound is LB = 10ŷ - q. The margin q is the ⌈0.9(n + 1)⌉-th smallest of the n negated out-of-fold residuals. These residuals come from the folds that selected the model but are applied to the refit on all training cells, so no finite-sample guarantee applies. Coverage was the fraction of held-out cells with L ≥ LB. A cell was decided-pass if LB ≥ 800. The earliest deciding horizon N* was the first N at which the Wilson 95 % lower end of the decided fraction exceeded 0.5. A decision was flagged unsafe when the 95 % upper end of the false-pass rate exceeded 0.10.
These settings and seven hypotheses were frozen before any model was evaluated. Separate code re-derived the labels, the elastic-net baseline and all coverage and decision arithmetic, but not the boosted-tree and GP searches.
3. Results
3.1 The published baseline was reproduced in part
The label rule reproduced the published cycle life of all 124 primary cells. On the published split at cycle 100, the variance model gave a root-mean-square error of 141.0 cycles against 138 published. The discharge model gave a mean absolute percentage error (MAPE) of 9.64 % against 8.6 %. The full-feature model missed two of its three targets (152.8 cycles against 100, 16.98 % against 10.7 %). The study proceeded under the pre-registered rule for a partial verdict.
3.2 Point error at cycle 100 matched the within-batch error
On batch 3 the MAPE was 23.8 % at N = 25, 9.73 % at N = 100 and 8.04 % at N = 300, a fall of 15.81 percentage points (95 % interval 12.22-19.59). At N = 100 the held-out MAPE of 9.73 % (7.01-12.68) was close to the within-batch 9.68 % (7.78-11.80), though the wide intervals do not show equal difficulty. The boosted trees gave 19.0 % (15.5-22.6) and the GP 15.8 % (12.3-19.6). At cycle 100 on this batch neither beat the elastic net, which supports no general ranking. On the second data set, with capacity-only features, the MAPE was 59.2 % (50.5-68.3), 6.09 times the held-out value.
3.3 Coverage fell below nominal after cycle 100
At N = 100 the bound covered 38 of 40 held-out cells (0.950, interval 0.831-0.994), classified as calibrated, although moderate under-coverage cannot be excluded. At N = 125 it covered 17 of 40 (0.425, 0.270-0.591). It was also under-covered at 150 (16 of 40) and 250 (20 of 40) (Figure 1a). At N = 125 most predictions lay above the true life, and 23 bounds did too (Figure 2).
3.4 The cycle-125 decision rests on bounds that fail for most cells
At N* = 125, 34 of 40 cells were decided-pass, with a Wilson lower end of 0.709 (Figure 1b). At N = 100 the decided fraction was 24 of 40, but its Wilson lower end was 0.446, so the point version of N*, based on the fraction itself, was 100. A two-sided reading, also using an upper bound, gave the same N*. Two of the 34 passes at cycle 125 were false, and the upper end of the false-pass interval was 0.197, so the decision was flagged unsafe. Two false passes in 34 cannot certify a rate below 0.10, nor show that it exceeds 0.10. At N = 200 the coverage of 34 of 40 was classified as calibrated, but 2 of 31 passes were false (upper end 0.214). No horizon met the false-pass rule.
The shortest life in batch 3 was 541 cycles, so no cell failed by cycle 300 and the rule made no fail decision.
4. Discussion
Point error alone would not have revealed the under-coverage, because at cycle 100 it matched the published within-batch error [1], [3]. The loss one horizon later can come from the bound construction, the batch shift or both, and our data cannot separate them. A shift is plausible, because batch 3 started eleven months after batch 1 and its cells were calendar-aged longer [1], [3]. A secondary check, neither pre-registered nor re-derived independently, found that the model over-predicted nearly every cell of batch 3 at cycles 125, 150 and 250, but found no pipeline defect. Split-conformal and CV+ bounds [6], [7] are untested on this batch, and weighting for covariate shift needs a model of the shift [8] that one batch without timestamps cannot supply. The transfer result agrees with the caution that laboratory lifetime models may not carry over to other conditions [9], but that test also changes the features, training cells and source.
For qualification, the decided fraction should be compared with the base rate. In batch 3, 34 of 40 cells lived at least 800 cycles, so passing every cell gives six false passes. The false-fail rate cannot be measured without fail decisions in the test batch.
5. Limitations
The main result rests on 40 cells of one batch, so one cell moves coverage by 0.025, and the intervals do not extend to unseen batches. The raw files hold no calendar timestamps, so rest-period effects could be neither tested nor excluded. Discharge features of the second data set were not computable on the frozen voltage grid, and its cell model is unconfirmed. Three data sets with unconfirmed reuse terms were excluded, so no chemistry shift was tested.
The bound is not split-conformal, although the pre-registration used that term, and carries no coverage guarantee. Two changes followed inspection of the data and preceded any fit. The 0.5 % stop margin of the label rule was chosen after seeing the stopping capacities, and we removed the constant-voltage hold because its original treatment is not published in the sources we held. A secondary run that kept the hold, not re-derived independently, moved the point version of N* from 100 to 50. The spline fit (tied voltages, grid ends), the solver limit, GP output normalisation, units and GP length scales at their frozen bound follow our own implementation readings. The hypotheses were not blind, because their author had seen the published cycle lives, and the outlier cell excluded from the baseline was identified by inference. The comparison hypothesis has two readings. A reading made before outcomes supports it, since a worse competitor does not falsify it, while its literal pass condition gives "inconclusive", since the boosted-tree interval lies wholly above that of the elastic net.
The reproduction of the published errors was partial. The inputs to the separate implementation were not sealed, and its independence rests on the recorded order of the computations; it refitted models only at cycle 100. The cause of the over-prediction at cycles 125, 150 and 250 is unresolved, and the check for it was not pre-registered.
Finally, the earliest deciding horizon of cycle 125 has a coverage of only 0.425. No claim is made about field cells or safety.
6. Conclusions
A pre-registered model first decided an 800-cycle criterion for most of 40 held-out cells at cycle 125, where its 90 % lower bound covered 17 of them. No horizon from 25 to 300 cycles met the false-pass certification rule. At cycle 100 point error matched published values and coverage was close to nominal, but not at the next horizons. Early-cycle bounds used for qualification should be tested for coverage on an unseen batch at the horizon of use.
References
- Severson KA, Attia PM, Jin N, et al. Data-driven prediction of battery cycle life before capacity degradation. Nature Energy 4(5), 383-391, 2019. DOI 10.1038/s41560-019-0356-8
- Toyota Research Institute. Battery cycling data for "Data-driven prediction of battery cycle life before capacity degradation". data.matr.io project 5c48dd2bc625d700019f3204, CC BY 4.0, 2019. https://data.matr.io/1/projects/5c48dd2bc625d700019f3204
- Attia PM, Severson KA, Witmer JD. Statistical learning for accurate and interpretable battery lifetime prediction. Journal of The Electrochemical Society 168(9), 090547, 2021. DOI 10.1149/1945-7111/ac2704
- Ma G, Xu S, Jiang B, et al. Real-time personalized health status prediction of lithium-ion batteries using deep transfer learning. Energy & Environmental Science 15(10), 4083-4094, 2022. DOI 10.1039/D2EE01676A
- Ma G. The dataset for "Real-time personalized health status prediction of lithium-ion batteries using deep transfer learning". Mendeley Data, version 2, CC BY 4.0, 2022. DOI 10.17632/nsc7hnsg4s.2
- Lei J, G'Sell M, Rinaldo A, Tibshirani RJ, Wasserman L. Distribution-free predictive inference for regression. Journal of the American Statistical Association 113(523), 1094-1111, 2018. DOI 10.1080/01621459.2017.1307116
- Barber RF, Candès EJ, Ramdas A, Tibshirani RJ. Predictive inference with the jackknife+. The Annals of Statistics 49(1), 486-507, 2021. DOI 10.1214/20-AOS1965
- Tibshirani RJ, Barber RF, Candès EJ, Ramdas A. Conformal prediction under covariate shift. arXiv preprint 1904.06019, 2019. https://arxiv.org/abs/1904.06019
- Sulzer V, Mohtat P, Aitio A, et al. The challenge and opportunity of battery lifetime prediction from field data. Joule 5(8), 1934-1955, 2021. DOI 10.1016/j.joule.2021.06.005