A 501(c)(3) non-profit organization Applied AI research for public benefit
CheAI Research Inc.

Four management-system elements are the most frequent reliably coded causes in 119 CSB investigations, and recommendations to government take a median 876 days longer to close

We coded 119 US Chemical Safety Board investigations, obtained a blind second coding for 40, and analysed closure times for 1,061 recommendations.

Abstract

Cause counts across investigations of the US Chemical Safety and Hazard Investigation Board (CSB) need evidence of consistent coding. We coded 119 completed CSB investigations against the 20 elements of Risk Based Process Safety (RBP) and a fixed list of safeguards, under a schema frozen before any report was read. A second coder coded 40 investigations blind, and only values with a kappa of at least 0.61 and a lower 95 % bound of at least 0.41 entered the headline frequencies. Of 68 agreement assessments, 24 passed. The most often coded element, hazard identification and risk analysis, failed. Among the passing elements, training and performance assurance (61 %), compliance with standards (59 %), operating procedures (48 %) and management of change (45 %) recurred most. For 1,061 recommendations, the median time to any closure was 1,845 days for government recipients and 969 days for industry, a difference of 876 days (95 % interval 593-1,223). An independent recomputation reproduced all 206 headline statistics exactly.

1. Introduction

The CSB investigates serious chemical incidents and issues recommendations to prevent a recurrence [1]. Baybutt coded 68 completed investigations and found non-routine work, safeguards, siting and procedures among the most frequent issues [2]. Kaszniak drew recurring oversights in process hazard analyses from CSB investigations [3]. The CSB's own count found 75 % of 733 recommendations closed by 2015 [4].

The reliability of cause coding is rarely measured, and where it was measured for an accident taxonomy, agreement was low [5]. Closure is usually a share at one date, which mixes the speed of closure with the age of the recommendations [6]. We asked which causes recur when only consistently coded causes count, and how long recommendations take to close. Four management-system elements recur most, and recommendations to government close years later than those to industry.

2. Methods

All data came from the public CSB website [1, 7], accessed on 2026-10-17. Of 134 indexed investigations, 130 had a primary document. Eleven had unreadable files, which left 119 coded investigations. Causes were coded against the 20 RBP elements [8]. Safeguards took one of 12 labels, with the status "failed" (in place but did not work) or "absent" (not provided). Every coded value carries a cited page and a quote that a script verified against that page. Company and facility names were anonymised. A pre-registration frozen on 2026-10-18 fixed the schema, the agreement sample and threshold, every statistic and four predictions.

Both coders were language models without network access. The first coded all 119 investigations. The second coded a stratified random sample of 40 without seeing the first coder's output. Cohen's kappa [9] was computed for each value, with a 95 % stratified bootstrap interval. A value passed when its kappa was at least 0.61 and its lower bound at least 0.41. A value that either coder recorded in fewer than 5 of the 40 investigations was too rare to judge.

The status export, dated 2026-09-16, lists 1,061 recommendations with recipient type, status and dates. Recipients were grouped as industry, government, standards bodies and other. The headline statistic was the Kaplan-Meier median time to any closed status [10], with open recommendations censored at the export date. The government-minus-industry difference used a bootstrap over whole investigations (108 clusters). A second analyst re-implemented the statistics from the raw files, and a headline value had to match exactly.

3. Results

3.1 The most frequent cause was not coded reliably

Of the 68 assessments, 24 passed, 24 failed and 20 were too rare to judge (Figure 1). Nine of the 12 safeguard labels and nine of the 20 RBP elements passed. No value of the equipment class, the initiating event or the RBP element linked to a safeguard passed. Hazard identification and risk analysis had a kappa of 0.77 but a lower bound of 0.36. Asset integrity and reliability had a kappa of 0.35. Both coders cited these two elements in most sampled investigations, and at such high prevalence a few disagreements move kappa a long way [11].

Kappa and intervals for each coded value
Figure 1. Cohen's kappa for each coded value with a kappa verdict in the 40 double-coded investigations, with 95 % bootstrap intervals. The lines mark the thresholds 0.61 (point) and 0.41 (lower bound).

3.2 Four management-system elements recur most

Among the passing elements, training and performance assurance was cited in 73 of 119 investigations (61 %, 95 % interval 52-70 %). Compliance with standards followed with 70 (59 %, 50-68 %), operating procedures with 57 (48 %, 39-57 %) and management of change with 54 (45 %, 36-55 %). The intervals overlap, so the order of these four elements is not resolved.

Safeguard frequencies differed between failed and absent states (Figure 2). Alarms or detection were absent in 46 investigations (39 %) and failed in 19 (16 %). Training or competence was absent in 48 (40 %) and failed in 11 (9 %). Emergency response showed the opposite pattern, with 31 failed (26 %) and 22 absent (18 %).

Shares of passing causes and safeguards
Figure 2. Passing causes (a) and safeguards failed (filled) or absent (open) (b) in 119 coded CSB investigations, with 95 % bootstrap intervals. An investigation can hold the same safeguard in both states. SIF, safety instrumented function.

3.3 Recommendations to government close slowly

Of the 1,061 recommendations, 932 were closed and 129 (12 %) were open. The median time to any closed status was 1,161 days (95 % interval 1,108-1,269), about 3.2 years. It was 969 days (895-1,044) for industry and 1,845 days (1,709-2,001) for government (Figure 3). The difference of 876 days had a cluster bootstrap interval of 593-1,223 days. By decade of issue, the medians ranged from 974 to 1,350 days, and the four intervals overlap. The 1990s median rests on 31 recommendations from only three investigations, and its interval ends at the point value.

Median days to closure by group and decade
Figure 3. Kaplan-Meier median time to any closure of 1,061 CSB recommendations, by recipient group (a, intervals not adjusted for clustering) and decade of issue (b, cluster bootstrap intervals).

3.4 Predictions and secondary results

All four pre-registered predictions held, but only the prediction of slower closure for government rests on headline values. The cause-ranking prediction held through hazard identification and asset integrity, which both failed the agreement threshold. The following values were not independently compared. Of all recommendations, 680 (64 %) were closed with acceptable action, 71 % for industry and 52 % for government. Five years after issue, the cumulative incidence of acceptable closure was 0.63 for industry and 0.34 for government, with other closures as competing events [6].

4. Discussion

In about two of every five investigations, the report states that alarms or detection, or training, were not provided at all. The pattern fits the reading that many sites lacked detection or training for the scenario, while their emergency response existed but did not work. The study did not test that reading.

A ranking from one coder can be led by elements that two coders do not apply in the same way. Operating procedures recur here in 48 % of investigations, in line with Baybutt [2]. Our data can neither confirm nor reject the hazard-analysis finding of Kaszniak [3], because that element failed the threshold.

The government median is about 2.4 years longer than the industry median, an unadjusted descriptive association. Rule changes may explain it, but recommendation content was not coded. The CSB reported 75 % of 733 recommendations closed by 2015 [4]. The 88 % closed here concerns a different date and set, so the two are not a trend. Recommendations from the 2020s had only 38 % closed with acceptable action because half were still open, yet their median time to closure was not longer.

5. Limitations

The data limit what the frequencies can describe. The CSB investigates selected incidents, so the frequencies describe investigated incidents, not industry as a whole. The export starts in 1998, and 36 of the 134 investigations could not be linked to it. The 11 uncoded investigations may not be a random subset, and one investigation was coded from only one of its two primary documents. CSB reuse rights are unresolved, so this report quotes no CSB report text. A planned machine-readable copy of the coded lessons table was not produced.

Coding reliability is the main methodological constraint. Agreement was mixed, with 24 of 68 assessments passing. The pre-registration allowed one schema revision, but the schema was not revised. Failing and too-rare values are reported as unreliable and kept out of the headline. Agreement rests on 40 double-coded investigations, and its intervals left several pass or fail decisions uncertain. No human coded the sample, so the agreement measures the consistency of two automated coders, not their accuracy against expert coding. The first coder's inference settings were not recorded. The safeguard status kappa is pooled over safeguards matched by label, which can pair different physical safeguards. Unmatched safeguards, 24 of the first coder and 73 of the second, were left out, and the failed and absent shares of each label have no reliability estimate of their own.

The closure statistics carry their own assumptions. Single-median intervals are not adjusted for clustering, and group comparisons are not adjusted for content or year. The Kaplan-Meier medians assume that censoring is unrelated to closure time. The 2020s recommendations have short follow-up, so their median depends strongly on that assumption, and the 1990s median rests on three investigations.

On verification and scope, the headline is restricted to 44 frequency rows (176 statistics) that passed the agreement threshold and 30 recommendation day statistics. The recomputation reproduced all 206 exactly, and every other value is secondary. The sensitivity check that drops the multi-incident studies is withheld. Its recomputation is a disclosed deviation from the pre-registration, because its inputs could not be supplied under the data-handling rules. No finding of fault beyond what the CSB published is made about any company.

6. Conclusions

Among causes that met the coding-agreement threshold, training and performance assurance, compliance with standards, operating procedures and management of change recur most, and absent detection and absent training are the most frequent safeguard gaps. Hazard identification and risk analysis and asset integrity, which lead the unchecked ranking, could not be coded reliably under the definitions used. Recommendations to government take much longer to close than those to industry. Analyses of investigation reports should publish the agreement statistic of each cause, and closure studies should use time-to-event methods that account for open recommendations and competing closures.

References

  1. U.S. Chemical Safety and Hazard Investigation Board. Completed Investigations. Data acquired 17 October 2026. https://www.csb.gov/investigations/completed-investigations/
  2. Baybutt P. Insights into process safety incidents from an analysis of CSB investigations. Journal of Loss Prevention in the Process Industries 43 (2016) 537-548. https://doi.org/10.1016/j.jlp.2016.07.002
  3. Kaszniak M. Oversights and omissions in process hazard analyses. Lessons learned from CSB investigations. Process Safety Progress 29 (2010) 264-269. https://doi.org/10.1002/prs.10373
  4. Tinney VA, Anenberg SC, Kaszniak M, Robinson B. Eighteen years of recommendations to prevent industrial chemical incidents. Results and lessons learned of the US Chemical Safety Board. Public Health 139 (2016) 183-188. https://doi.org/10.1016/j.puhe.2016.04.011
  5. Olsen NS, Shorrock ST. Evaluation of the HFACS-ADF safety classification system. Inter-coder consensus and intra-coder consistency. Accident Analysis and Prevention 42 (2010) 437-444. https://doi.org/10.1016/j.aap.2009.09.005
  6. Putter H, Fiocco M, Geskus RB. Tutorial in biostatistics. Competing risks and multi-state models. Statistics in Medicine 26 (2007) 2389-2430. https://doi.org/10.1002/sim.2712
  7. U.S. Chemical Safety and Hazard Investigation Board. Recommendations. Data acquired 17 October 2026. https://www.csb.gov/recommendations/
  8. Center for Chemical Process Safety. Guidelines for Risk Based Process Safety. Wiley (2007). https://doi.org/10.1002/9780470925119
  9. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1960) 37-46. https://doi.org/10.1177/001316446002000104
  10. Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. Journal of the American Statistical Association 53 (1958) 457-481. https://doi.org/10.1080/01621459.1958.10501452
  11. Feinstein AR, Cicchetti DV. High agreement but low kappa. I. The problems of two paradoxes. Journal of Clinical Epidemiology 43 (1990) 543-549. https://doi.org/10.1016/0895-4356(90)90158-L