A 501(c)(3) non-profit organization Applied AI research for public benefit
CheAI Research Inc.

Six in ten testable ThermoML vapour-liquid equilibrium data sets fail each of two consistency tests

We applied two published consistency tests to every testable binary vapour-liquid equilibrium data set in a NIST archive snapshot.

CheAI Research Inc. | Research article | December 2025

Abstract

Gibbs-Duhem consistency tests are usually applied one paper at a time, so the share of published vapour-liquid equilibrium (VLE) data that fails them is not known. We tested every complete binary isothermal and isobaric data set in the ThermoML archive snapshot of 30 September 2020. Rules and criteria were frozen before any outcome, and activity coefficients followed modified Raoult's law. Of 4,115 candidate sets, 1,233 could be tested with the Herington-type area test and 1,948 with the Van Ness test. The area test failed 738 sets (0.599, 95 % interval 0.558-0.637), and the Van Ness test failed 1,195 (0.613, 0.579-0.648). On a random sample of 200 sets, an independent re-implementation agreed on 48 of 53 tested area-test verdicts and 91 of 91 Van Ness verdicts. The results come from a corrected re-run decided after first outcomes had been seen.

1. Introduction

Activity-coefficient models fitted to binary VLE data size distillation, absorption and extraction equipment. The Gibbs-Duhem equation links the activity coefficients of the two components, so complete temperature, pressure and composition data are redundant. Consistency tests use this to find sets whose measured quantities disagree [1], [2]. Two such tests are the area test [3], [4] and the Van Ness test [5].

Published VLE data from five journals are collected in the ThermoML archive of the NIST Thermodynamics Research Center (TRC) [6], [7]. To our knowledge no open, archive-wide count of failures exists. We asked what fraction of the testable binary sets in the archive fail each of two published tests. About six in ten failed each test.

2. Methods

The snapshot of 30 September 2020, maintained by NIST TRC, holds entries published to the end of 2019. A data set is one series of complete points from one article, with two components and one class, isothermal or isobaric. Activity coefficients followed modified Raoult's law, without vapour fugacity or Poynting corrections,

γi = yip / (xipisat(T)),   i = 1, 2,

where xi and yi are the liquid and vapour mole fractions, p the pressure and pisat is the vapour pressure at temperature T. Vapour pressures came from the same article, else from the ChemSep databank [8] within its temperature range. ChemSep failed a frozen boiling-point check, so 46 outlying records were excluded.

The area test (T1) compares the areas A and B above and below zero of ln(γ1/γ2) against composition through D = 100|(A - B)/(A + B)|. For isobaric sets J = 150|ΔTmax/Tmin| adds the boiling range. A set passed with D < 5 (isothermal) or |D - J| < 10 (isobaric), as quoted by Kang et al. [4], without their extra pass rule for very small areas. The Van Ness test (T2) fitted a five-parameter non-random two-liquid (NRTL) model to measured pressures only. A set passed when the mean absolute deviations of pressure, in percent, and of vapour mole fraction, times 100, were both below 1. The failure fraction is FAIL/(PASS + FAIL). Its 95 % interval came from 10,000 bootstrap replicates that resampled whole articles.

After the first run, we found that compositions tagged to the reference component had been stored as 1 - v, which affected 2,001 sets. We corrected the defect and re-ran all tests with the same frozen rules. A second implementation of T1 and T2, written without access to the primary code, computed verdicts on 200 randomly drawn sets.

3. Results

3.1 Fewer than half of the candidate sets can be tested

The archive gave 4,115 candidate sets from 874 articles. T1 could be applied to 1,233 sets from 448 articles and T2 to 1,948 sets from 592 articles (Figure 1). For both tests, 682 sets had a supercritical component and 1,001 more had no usable vapour pressure. T1 lost 874 more sets, because it needs liquid compositions within 0.10 of both pure components.

Stacked outcome bars for T1 and T2.
Figure 1. Outcome of every candidate set for the area test (T1) and the Van Ness test (T2). Black is FAIL, white is PASS and grey shades are the reasons a set was not testable. Coverage and span are merged as composition range, and one composition-gap set per test is merged with too few points. The single T2 computation failure is too narrow to show.

3.2 Six in ten testable sets fail each test

T1 failed 738 of 1,233 testable sets, 0.599 (95 % interval 0.558-0.637, Figure 2). T2 failed 1,195 of 1,948 sets, 0.613 (0.579-0.648). T1 failed 426 of 575 isothermal sets (0.741) and 312 of 658 isobaric sets (0.474). The classes use different criteria and sets, which limits a direct comparison.

Failure fractions of T1 and T2 with intervals.
Figure 2. Failure fractions of T1 and T2 with 95 % article-cluster bootstrap intervals. Numbers at the right are FAIL and testable counts.

3.3 An independent implementation agreed on most sampled verdicts

The two implementations gave the same verdict category for 195 of 200 T1 sets and all 200 T2 sets, above the frozen threshold of 189. Most were sets neither side could test. Among the sets that both sides tested, T1 agreed on 48 of 53 and T2 on all 91. All five disagreements were isobaric sets that the independent side failed. They are unresolved, because the re-implementation was not re-run. The statistics D and the two T2 deviations agreed on every tested set, but |D - J| agreed on only 9 of 33 isobaric sets. Separate recomputations reproduced the archive-wide counts, fractions and T1, T2 and class-split intervals.

3.4 Secondary results, not checked independently

Our implementation of the combined quality factor of Kang et al. [4] failed 18 of 1,947 sets (0.009), while its infinite-dilution component failed 0.928. It used ChemSep vapour pressures and was not compared with ThermoData Engine scores. T1 and T2 agreed on 0.600 of the 1,228 sets that both tested (Cohen's kappa 0.181), only slightly better than chance. T1 failed 0.510 of sets up to 200 kPa and 148 of 150 sets above 1 MPa, and T2 rose from 0.506 to 0.892. The three large journals differed by at most 0.016 for T1 and 0.079 for T2. Contrasts outside the registered predictions are exploratory, and no multiplicity correction was applied. Of twelve registered predictions, five held, five were inconclusive and two were refuted. Because the run was corrected after first outcomes, outcomes that changed do not confirm the predictions.

4. Discussion

A high failure fraction does not mean that six in ten published measurements are wrong. Each test compares the data with a model. The recount and the second implementation check the verdicts, but neither shows that the published criteria and the model assumptions suit the data. Modified Raoult's law ignores vapour non-ideality, an error that grows with pressure. The rise in failure above 1 MPa is consistent with this error but does not isolate it, since high-pressure sets also differ in chemistry. A vapour-pressure mismatch offsets γ systematically [9]. ChemSep-only vapour pressures raised the T1 and T2 fractions by 0.016 and 0.033 where both sources existed.

The Van Ness verdict depends on the fit objective, which Kang et al. leave unstated. In one post-freeze check on an ethanol + water set, our pressure-only fit gave a vapour-composition deviation of 1.18 and a fail. Kang et al. report 0.67 and a pass, and a fit to pressure and vapour composition gave 0.69 and a pass. The effect on the archive-wide fraction was not computed, so 0.613 refers to the pressure-only fit.

The choice of test changes the answer more than the choice of journal. Earlier comparisons on selected data also found frequent disagreement [2], [10], and the low kappa shows this at archive scale.

5. Limitations

The fractions come from a corrected re-run decided after first-run outcomes had been seen, so they are a post-outcome correction. Four deviations from the pre-registration were disclosed, one of which was to continue after the ChemSep check failed. No test reproduced a worked example from its source paper, and none of five planned hand checks was completed. The one post-freeze check reproduced 2 of 4 targets. Herington's 1951 paper was not held, so the isobaric T1 criterion is quoted second-hand from Kang et al. [4]. The independent re-implementation covered only T1 and T2 verdicts on 200 sets. Isobaric |D - J| values differed on 24 of 33 sets, which were not diagnosed one by one, and Section 3.4 rests on the primary implementation alone.

On the method side, four choices can each fail a consistent set. They are modified Raoult's law, the vapour-pressure sources, the pressure-only T2 fit and our readings of misprinted equations. The Herington test was designed for isobaric data [9]. The isothermal criterion D < 5 is a Redlich-Kister-type area test [11], and for near-ideal mixtures [4], D is a ratio of two small areas.

The data set its own limits. The snapshot gives no author-stated uncertainty for any of the 4,115 candidate sets. No verdict accounts for it, and the planned Monte Carlo propagation was not run. A third test, which needs heats of vaporization, was not run either. A verdict refers to a set as encoded in the archive and joined by our rules, so an encoding error cannot be told apart from an experimental error. In 3 articles, 8 sets with conflicting component tags were not tested.

The recorded archive terms permit local analysis and derived statistics with acknowledgement and a change notice. Whether raw values may be redistributed is unclear, and none are reproduced here. NIST, TRC and the publishers have not endorsed these results. Method papers read under library access were used for citation and checking only, and none of their material is reproduced.

6. Conclusions

Under published criteria and modified Raoult's law, about six in ten testable binary VLE sets in the ThermoML archive failed the Herington-type area test, and about as many failed the Van Ness test. In a sample, a second implementation agreed with 48 of 53 area-test and all 91 Van Ness verdicts. Fewer than half of the candidates could be tested. The secondary results indicate that the answer depends strongly on the test and on pressure. Anyone fitting a model to archived VLE data should test each set and report the test, criterion and model used.

References

  1. Van Ness HC. Thermodynamics in the treatment of vapor/liquid equilibrium (VLE) data. Pure and Applied Chemistry 67, 859-872, 1995. DOI 10.1351/pac199567060859
  2. Wisniak J, Apelblat A, Segura H. An assessment of thermodynamic consistency tests for vapor-liquid equilibrium data. Physics and Chemistry of Liquids 35, 1-58, 1997. DOI 10.1080/00319109708030571
  3. Herington EFG. A thermodynamic test for the internal consistency of experimental data on volatility ratios. Nature 160, 610-611, 1947. DOI 10.1038/160610b0
  4. Kang JW, Diky V, Chirico RD, Magee JW, Muzny CD, Abdulagatov I, Kazakov AF, Frenkel M. Quality assessment algorithm for vapor-liquid equilibrium data. Journal of Chemical and Engineering Data 55, 3631-3640, 2010. DOI 10.1021/je1002169
  5. Van Ness HC, Byer SM, Gibbs RE. Vapor-liquid equilibrium. Part I. An appraisal of data reduction methods. AIChE Journal 19, 238-244, 1973. DOI 10.1002/aic.690190206
  6. Riccardi D, Trautt Z, Bazyleva A, Paulechka E, Diky V, Magee JW, Kazakov AF, Townsend SA, Muzny CD. Towards improved FAIRness of the ThermoML Archive. Journal of Computational Chemistry 43, 879-887, 2022. DOI 10.1002/jcc.26842
  7. NIST Thermodynamics Research Center. ThermoML Archive, snapshot ThermoML.v2020-09-30. National Institute of Standards and Technology, data set, 2021. DOI 10.18434/mds2-2422
  8. Kooijman HA, Taylor R. ChemSep pure component databanks, versions 8.3 and 8.31, as distributed with DWSIM. 2022. www.chemsep.org
  9. Zbytovský J, Sommer T, Zapletal M, Trejbal J. A new method for testing thermodynamic consistency of vapor-liquid equilibrium data. ACS Omega 10, 46809-46821, 2025. DOI 10.1021/acsomega.5c04650
  10. Wisniak J, Ortega J, Fernández L. A fresh look at the thermodynamic consistency of vapour-liquid equilibria data. The Journal of Chemical Thermodynamics 105, 385-395, 2017. DOI 10.1016/j.jct.2016.10.038
  11. Redlich O, Kister AT. Algebraic representation of thermodynamic properties and the classification of solutions. Industrial and Engineering Chemistry 40, 345-348, 1948. DOI 10.1021/ie50458a036