BMP test validation criteria according to Holliger et al. (2016) and their refinements

The publication by Holliger et al. (2016) summarises the work of an international group of researchers on harmonising BMP tests. It defines the mandatory elements of validation: the minimum number of replicates, the blank and the positive control (cellulose), the test duration criterion, the method of calculating BMP and the criteria for rejecting a test. In addition, it recommends, among other things, an I/S of at least 2 on a VS basis and full reporting of conditions. The numerical validation criteria were subsequently refined on the basis of two international inter-laboratory studies (Hafner et al. 2020) and incorporated into the updated guidelines (Holliger et al. 2021) — this current state is what we describe below. It is not a legal act. It is a point of reference for laboratories and clients.

Scope of the recommendations

Results meeting all the criteria can be described as "validated"; otherwise the test should be repeated. The current, openly available versions of the documents (including document 100, "Requirements for measurement and validation of BMP") are maintained by the group of authors on the DBFZ website (dbfz.de/en/BMP).

Termination criterion

The test ends when the daily methane production from a single bottle over three consecutive days is lower than 1.0% of the cumulative net methane increase from the substrate (substrate sample minus the mean of the blanks). Where measurements are taken less often than daily, the test can be ended after the first interval between measurements (at least 3 days in total) in which the rate has fallen below this threshold. In practice, for most substrates this means from over a dozen to several dozen days, and longer for lignocellulosic ones. Some laboratories adopt a fixed time, e.g. 28 or 30 days, but such a choice may understate the result for slowly degradable materials. The report states the actual duration and the final production rate.

Repeatability and rejection of results

The deviation between replicates is the basic indicator of quality; for cellulose the current criterion is an RSD of no more than 6%. If one bottle clearly stands out (a leak, a weighing error), it can be rejected, but only with a documented reason. Arbitrary removal of "inconvenient" results distorts the uncertainty. With a small number of replicates the uncertainty is poorly estimated.

Reporting

A correct report contains:

1. A description of the substrate: origin, preparation, TS, VS.
2. The inoculum: source, TS, VS, pH, degassing.
3. I/S and weighed portions.
4. Temperature, duration, gas measurement method and formulas.
5. Results of the blanks and of the cellulose control.
6. Substrate results: mean, deviation, number of replicates, cumulative curves.
7. Information on deviations from the protocol.

From 2016 to 2021: refinement of the criteria

After the workshop in Leysin (June 2015) and the publication of the guidelines in 2016, two international inter-laboratory studies were carried out, and in 2018 another workshop was held (Freising). Hafner et al. (2020) analysed more than 400 BMP values from more than thirty laboratories in fourteen countries. The relative standard deviation between laboratories was 7.5–24%. After the validation criteria were applied, it fell below 8% for all four substrates studied. The recommended validation criteria are: test duration (end when the daily production over 3 consecutive days is lower than 1% of the cumulative net increase), cellulose RSD of no more than 6% and mean cellulose BMP in the range of 340–395 NmL CH₄/g VS. The authors pointed out that the criterion for cellulose proved particularly important. The updated guidelines were presented in the commentary by Holliger et al. (2021, online 2020) and are maintained as an open document, "Requirements for measurement and validation of BMP".

A separate matter is the way methane production is calculated from measurements (dry gas, standard conditions, headspace volume) — it is described in the documents of the 200 series of the same collection of methods; the work of Hafner et al. (2020) indicated that errors in data processing and entry were among the significant sources of scatter.

Example of how a client can use the criteria

You commission a silage test and receive a result of 340 NmL/g VS. Check in the report: whether I/S is given (≥ 2 recommended), whether the mean cellulose BMP falls within 340–395 NmL/g VS with an RSD ≤ 6%, the number of replicates (at least 3) and the termination criterion. The absence of any one element does not automatically invalidate the result, but it reduces its usefulness — ask for it to be supplied.

Limitations

• The recommendations apply to batch BMP tests; they do not cover continuous tests.
• Some substrates (fats, strongly inhibitory materials) require modification of the protocol.
• Compliance with the guidelines does not eliminate scatter resulting from sample heterogeneity.

Guidance for laboratories

1. Introduce an internal procedure based on the current guidelines.
2. Take part in inter-laboratory studies, if available.
3. Archive control charts of cellulose results.
4. Update procedures after reading newer papers.

How to use the guidelines in a contract with a laboratory

When ordering BMP tests, specify in the order: the protocol (e.g. in line with the recommendations of Holliger et al. 2016 in the version updated in 2021), the minimum number of replicates, the required I/S, the termination criterion, the requirement for a cellulose control and the report format with raw data. This later makes it possible to compare results from different series and laboratories and to avoid disputes over interpretation.

What has changed since 2016

The number of inter-laboratory studies is growing, as is work on calculations and on tests with automated measurement systems. Nevertheless, the 2016 guidelines remain the most frequently cited point of reference. The recommendations are updated on the basis of new inter-laboratory studies, so it is worth checking the current version of the documents and stating in reports which version the test was carried out to.

Example of a checklist for accepting a report

• [ ] The guidelines according to which the test was carried out are indicated.
• [ ] The number of replicates and the deviation are given.
• [ ] The cellulose and blank results are presented.
• [ ] I/S and temperature are given.
• [ ] The termination criterion and duration are explained.
• [ ] The reference conditions for the gas volume are indicated.
• [ ] Cumulative curves are attached.