Establishing Analytical Performance Value of Laboratory Instrument Methodology Made Easy

A need for the simplified review of method performance capability based on precision statement is required for commercial analytical instruments. A simplified applied statistical approach would be benefitial and assist laboratory managers, process engineers, commercial traders, lab chemists, lab technicians and operators. The numerical expression rating system based on performance relative to two or more points within the operating value of each parameter defined in any standard method containing full precision defined as repeatability and reproducibility. A process and computational expression is described and defined as analytical performance value.

The energy and chemicals industries would benefit from a reliable and straightforward statistically based system which allows for determining the value-added performance of any test method. This performance value would be contained in any standard method for use as a quantitative evaluation of test method performance. A straightforward approach which provides users with a performance based reference table for comparison and selecting the appropriate test methods.

The proposed technique is to define a systematic method which provides for establishing an analytical performance value (APV) based on precision criteria relative to parameter measurements. Standard method precision criteria are used to establish an estimated APV value based on repeatability and reproducibility at the lowest and highest operating concentration range. Expressed in a different way, an APV values are quantitative instrument performance criteria for users who may not possess extensive statistical technical knowledge and need to express and describe statistical precision to those that users that do not possess statistical knowledge. 

Industry precedent has been established and incorporated into corporate finance defined as economic value added (EVA¹)²or return on assets (ROA). ROA is defined as the ratio of earnings to total assets. This could lead to misleading results as the percentage expressed does not factor in the total capital brought in after an investment. The biggest difference between the two systems is that EVA is expressed in a monetary value. While the ROA may decrease in percent value for a larger investment, EVA may increase due to a larger return on the initial investment. Thus, investments made using an EVA system can be worthwhile and beneficial while an ROA determines them to be harmful. The equation for EVA is based on the difference between the ROA and average cost of capital multiplied by the total capital. Simply put, it is the net earnings after an initial investment which very clearly determines whether it is an endeavor worth pursuing.

In a similar way to EVA, the analytical performance value (APV) are expressed in equation 1, 2, below where repeatability and reproducibility are expressed relative to a known value.

Eq. 1 – Analytical Performance Value

APV = estimated repeatability value / minimum detection limit * 100

Eq. 2 - Scalable APV Assigned for Each Method Parameter

APV₁ = assigned when APV = X or< 5%

APV₂ = assigned when APV = X or> 5% or < 10%

APV₃ = assigned when APV = X or> 10% or 15%

APV₄ = assigned when APV = X or > 15%

X = a value established by industry experts or commercial production and trade requirements.

Many laboratory decision makers, laboratory managers, and other laboratory personnel express a similar desire to access simplified test method precision criteria. The following specific examples demonstrate the proposed scheme:

• means to quickly apply test method performance based on established parameter value in units of measure;

• individual parameters and units of measure are evaluated based on the repeatability or reproducibility;

• comparison of either individual laboratory or intralaboratory data is often performed daily;

• time required to perform each calculation can be extensive when all of events are taken into consideration.

The simplified scheme proposal is to apply equations 1 and 2 (or one obtained by consensus) to both method repeatability and reproducibility at the defined minimum and maximum operating limit values defined within the scope of the standard test method. These equations provide a simplified pre-calculated degree of variation relative to the specific points of the method operating window. It also provides a strategy for assessing a test method’s acceptability based on its specifications and corresponding parameters with sufficient confidence. This evaluation can justify the precision for between-laboratory testing in order to standardise the method’s performed in an efficient way.

A simplified scheme to provide a quantitative APV expressed as a scaled value percentage which defines the degree of variation based on a point within the operating range of any instrument is useful within the commercial trade industry. A standard method should contain an APV section which would contain the tabulated values for each parameter estimated by calculating repeatability at (for example) the minimum operating value (i.e. limit of detection) defined in the scope of any analytical method. The APV table provides repeatability values within the operating range minimum to maximum value defined in the scope of any standard defined as APVr. Reproducibility can be estimated using Eq.1 at the same operating range to establish APVR. Once the APV values have been properly evaluated, the acceptability or capability can be determined.  An APV of <10% is defined as an acceptable and capable test method. From 10 – 30%, the test method is considered marginally acceptable. Anything with an APV value of >30% represents a challenge or issue within the test method, thus becoming unreliable and must be reviewed. Additionally, test methods that report results in a temperature, APV values should be set equal to their exact repeatability and reproducibility respectively due to its qualitative versus quantitative nature. 

An example of this concept, the first equation (Eq. 1) has been applied to test method ASTM D7423-16^e1, as shown in Table 1, Table 2, and Table 3. In the example provided, a rating tolerance value is provided in Table 2, whilst Table 3 is based on industry typical values. The rating tolerance applied to methods would be defined either by industry production operating specification requirements, governing bodies and initial technology prime tolerances. Initial tolerance prime tolerance values are defined as examples of cases where disruptive technology is initially introduced into commercial applications and industry standardised values are not available.

The APV technique can be used to compare any standard developed for testing the same parameter. An example of the ease of use is outlined in Tables 4 and Table 5 where APV values are tabulated for the same parameters obtained by ASTM D3606-10 and ASTM D5769-15 at the minimum operating concentration specified in the scope of each standard, accordingly. The minimum APV (L-APVr1) data comparison eliminates the calculation steps required and allows for direct comparison of both standards. In this case, the L-APVr1 data for the compound benzene at similar concentrations show test method ASTM D5769-15 has better performance when compared to ASTM D3606-10.

Comparison of Table 6 and Table 7 provides a quantitative comparison of the same parameters for ASTM D3606-10 versus ASTM D5769-15 at the maximum operating concentration defined in the scope of each standard. The maximum APV (H-APVR1) data shows that the use of ASTM D5769-15 for testing benzene at 4.0 volume percent would provide better performance.

Along with APV calculations, the precision ratio (PR) should be considered to determine whether a test method can be standardised or not. The PR is simply the ratio between reproducibility and repeatability respectively. Generally, a PR value >4 indicates a significant difference between reproducibility and repeatability such that between-laboratory bias is the dominant contributor to the reproducibility which implies further work on the test method is to be done. However, it is important to note that PR is considered an associative calculation where one cannot solely rely on its results. If the PR for any given test is close to 1 but the APVr is large, that test cannot be standardised despite the insignificance of between-laboratory common causes to reproducibility and in-laboratory repeatability.

In summary, the analytical performance value technique as presented establishes an effective process which eliminates wasted effort and provides reliable and accurate means for evaluating the performance of any standard. The effectiveness improves with increasing complexity for standards which contain multiple test parameters. The analytical performance value concept has been presented to ASTM Committee D02 Coordinating Subcommittee 94 on Quality Assurance and Statistics. The analytical performance value has been incorporated into a proposed standard guide – “Evaluating Test Method Capability and Fitness for Use”. 

Mr. George Gonzalez serves as the Global segment marketing manager for energy and chemicals with Agilent Technologies, Wilmington, DE, and Dr. Raj Shah serves as a Director at Koehler Instrument Company, Holtsville, NY. The authors wish to thank the support of all the ASTM members who have been active in Subcommitee 94.