5.7 A Note On Numerics

It is difficult to ensure the numeric stability of all possible calculations made by any software that uses floating-point arithmetic. To guard against the loss of precision during ANOVA computation, Xynk assumes that

1. your data is well-conditioned (i.e., that the magnitude of individual data points are similar),

2. neither extremely small nor extremely large values are included in the data set, and

3. the total number of data points is not too great.

Xynk takes the following precautions to avoid the most common errors in floating-point arithmetic.

• double-precision (64-bit) floating-point numbers are used for all statistical calculations
• the IEEE 784 floating point exceptions are monitored during calculations, and errors are reported (i.e. FE_INVALID, FE_DIVBYZERO, FE_OVERFLOW, FE_UNDERFLOW, and FE_INEXACT).
• Summations, e.g. when calculating means or variance, are performed in two-stages and with Kahan compensated summation (to minimize rounding errors caused by adding small terms to large sums).

If you require higher precision or have a very large number of data points with high dynamic range, we recommend you use a more sophisticated package, e.g., software that uses exact precision numbers or symbolic calculation.

A Note on Data

In accordance with the Apple Style Guide and common usage, we use data as a singular collective noun. Insisting that data are plural may be on your agendum, but I will continue to read a magazine while a panini is prepared. My stamina is not great enough to argue.

References

Goldberg, David (March 1991), What every computer scientist should know about floating-point arithmetic., ACM Computing Surveys 23 (1): 5–48, doi:10.1145/103162.103163

Chan, Tony F.; Golub, Gene H.; LeVeque, Randall J. (1979), Updating Formulae and a Pairwise Algorithm for Computing Sample Variances, Technical Report STAN-CS-79-773, Department of Computer Science, Stanford University.

Chan, Tony F.; Golub, Gene H.; LeVeque, Randall J. (1983). Algorithms for Computing the Sample Variance: Analysis and Recommendations. The American Statistician 37, 242-247. https://www.jstor.org/stable/2683386

Higham, Nicholas (2002). Accuracy and Stability of Numerical Algorithms (2 ed) (Problem 1.10). SIAM.

Pebay, P.; Terriberry, T.B.; Kolla, H. ;Bennett, J. Formulas For The Computation of Higher-Order Central Moments

Manning, E. Floating-point Summation Dr. Dobbs September 01, 1996