Skip to content

High Precision Polynomials and Rationals

TableCurve 2D has traditionally limited polynomials to order 10 and rationals to orders 5/5 due to precision considerations. In general, an 11x11 matrix (order 10 polynomial) is about all the 80-bit long double precision of the Intel floating point hardware can reasonably manage. For 64-bit double precision math, a reasonable limit is a 9x9 matrix (order 8 polynomial).

With release 4, TableCurve 2D offers polynomials through 20th order, and rationals through 10/10th order. The coefficient upper limit in TableCurve 2D has thus gone from 11 to 21. In order to produce effective fits of these higher order models, TableCurve 2D performs all fit computations, including matrix solution, using an entirely separate high-precision math engine. Using 32-bit assembly language, and precision twice that of the Intel hardware (38 digits), a very fast and effective means for safely fitting these higher order polynomials and rationals is offered. Although the equation is evaluated with the Intel precision, the confidence and prediction intervals are computed using this same 38-digit high-precision math.

If you are fitting full precision generated data, it is now possible for TableCurve 2D to produce fractional residuals near the 1E-18 limit for long double precision (TC2D, SigmaPlot, Lotus 123), and near the 1E-15 range for double precision (Excel, Mathcad). Previously, on near perfect 11 coefficient polynomial or rational fits, it was seldom that fractional residuals better than 1E-12 could be realized. For this reason, TC2D's implementation of these high precision models begins with an order 4 polynomial and an order 2/2 rational.

The new equations which use this 38-digit high-precision math engine are as follows:

  • Polynomials: 4th order through 20th order, Equations 6054-6070
  • *Rationals: 2/2 order through 10/10 order, Equations 7054-7070*

While the TC2D high-precision software floating point is exceptionally fast, it is still a matter of software emulation, and as such, the fitting of these equations may appear to be considerably slower than those fitted using the Intel floating point unit. This may be particularly evident when confidence or prediction intervals are being graphed.