Precision Summary

The Precision Summary is used to determine how much precision is preserved in the current equation when the successively fewer digits of precision is used in the coefficients. The absolute error is measured at the eight points defining the boundary of the data and the point closest to the center of the data region.

Rank 3 Eqn 522 Fourier Series Bivariate Order 2x3 Precision Avg Abs Error Min Abs Error Max Abs Error 18 3.376567e-19 2.087077e-19 3.969445e-19 17 1.772923e-18 8.120427e-19 2.374379e-18 16 1.133997e-17 5.305282e-18 1.736522e-17 15 3.52803e-17 8.245356e-18 5.556047e-17 14 5.666004e-15 4.999588e-15 6.485972e-15 13 2.622666e-14 1.609856e-14 3.467439e-14 12 5.679146e-13 4.712875e-13 8.780627e-13 11 4.544998e-12 2.381442e-12 5.320463e-12 10 5.817554e-11 4.932095e-11 8.147465e-11 9 1.685869e-10 8.576211e-11 3.184382e-10 8 6.79325e-09 5.977878e-09 9.963009e-09 7 6.548798e-08 5.947627e-08 8.484949e-08 6 3.520908e-07 2.192526e-07 5.671748e-07 5 2.908907e-06 2.365605e-06 4.471654e-06 4 6.956808e-05 5.9644e-05 0.0001030723 3 7.133687e-05 4.286841e-06 0.0001211894 2 0.0037096721 0.0027285106 0.0047661149

 

Removing Avg Abs Error Min Abs Error Max Abs Error a 1.0596242226 0.9923359933 1.3378096069 b 0.0091994726 0.000267876 0.0148408101 c 0.0031139932 9.725343e-05 0.0046814562 d 0.0264129175 0 0.094386025 e 0.0470573894 0 0.161055115 f 0.0148695223 0.013929169 0.0187636483 g 0.0114416013 0.010718163 0.0144381793 h 2.662055e-05 0 9.512798e-05 i 7.925917e-05 0 0.0002712665 j 0.0002533087 2.16514e-05 0.0004000529 k 0.0030095073 0.0002754973 0.0044228393 l 0.0021569034 0 0.0077076507 m 0.0054859386 0 0.0187757647 n 0.0009663217 1.129649e-06 0.0021354339 o 0.0002939174 0 0.0013395327 p 0.0005827012 0 0.0025915142 q 0.0078616272 0.0002289839 0.0126760863 r 0.0007002683 2.187599e-05 0.001052205 s 8.607295e-06 0 3.922787e-05 t 1.526588e-05 0 6.789369e-05 u 0.0075530137 0 0.0679771236 v 0.027681942 0 0.0988736888 aa 0.0108127799 0 0.0369888778 ab 4.343673e-06 0 3.909306e-05 ac 3.155758e-05 0 0.0002840182

 

For linear equations, the impact of removing (zeroing) each parameter is also shown.

The Precision Summary is toggled on and off by the Precision button in the Surface-Fit Graph window. The window size and position you choose for the Precision Summary is automatically saved across sessions. The Precision Summary uses the TableCurve Studio Text Viewer to display, modify, and print the information.

For 2D (XY) Data

For a 2D (XY) fit, the error is measured at just three points -- Xmin, Xmean, and Xmax -- rather than the eight boundary points and center point used for a 3D surface. The Precision Summary is particularly recommended when contemplating the use of Chebyshev converted coefficients.