Eigendecomposition Denoising

The Eigendecomposition Denoising option in the Filter menu or the main toolbar is a specialized Eigendecomposition filtration procedure that sets a signal strength threshold for retaining all signal-bearing eigenmodes and filtering out all noise-containing ones. This option is used exclusively to remove noise. In general, eigenfiltration is the most effective method for denoising.

Uniformly spaced data are needed so that no additional noise is introduced. This procedure does not automatically generate uniform data for the Eigendecomposition. If needed, the Spline Estimation or Local Regression Spline Estimation option can be used to create an interpolated data set with constant X-spacing.

The option presents a dual TableCurve 2D graph with the eigenvalues in the upper graph and the input and output time domain data in the lower graph.

The graph's toolbar has a button that can toggle on and off and change the proportions of the two graphs in the dialog. The default has the eigenvalue graph using the upper third of the region, and the output graph using the lower two-thirds.

Algorithm

The eigendecomposition algorithms vary only by how a lagged covariance matrix is constructed:

• CovM Fwd - the covariance matrix equivalent of processing a forward prediction data matrix
• CovM FB - the covariance matrix equivalent of processing a forward-backward prediction data matrix
• CovM Tpltz - a Toeplitz symmetric covariance matrix constructed using the Burg procedure

In general, the differences between the methods will be small. Because it imposes a matrix structure, the CovM Tpltz algorithm is typically the least effective in mapping data trends. If the AR algorithm properties apply, the CovM FB algorithm should be the most effective of the procedures.

Full data matrix procedures are not needed since no parametric model is computed. The decomposition of a forward or forward-backward data matrix produces the same eigenvalues and eigenvectors as the equivalent covariance matrix, and requires more processing time with large data sets. Similarly, no backward prediction data/covariance matrix algorithm is needed since it would produce the same decomposition as the forward data/covariance matrix.

In this procedure, the main task is to set a sorted eigenvalue threshold whereby the noise that is present in the signal can be removed. All of the eigenmodes are used in the reconstruction. The principal components of the signal will be captured in the initial eigenmodes. The noise is broken into low power elements that generally form a sloping floor beyond the signal eigenmodes in the eigenvalue plot.

Order

Establishing the eigendecomposition order for harmonic signals in the absence of noise is a simple matter. A model order of two is needed to fully describe one oscillation. Similarly an order of four is needed to fully model two oscillatory components. For noise-free data, the minimum order needed is twice the number of signal components introducing oscillations in the data. These oscillatory components can be harmonic, such as undamped or damped sinusoids, sawtooth, and others, or they can be anharmonic: oscillations where the model is not readily apparent.

Since there is usually some level of noise present in the data, a higher order model is needed to also account for the oscillations introduced by noise. To achieve a reasonable signal-noise separation within an eigendecomposition, it is necessary to use a high enough order so that the primary eigenvectors span only signal space. In general, the order is not particularly critical so long as it is sufficient to produce an effective partitioning of the signal and noise. What is crucial is the accurate specification of the signal space threshold.

Signal Subspace Selection

The number of principal eigenmodes comprising signal components can be specified numerically if you know with a certainty the number of spectral components present in the data. Again, you must enter a value that is twice the number of components. If three spectral components are known to exist, the signal subspace must be set to six. Even when the spectral component count is known, you should use the graphical inspection of eigenvalues to insure that a high enough order is being used to achieve the desired signal-noise separation.

When there is sufficient signal-noise separation in the eigenmodes, the eigenvalue plot reveals one or more sharp transitions between the signal subspace and the noise subspace floor. The last eigenmode before the long sloping noise floor represents the last element of signal space. Assuming a high-enough eigendecomposition order is used, this signal-noise separation does not become difficult until the noise level approaches that of the signal. At this point, the sharp characteristic transition may disappear.

To select the signal space graphically, left click on the last of the eigenvalues judged to represent signal space. All of the eigenvalues used in the reconstruction will be circled.

Estimated Noise Reduction

TableCurve 2D offers a robust noise estimation procedure that may be of some value for low-frequency signals. A cubic polynomial interpolation is made for each point using the two points to the left and the two to the right (excluding the current point). The difference between the interpolated and data values is used to generate a measure of the white noise present in the data. This assumes that the data can be locally characterized by a smooth cubic interpolant. Also, the signal component(s) should exist only in the lower quarter of the Nyquist range. If a high frequency signal component is present, these estimates of noise will be invalid.

The In value reports the estimated white noise in the incoming data, the Out value the estimated white noise for the filtered signal. The percent is given as the amount of estimated noise remaining after filtration.

Power Reduction

In this section the In value reports the TISA power in the incoming data. The Out value is the TISA power for the filtered signal. The percent is given as the amount of power remaining after filtration. For most S/N ratios, the reduction in power should be minimal. These value can alert you when signal components are being discarded.

Correlation Coefficient

The r-squared correlation coefficient should also remain high. An r² of 1 is a perfect correlation while a value of 0 means the filtered and unfiltered signals are uncorrelated. Low r² values are also indicative of signal components being lost in the thresholding.

List

The List Data option lists the index, time, and output signal in a three column table. The listing uses the TableCurve 2D text viewer facility.

Copy

The Copy Data to Clipboard option copies the time and output signal values to the clipboard. Formats include full precision binary (for spreadsheets such as Excel) and ASCII (for pasting into text editors).

Save

The Save Data to Disk option writes the time and output values to a supported file format. These formats include ASCII, Excel 97/2000, Excel 95, Lotus WK3, Lotus WK1, SPSS, or Systat.

Production Facility

The TableCurve 2D Automation facility allows unattended processing of large numbers of data sets. The data sets can be consolidated in an Excel file or acquired using a DLL. The graphs and reports can be exported to an MS Word/RTF file, while the processed data can be exported to an Excel 95 or Excel 97/2000 file.

The Reset button restores the data to its state when first entering the procedure. Note that if you implement sequential local procedures, all of the revisions are discarded upon reset. If an Automation Session is in progress, the Reset button can be used to terminate the automated processing.

Residuals

The Residuals button is used to graphically display the difference between the original and filtered data.

Numeric Summary

The Numeric Summary offers an eigendecomposition report. The report includes a table of eigenvalues, normalized eigenvalues, and partial r² correlation coefficients.

MS Word/RTF Export

The MS Word/RTF File Export option is used to save the current graphs and reports to either an MS Word file or a portable RTF (Rich Text Format) file. The graphs are inserted into the file as Windows metafiles.

Automatic Update

The List, Residuals, and Numeric Summary windows are automatically updated when any change is made to the algorithm's settings.

Updating the Data Table

When exiting this procedure with the OK button, an option will be presented to update TableCurve 2D's main data table with the filtered data.

If Background Thread Fitting is active, the fitting will be initiated as soon as the filtering is accepted and the data are rescanned.