Fourier Denoising¶

The Fourier Denoising option in the Filter menu or the main toolbar is a specialized Fourier filtration procedure that sets either a frequency threshold for low pass frequency-domain filtration, or a signal threshold for zeroing all spectral elements below a given dB (normalized) value. This option is used exclusively to remove noise via a Fourier decomposition.

Uniformly spaced data are needed so that no additional noise is introduced. This procedure does not automatically generate uniform data for the Fourier decomposition. If needed, the Spline Estimation or Local Regression Spline Estimation option can be used to create an interpolated data set with constant X-spacing.

The option presents a dual TableCurve 2D graph with the frequency domain decomposition in the upper graph and the input and output time domain data in the lower graph.

The graph's toolbar has a button that toggles on and off and changes the proportions of the two graphs in the dialog. The default has the Fourier graph using the upper half of the region and the output graph using the lower half.

Algorithm¶

This procedure uses the Best Exact n FFT procedure to create the frequency spectrum in the upper graph. The Best Exact n FFT procedure is also used for the FFT inverse that reconstructs the time domain data from the filtered frequency domain representation. The spectrum is always displayed in a normalized decibel format.

If the data set size is 512 or lower (257 or lower points in the frequency spectrum), a bar format is automatically used. Otherwise, the points in the spectrum will be rendered directly with lines connecting the points. You can override either of these point formats using the Modify Point Format option and their respective states will be saved across sessions.

In this procedure, the main task is to set a threshold, either in frequency or spectral magnitude, whereby the noise that is present in the signal can be removed. When a Frequency threshold is selected, all frequency domain information beyond this specified frequency is zeroed in order to create the output data. It is quite common for a signal to exist at low frequencies, whereas white noise exists across all frequencies. This low-pass filtration preserves the signal and discards the noise within the higher frequencies. The noise that exists within the retained signal frequencies is not filtered.

Unlike time domain filters, a Fourier domain lowpass filter can achieve an absolute truncation at the specified frequency. Whereas discontinuities in time domain data introduce real problems with Fourier analysis, this "cliff" or "brickwall" discontinuity that is produced in the Fourier spectrum of the output data is not usually a concern. In fact, it is usually the ideal in terms of noise reduction and smoothing. If the output data are subsequently transformed into the Fourier domain using an exact-n FFT, it will look exactly like the FFT for the active or retained channels in the upper graph. The spectral magnitudes for the frequencies that had been zeroed will exist at approximately -300dB, the machine precision limit, in a normalized dB plot.

If a dB threshold is selected, the frequency channels are zeroed based on a specified signal threshold. If the signal of interest can be assumed to exist primarily in the highest dB Fourier channels, as is often the case, it is possible that a much higher percentage of the overall channels can be zeroed. This is the only way to threshold a signal with high frequency components within the Fourier domain.

If possible, it is usually a good idea to retain the adjacent sidelobes of each spectral component.

Unlike the frequency thresholding, dB thresholding can result in multiple "cliff" or "brickwall" discontinuity zones in the Fourier spectrum of the output data. This is especially true when multiple components are present. There may be any number of zones at approximately -300dB in a normalized dB spectrum. Again, this is generally the ideal for removing noise.

If you really need to produce soft transitions between the retained and discarded frequencies, you can use the Fourier Filtering procedure with a data tapering window to implement this same type of frequency or dB thresholding. When this is done, a subsequent transform of the filtered data into the Fourier domain using an exact-n non-windowed FFT will produce the transitions typical of time-domain filter algorithms.

The dB thresholding, in one simple step, accomplishes what could require a number of bandpass filters in the time domain and it does so with brickwall cutoffs. In this basic procedure, the dB threshold is fixed for all components. To filter using a different dB threshold for each component, the Fourier Filtering option must be used.

Thresholding¶

The Frequency and dB threshold values can be entered numerically or graphically. To enter a frequency graphically left click the mouse on the upper graph at the frequency desired (the dB position is unimportant). To enter a dB value graphically, left click the mouse in the upper graph at the desired dB (the frequency is unimportant). In this option, it is not possible to graphically toggle individual frequency channels on and off.

The Reset Previous button restores the threshold value last used in this procedure.

Removal of Linear Trend¶

Check the Remove Trend box to remove a two point (initial and final data value) linear trend from the data stream. The data trend is automatically restored in the reconstruction.

Data Tapering Window¶

"Raised" data taper windows (those that are non-zero at the bounds) are an effective way to reduce spectral leakage and achieve a more effective noise removal. They should be used with caution since a taper discards much of the information near the bounds of the data.

The following data taper windows are offered:

None
cs2 Hamming
Chebyshev 2
Chebyshev 2.5
cs3 BHarris3
cs3 Blckmn Exact
cs3 Nutall min
Chebyshev 3
Chebyshev 3.5

The full selection of data windows is offered in the Fourier Filtering option.

Estimated Noise Reduction¶

TableCurve 2D offers a robust noise estimation procedure that may be of some value for low-frequency signals. A cubic polynomial interpolation is made for each point using the two points to the left and the two to the right (excluding the current point). The difference between the interpolated and data values is used to generate a measure of the white noise present in the data. This assumes that the data can be locally characterized by a smooth cubic interpolant. Also, the signal component(s) should exist only in the lower quarter of the Nyquist range. If a high frequency signal component is present, these estimates of noise will be invalid.

The In value reports the estimated white noise in the incoming data, the Out value the estimated white noise for the filtered signal. The percent is given as the amount of estimated noise remaining after filtration.

Power Reduction¶

In this section the In value reports the TISA (time-integral squared amplitude) power in the incoming data. The Out value is the TISA power for the filtered signal. The percent is given as the amount of power remaining after filtration. For most S/N ratios, the reduction in power should be minimal. These values can alert you when signal components are being discarded.

Correlation Coefficient¶

The r-squared correlation coefficient should also remain high. An r² of 1 is a perfect correlation while a value of 0 means the filtered and unfiltered signals are uncorrelated. Low r² values are also indicative of signal components being lost in the thresholding.

List¶

The List Data option lists the index, time, and output signal in a three column table. The listing uses the TableCurve 2D text viewer facility.

Copy¶

The Copy Data to Clipboard option copies the time and output signal values to the clipboard. Formats include full precision binary (for spreadsheets such as Excel) and ASCII (for pasting into text editors).

Save¶

The Save Data to Disk option writes the time and output values to a supported file format. These formats include ASCII, Excel 97, Excel 95, Lotus WK3, Lotus WK1, SPSS, or Systat.

Production Facility¶

The TableCurve 2D Automation facility allows unattended processing of large numbers of data sets. The data sets can be consolidated in an Excel file or acquired using a DLL. The graphs and reports can be exported to an MS Word/RTF file, while the processed data can be exported to an Excel 95 or Excel 97/2000 file.

The Reset button restores the data to its state when first entering the procedure. If an Automation Session is in progress, the Reset button can be used to terminate the automated processing.

Residuals¶

The Residuals button is used to graphically display the difference between the original and filtered data.

Numeric Summary¶

The Numeric Summary offers a full FFT report. The report optionally includes a listing of the interpolated spectral peaks, a frequency analysis, and a linear sinusoidal least-squares fit summary.

FFT Summary¶

The List Frequency Domain Data option offers an extended FFT data summary. The listing uses the TableCurve 2D text viewer facility. The FFT channel number, frequency, and magnitude are always listed. The Format menu offers the optional selection of the following:

Add Real,Imag
Add Amplitude
Add Wavelength
Add Phase (Sine-based)
Add dB
or Add dB Normalized
Add Power Spectral Density, Sum Squared Amplitude
or Add Power Spectral Density, Mean Squared Amplitude
or Add Power Spectral Density, Time-Integral Squared Amplitude

In the following table, Re is the real component of the FFT at a given frequency, Im is the imaginary component, n is the data set size, and dx is the sampling interval:

Magnitude, sqrt(Re*Re+Im*Im)
Phase, sine-based, Pi/2+atan(Im/Re)
Amplitude, 2.0*sqrt(Re*Re+Im*Im)/n
dB, decibels, 10.0*log10(Re*Re+Im*Im)
dB Norm, decibels, normalized to 0 for frequency channel with maximum power
PSD SumSq, Power as Sum Squared Amplitude, 2.0*(Re*Re+Im*Im)/n
PSD MeanSq, Power as Mean Squared Amplitude, 2.0*(Re*Re+Im*Im)/n/n
PSD TimeInt, Power as Time-Integral Squared Amplitude, 2.0*dx*(Re*Re+Im*Im)/n

In an amplitude plot, you see the actual amplitude of sine components. In a normalized decibel plot, the highest peak is at 0dB, a peak at -3dB would have half the power, and a peak at -6dB would have half the amplitude. The PSD TISA (time-integral squared amplitude power) is the actual integral under the curve defined by the square of the raw data.

The amplitude and phase of each component in the FFT is derived from sine-based conversion. Each of the components in the FFT can be reconstructed using:

Y=Amplitude*sin(2*PI*Frequency*X+Phase)

MS Word/RTF Export¶

The MS Word/RTF File Export option is used to save the current graphs and reports to either an MS Word file or a portable RTF (Rich Text Format) file. The graphs are inserted into the file as Windows metafiles.

Automatic Update¶

The List, Residuals, Numeric Summary, and Fourier Summary windows are automatically updated when any change is made to the algorithm's settings.

Updating the Data Table¶

When exiting this procedure with the OK button, an option will be presented to update TableCurve 2D's main data table with the denoised data.

If Background Thread Fitting is active, the fitting will be initiated as soon as the filtering is accepted and the data are rescanned.