Fourier Estimation

The Fourier Estimation option in the Estimate menu or the Process toolbar offers true interpolation and smoothing based upon the frequency spectrum. When all frequency channels are active, interpolation occurs. When thresholds are set, this option performs filtering and reconstruction. For performance reasons, this procedure is not recommended for very large data sets.

Uniformly spaced data are needed so that no additional noise is introduced. This procedure does not automatically generate uniform data for the Fourier decomposition. If needed, the Spline Estimation or Local Regression Spline Estimation option can be used to create an interpolated data set with constant X-spacing.

This option presents a dual TableCurve 2D graph with the frequency domain decomposition in the upper graph and the input (unfiltered) and output (filtered) time domain data in the lower graph.

Interpolation

Fourier interpolation is based upon the frequency spectrum. This means that any size reconstruction will generate the same Fourier spectrum. For interpolation only, the thresholds should remain empty.

Filtering

This procedure is similar to the Fourier Filtering option, in that frequency and spectral thresholds can be set which include or exclude specific FFT channels from the reconstruction.

Reconstruction

In this procedure, the reconstructed data are computed directly from the amplitude, frequency, and phase of the continuous sine components. While this can be appreciably slower than an inverse FFT, this does offer a reconstruction where the data count and limits are variable. In addition to reconstructing the basic data, this option also makes it possible to reconstruct the first, second, third, or fourth derivatives. If a derivative is reconstructed, the thresholding should be specified to sufficiently smooth the data in order to produce the cleanest possible derivative.

Post-Filtering

When this option is used to increase the data count within the original time interval, the Nyquist frequency will increase. A follow up FFT of the reconstructed data will contain frequencies that lie beyond those in the original FFT. This may be fine, representing a valid extrapolation and gentle decay. On the other hand, all frequencies beyond the original Nyquist can be zeroed by checking the Low Pass post-filter option.

Transform

This procedure fixes the FFT method to the Best Exact N algorithm. It is used to produce the Fourier decomposition and for the forward and inverse transforms in the post-filter procedure. The Best Exact N composite algorithm produces the fastest possible exact n FFT.

Plot

The frequency domain information can be plotted in a variety of formats. In the following table, Re is the real component of the FFT at a given frequency, Im is the imaginary component, n is the data set size, dx is the sampling interval, and var is the variance of the data series.

• Magnitude, sqrt(Re*Re+Im*Im)

• Amplitude, 2.0*sqrt(Re*Re+Im*Im)/n

• Phase, sine-based, Pi/2+atan(Im/Re)

• Variance, Power normalized by variance, (Re*Re+Im*Im)/n/var

• dB, decibels, 10.0*log10(Re*Re+Im*Im)

• dB Norm, decibels, normalized to 0 for frequency channel with maximum power

• PSD SumSq, Power as Sum Squared Amplitude, 2.0*(Re*Re+Im*Im)/n

• PSD MeanSq, Power as Mean Squared Amplitude, 2.0*(Re*Re+Im*Im)/n/n

• PSD TimeInt, Power as Time-Integral Squared Amplitude, 2.0*dx*(Re*Re+Im*Im)/n

In an amplitude plot, you see the actual amplitude of sine components. In a normalized decibel plot, the highest peak is at 0dB, a peak at -3dB would have half the power, and a peak at -6dB would have half the amplitude. The PSD TISA (time-integral squared amplitude power) is the actual integral under the curve defined by the square of the raw data.

If the data set size is 512 or fewer (257 or fewer points in the frequency spectrum), a bar format is automatically used. Otherwise, the points in the spectrum will be rendered directly with lines connecting the points. You can override either of these point formats using the Modify Point Format option and their respective states will be saved across sessions.

Frequency Thresholds

There are several ways to set the thresholds for Fourier filtering. The first is to define a range of frequency space. This is primarily used to isolate and reconstruct signal components. The Freq1 and Freq2 fields set limits in frequency. The frequency boundaries of this region can either be included (Incl) or excluded (Excl). If a frequency range is included, this means that only the frequencies that fall within this range are used in the data reconstruction. If the region is excluded, only those frequencies that fall outside the region are used for the reconstruction.

The frequencies can be entered numerically. If both fields for frequency are left blank, the full range is assumed for the Incl option, and no range is specified for the Excl option. If only a lower limit is entered, the variable range is assumed to be from this limit and greater for the Incl option, and all values below this limit if the Excl option is set. If only an upper limit is entered, the variable range is assumed to be from this limit and lower for Incl, and all values above this limit for Excl.

It is generally easier to set a frequency range graphically. When the X Sectioning mode is selected, the mouse is used in the upper graph to select the frequency range of interest. Frequencies are toggled off by holding the left mouse button down and sliding right and toggled back on by moving left. Individual frequencies can be toggled on and off by placing the cursor at the point (or top of a bar) and left clicking the mouse.

Spectral Thresholds

Another way to set the thresholds for Fourier filtering involves defining spectral thresholds. This is primarily used to separate signal components by power. It can also be used for noise removal, replicating the functionality in the Fourier Denoising option, or for signal removal, generating a signal-free background for subsequent analysis.

The Spec1 and Spec2 fields set the spectral limits. These can only be entered numerically. If only Spec1 is set, all values at or above this threshold will be used in the reconstruction if Incl is set, and all below this threshold will be used if Excl is checked. If only Spec2 is set, all values at this threshold and below will be used in the reconstruction if Incl is set, and all above this threshold will be used if Excl is set. For this option, the normalized dB values must use the minus sign.

Frequency-Spectral Thresholds

The thresholds for Fourier filtering can also consist of a frequency-spectrum region. This allows signal components to be isolated by both frequency and spectral power. This is the most effective way to isolate and reconstruct a single signal component when some number of components are present.

The Freq1 and Freq2 fields set limits in frequency while the Spec1 and Spec2 fields set the spectral limits. The values can be entered numerically. If only a lower limit field is set, all values at or above this threshold will be used in the reconstruction if Incl is set, and all below this threshold will be used if Excl is checked. If only the upper limit is set, all values at this threshold and below will be used in the reconstruction if Incl is set, and all above this threshold will be used if Excl is set. Again, the normalized dB values must use the minus sign.

It is generally easier to set a frequency-spectrum region graphically. When the XY Sectioning mode is selected, the mouse is used in the upper graph to enclose the frequency-spectral region of interest. To include the region that is to be graphically defined, click and hold down the left mouse button at one corner of the frequency-spectral rectangle in the upper graph and drag the rectangle so that it encloses the region desired. This will automatically enter both the frequency and spectral limits and set the process state to include the FFT channels located within the region.

To exclude the region that is to be graphically defined, the same graphical procedure is followed except that the right rather than the left mouse button is used. In this case, the process state will be set to exclude the FFT channels within the region. This is useful for reconstructing data with a single component removed, such as line noise.

Note that it is possible to reconstruct only one frequency-spectral region at a time.

The Clear button is used to clear all of the threshold fields.

The Apply button can be used to immediately update the processing rather than wait for the automatic update.

Popup Information

The Toggle Popup Information Window option is used to report estimated noise and power reductions as well as a correlation coefficient between the filtered and unfiltered data.

TableCurve 2D offers a robust noise estimation procedure that may be of some value for low-frequency signals. A cubic polynomial interpolation is made for each point using the two points to the left and the two to the right (excluding the current point). The difference between the interpolated and data values is used to generate a measure of the white noise present in the data. This assumes that the data can be locally characterized by a smooth cubic interpolant. Also, the signal component(s) should exist only in the lower quarter of the Nyquist range. If a high frequency signal component is present, these estimates of noise will be invalid. The Noise In value reports the estimated white noise in the incoming data, the Noise Out value the estimated white noise for the filtered signal. The Noise % is given as the amount of estimated noise remaining after filtration.

The TISA In value reports the TISA power in the incoming data. The TISA Out value is the TISA power for the filtered signal. The TISA % is given as the amount of power remaining after filtration.

The r-squared correlation coefficient is also reported. An r² of 1 is a perfect correlation while a value of 0 means the filtered and unfiltered signals are completely uncorrelated.

Output Data

Since the Fourier decomposition can be expressed as continuous infinite-length phase-bearing sinusoids, it is possible to generate interpolated output for any number of values (n) and any range of times (Time1 to Time2). Since the Fourier reconstruction reproduces the same data series outside the range of original data, there is no value in generating extrapolations. This procedure cannot be used for prediction.

In addition to reconstructing the sinusoidal basis functions (Function), it is also possible to reconstruct the first through fourth derivatives (1st Deriv, 2nd Deriv, 3rd Deriv, 4th Deriv). For derivatives, be very cautious with noisy data. A significant level of filtering is generally needed to produce accurate derivatives. In general, the Savitzky-Golay Spline Estimation option is recommended for serious derivative evaluations.

The output data stream is limited to 65536 values.

Post-Filter

If the Low Pass item is checked, an additional Fourier-based low pass filtration is performed on the output data. All frequencies greater than the Nyquist limit of the original data series will be zeroed and an inverse FFT will generate the final output. Note that the additional frequency information arising from interpolating a higher data density is often a valid extrapolation. Unlike the Fourier Upsampling procedure, this post filter is optional.

List

The List Data option lists the index, time, and output signal in a three column table. The listing uses the TableCurve 2D text viewer facility.

Copy

The Copy Data to Clipboard option copies the time and output signal values to the clipboard. Formats include full precision binary (for spreadsheets such as Excel) and ASCII (for pasting into text editors).

Save

The Save Data to Disk option writes the time and output values to a supported file format. These formats include ASCII, Excel, SPSS, or Systat.

Production Facility

The Automation facility allows unattended processing of large numbers of data sets. The data sets can come from a text/CSV file or an Excel workbook. The graphs and reports can be exported to an RTF file or streamed to Microsoft Word, while the processed data can be exported to an Excel (xlsx) workbook.

The Reset button restores the data to its state when first entering the procedure.

Residuals

The Residuals button is used to graphically display the difference between the original and filtered data.

Numeric Summary

The Numeric Summary offers a full FFT report. The report optionally includes a listing of the interpolated spectral peaks, a frequency analysis, and a linear sinusoidal least-squares fit summary.

FFT Summary

The List Frequency Domain Data option offers an extended FFT data summary. The listing uses the TableCurve 2D text viewer facility. The FFT channel number, frequency, and magnitude are always listed. The Format menu offers the optional selection of the following:

• Add Real,Imag

• Add Amplitude

• Add Wavelength

• Add Phase (Sine-based)

• Add dB

• or Add dB Normalized

• Add Power Spectral Density, Sum Squared Amplitude

• or Add Power Spectral Density, Mean Squared Amplitude

• or Add Power Spectral Density, Time-Integral Squared Amplitude

In the following table, Re is the real component of the FFT at a given frequency, Im is the imaginary component, n is the data set size, and dx is the sampling interval:

• Magnitude, sqrt(Re*Re+Im*Im)

• Phase, sine-based, Pi/2+atan(Im/Re)

• Amplitude, 2.0*sqrt(Re*Re+Im*Im)/n

• dB, decibels, 10.0*log10(Re*Re+Im*Im)

• dB Norm, decibels, normalized to 0 for frequency channel with maximum power

• PSD SumSq, Power as Sum Squared Amplitude, 2.0*(Re*Re+Im*Im)/n

• PSD MeanSq, Power as Mean Squared Amplitude, 2.0*(Re*Re+Im*Im)/n/n

• PSD TimeInt, Power as Time-Integral Squared Amplitude, 2.0*dx*(Re*Re+Im*Im)/n

The amplitude and phase of each component in the FFT is derived from sine-based conversion. Each of the components in the FFT can be reconstructed using:

Y=Amplitude*sin(2*PI*Frequency*X+Phase)

MS Word/RTF Export

The MS Word/RTF File Export option is used to save the current graphs and reports to either an MS Word file or a portable RTF (Rich Text Format) file. The graphs are inserted into the file as Windows metafiles.

Evaluation

The Evaluation option offers the means to more extensively evaluate the sum of sinusoids model generated by the Fourier decomposition. The evaluation can be be direct, or the computation can consist of first or second derivatives, roots, or integrated areas. This option can be used to generate a table or file of any size using a generated X grid or by importing X data from supported file formats.

Automatic Update

The List, Residuals, Numeric Summary, Fourier Summary, and Evaluation windows are automatically updated when any change is made to the algorithm's settings.

Updating the Data Table

When exiting this procedure with the OK button, an option will be presented to update TableCurve 2D's main data table with the estimated data.

If Background Thread Fitting is active, the fitting will be initiated as soon as the estimation is accepted and the data is rescanned.