Data Quality Assurance for CBM Sensors Jeremy Branning
[email protected] RMCI, Inc. US Army Aviation Engineering Directorate Redstone Arsenal, AL
Matt Szelistowski
[email protected] RMCI, Inc. US Army Aviation Engineering Directorate Redstone Arsenal, AL
Stephen Potts
[email protected] RMCI, Inc. US Army Aviation Engineering Directorate Redstone Arsenal, AL
ABSTRACT Helicopter onboard diagnostic systems provide advanced fault detection for degraded mechanical components. The Army’s ADS-79C provides a framework for granting maintenance relief and presents a well-established process for developing and validating mechanical diagnostic algorithms. Diagnostic system manufacturers have implemented a wide variety of methods for ensuring the quality of the data upon which the mechanical diagnostics are performed. This paper calls for a standard approach for mitigation of all critical data quality issues which could affect mechanical diagnostics. Lessons learned from the Army’s data analysis are presented, as well as recommendations for a standardized full-coverage data quality algorithm suite for ADS-79 implementation.
ACRONYMS ADC ADS AFD ATD BIT CBM DC DSC DQ DQA FFT HUMS MR 1P P2P SNR
Analog to Digital Converter Aeronautical Design Standard Asynchronous Frequency Domain Asynchronous Time Domain Built-in Test Condition Based Maintenance Direct Current Digital Source Collector Data Quality Data Quality Amplifier Fast Fourier Transform Health and Usage Monitoring System Main Rotor Once Per Revolution Vibration Peak to Peak Signal to Noise Ratio
INTRODUCTION Vibration analysis is a key enabler for Condition Based Maintenance (CBM). Condition indicators (CI) have been developed to provide an indirect health assessment on shafts, gears, and bearings within the drivetrain. These CIs mathematically extract features from data collected by permanently installed piezoelectric accelerometers. A rigorous approach for CI design, application, and validation is provided in ADS-79C1 Appendices D, I, and K. The 1
Aeronautical Design Standard 79C HDBK, Army Materiel Command Standardization Office,
manufacturers of the Army’s digital source collectors (DSC) have implemented a wide variety of approaches to ensure that the raw data used by these CIs for decision making is trustworthy. The Army has been evaluating the effectiveness of these and a variety of other approaches. General recommendations for data quality are given in ADS-79C sections E.3.2.1, F.6, L.5, M.6.1.1.1, and M.6.1.1.3. The logical next step is to define detailed recommendations for ensuring data quality. A standardized, rigorous process can be implemented which will leverage and complement the mechanical diagnostic validation techniques. A general overview of data quality failures and detection techniques are presented in the following sections. This discussion is not comprehensive, but is an example of the type and level of detail which will be necessary in forming the future requirements for accelerometer data quality. Each of these algorithm types has been applied to Army CBM systems with varying degrees of success.
BIT vs DQ Built-in-tests (BITs) for accelerometers are nothing more than gross tests of the accelerometer and wiring circuitry using Ohm’s law. In many cases, BITs are conducted when power is first applied to the DSC, AMSAM-RA-IM-IS-IL, 2012.
Redstone
Arsenal,
AL,
while the aircraft is not running. In contrast, data quality (DQ) is an evaluation of the data collected from the accelerometer by the DSC. DQ checks are always performed with data from a running aircraft. A BIT simply indicates that there is an accelerometer attached to a circuit with a reasonable voltage drop. DQ determines whether the information collected by the accelerometer is reliable and usable. A sensor may pass BIT, but fail DQ. There are a broad range of DQ issues that may be encountered when installing accelerometers on a helicopter. A brief discussion of common DQ approaches is presented in the following sections.
Kurtosis Kurtosis is a quantification of the outlier portion of a distribution. Mathematically, kurtosis is defined as the fourth standard moment of a signal about the mean, normalized by the fourth power of the standard deviation. Kurtosis is literally a measure of the peakedness of a distribution. A signal with a normal distribution has a kurtosis value of 3.0, but a more outlier-heavy distribution will have a kurtosis of greater than 3. kurtosis = When applied to residual signals in a relatively quiet environment, such as an isolated test stand, kurtosis has demonstrated some success in identifying impulsive behavior, like a broken tooth on a spur gear. The kurtosis of the residual signal is very susceptible to minor noise sources and has largely 2
been discredited for use on helicopters. However, when applied to the raw, asynchronous time domain (ATD) signal from an accelerometer, the kurtosis can be a useful discriminator of data quality. When an accelerometer is physically loose, large steps may be observed in the signal output as the sensor physically shifts around. This can result in data spiking, discontinuities in data, or even very large impulses, depending on the looseness. Other 2
Antolick, L., et al., “Evaluation of Gear Condition Indicator Performance on Rotorcraft Fleet,” AHS 66th Annual Forum Proceedings, May 2010.
conditions, such as an object or wire physically contacting a properly mounted sensor, can produce a similar result. Intermittent electrical issues, or sensor saturation causing unusual behavior from the amplifier, can also cause shifts. Figure 2 shows a nominal sample of ATD data, with a kurtosis value of 2.97. Figure 3 shows the same ATD sample but with several discontinuous steps added by shifting the data. This shift results in a kurtosis value of 8.16. While impulsive content is normal, most impulses in the gearbox structure are underdamped, gradually “ringing” as they decay. The contribution of these underdamped impulses to the ATD signal does not significantly impact the kurtosis because much of their content is low-amplitude. The chief pitfall of using kurtosis for data quality on ATD data is that it is possible for any rapid impulse to cause a rise in kurtosis, even impulses caused by environmental effects. This has been observed on the UH-60 Black Hawk tail rotor gearbox output accelerometer. Initial observations revealed that nearly half of all acquisitions across the fleet resulted in data quality exceedances for kurtosis on this sensor. Other sensor locations did not experience kurtosis issues. Further analysis showed that the rate of kurtosis exceedances on the tail gearbox output was correlated with airspeed (Figure 4). Due to the strong correlation with airspeed, it was presumed that aerodynamic effects were at the root of the kurtosis issues. A comparison in asynchronous frequency domain (AFD) data for any aircraft with this issue reveals notable deltas between ground and flight from about 1-3 kHz. Demodulation in this band identifies the tail rotor blade 4P as the primary contributor. Large spikes are apparent in the ATD which make it easy to see why kurtosis would be triggered. The nature of the vibrations at the tail gearbox on the Blackhawk makes kurtosis unusable for DQ. The most significant problem encountered when using kurtosis as a data quality indicator is its sensitivity to actual component faults. Experience on the University of South Carolina AH-64 seeded fault test stand has shown that a broken tooth on a tail gearbox gear can result in a kurtosis value of over 10.0 (Fig. 4). The data in Figure 4 has unusually high
outlier content. If this significant fault occurred on an aircraft using kurtosis as a data quality check, it could be attributed to bad data. This situation illustrates the need for data quality algorithm validation. Unlike with mechanical diagnostics, false
positives can increase risk. Had kurtosis been applied to this signal, the data would be filtered out, and an alarm would not be triggered for the gearbox.
5 4 3
g
2 1 0 ‐1 ‐2 ‐3 0
0.05
0.1
0.15
0.2
0.25
time (sec)
Figure 1. Nominal asynchronous time domain data, kurtosis 2.97
5 4 3
g
2 1 0 ‐1 ‐2 ‐3 0
0.05
0.1
0.15
0.2
time (sec)
Figure 2. Discontinuous synchronous time domain data, kurtosis 8.16
0.25
Low Frequency Artifacts
Percentage Fleet Data Over Limit
Kurtosis Values Over Limit
Poor electrical connections between the accelerometer and the DSC often appear as a DC bias in the ATD signal. An FFT of this ATD reveals unrealistically high vibration appearing at low frequencies in the AFD spectrum. This is one of the most common data quality issues, and is often referred to as a “ski slope”, because of its appearance (Fig 5). This artifact renders the data unreliable at low frequencies, with a noise floor often extending to 500Hz or more.
100% 80% 60% 40% 20% 0%
Regime (kts)
Figure 3. Kurtosis exceedances as a function of airspeed, UH-60M TGB output sensor
Several related approaches have been implemented to detect DC bias in the accelerometer signal. All of the methods presented here operate on the AFD spectrum.
600 500 400
Low Frequency Slope/Intercept
300 200 100 0 -100 -200 -300 -400 0.5
1
1.5
2
2.5 5
x 10
Acceleration (g)
Figure 4, Time domain data from AH-64 TGB with broken tooth, kurtosis 11.6
40 30
Categorizing the shape of the low frequency content in the AFD spectrum can be successful in determining a DC bias condition. An implementation of this approach calculates the linear slope of the first few bins and the y-intercept. Care must be taken with this method to exclude bins around known low frequency contributors, such as a helicopter main rotor once per revolution (MR 1P) vibration. Some drawbacks to this method are that the “ski slope” shape is not always distinctive, especially with small levels of DC content. Also, this method is less elegant in that the slope and intercept must be calculated, combined, and stored separately.
Low Frequency Energy
20 10 0 0
200
400
Frequency (Hz)
Calculating the energy content of the low frequency bins of the ATD is an effective method for identifying DC bias. Implementation of this approach generally uses a root sum square (RSS) at low frequencies:
Figure 5: AFD spectrum from a UH-60L main transmission accelerometer with a "ski slope" caused by a DC bias
∑ 2
On the AH-64, the most common frequency band for low frequency energy calculation is 0-10Hz. This band also captures the MR 1P vibration. A wide variation in 1P levels makes it less practical to
threshold the low frequency energy for very subtle levels of DC content. This method of DC bias detection is more computationally efficient than slope/intercept, but less sensitive.
DC Offset DC Offset is the most robust and sensitive method for detection of spurious low frequency content in an accelerometer signal. As implemented by the Army, DC Offset calculates the amplitude of the highest peak less than 3Hz in the AFD spectrum. Since this excludes any known content such as MR 1P, any peak in this range can be assumed to be nonmechanical. This enables better thresholding, and consequently an earlier detection of DC bias issues.
bearings must transmit through the bladder to be measured by the accelerometer. The bladder is extremely effective at reducing shaft vibrations transmitted to the airframe, causing a reduction of about 30 dB across the spectrum. Because of the reduction caused by the bladder, the signal is extremely quiet. The quiet signal requires significant amplification; consequently, a high percentage of UH-60 tail shaft data collections are flagged for gain.
Signal to Noise Ratio (SNR) SNR operates on the premise that, with healthy data, certain frequencies will always stand out from the noise floor. SNR is calculated on the frequency domain, comparing the amplitude of expected frequencies to the amplitude of the adjacent bins. SNR requires knowledge about the expected frequencies, and must be set up with care in order to prevent nuisance exceedances. SNR must be configured carefully, with many variables affecting the calculation. Items to consider include the number of bins used for the expected tone and the adjacent region, the FFT windowing method used, and the drivetrain speed variation. Severe variations can occur with the expected peaks due to real physical conditions, making SNR an unreliable indicator of data quality.
Low Range Utilization Gain Some level of amplification is required at the analog to digital converter (ADC) to bring the signal to a usable range. The level of amplification required is indicative of the strength of the incoming signal. An abnormally high gain level or ratio would mean that the signal is very quiet, perhaps due to a dead sensor. A pitfall of gain has been experienced on the UH-60 platform, where tail drive shaft sections are supported by bearings at each end which are isolated from the airframe by a viscous bladder. Vibration from the
Figure 6: Gain Values for UH-60L Viscous Bearing Sensors As seen in Figure 6, gain for the viscous bearing sensors is distributed higher than gain for the remainder of the sensor suite. The vertical dashed red line is the threshold, so the next possible gain value of 128 would be considered bad data by this CI.
Data Quality Amplifier (DQA) Like gain, DQA is designed to detect excessively quiet signals which are indicative of a dead sensor. Most standard temperature accelerometers used in CBM systems are Integrated Electronics Piezo Electric (IEPE) type. The DQA algorithm is so named because it was initially thought that many IEPE sensors failed due to a bad internal amplifier. That presumption has not yet been confirmed, but DQA is effective at finding exceptionally quiet sensors. DQA is calculated by taking the reciprocal of the energy in the frequency domain spectrum. DQA is preferable to gain because thresholds can be set statistically and thresholded much more effectively. Gain, as computed at the ADC, must fall into a particular bin, the level at which the ADC will amplify. DQA has an infinite range of values, and a true distribution of values can be analyzed.
Peak to Peak (P2P) An alternate approach for detecting excessively low signals is to measure the peak to peak value (P2P) of the ATD signal. P2P is the difference between the highest and lowest values of the ATD signal. A threshold would be applied to P2P such that values above the limit would be considered normal, and values below the limit would be considered excessively quiet. Figure 7 shows P2P distributions from the same aircraft data used to illustrate gain.
blue distributions were identified as healthy. When the red and blue distributions are plotted together, they show significant overlap, demonstrating that the signals identified as “dead” by the gain indicator do in fact contain reasonable levels of activity. The points to the left of the dashed threshold are clearly separated from the rest of the data, and upon further investigation, these points clearly had dead signals. Accordingly, data graded against this P2P threshold provide a much better distinction between healthy and unhealthy DQ than gain indications do.
The red distributions are the data previously identified by gain as experiencing bad DQ, while the
Figure 7: P2P Values for UH-60L Viscous Bearing Sensors
High Range Utilization
CONCLUSION/RECOMMENDATIONS
Clipping
A standardized approach to DQ should be established, utilizing the knowledge base that the Army and the larger HUMS community has built. The final detailed process for DQ implementation will include a detailed survey of the CBM system’s possible DQ failure modes, rates, and implications. A FMECA-like framework should be used to classify the potential failures. Mitigating factors for all relevant DQ failure modes should be considered when proceeding with the steps to achieve oncondition status.
When the dynamic range of the sensor or ADC is exceeded, clipping may occur. Clipping can result in severe signal distortion, depending on the severity. Clipping is so named because a saturated accelerometer signal, or a saturated ADC, will “clip” off the portion of data over the limit, resulting in flattopped data. An FFT of a clipped signal will show significant frequency content that is not present in the unclipped signal. Clipped signals can be the result of incorrect ADC or sensor selection. Clipping can also occur when an impact is occurring directly on an accelerometer, such as a loose wiring bundle vibrating against the sensor. A general clipping algorithm simply calculates the number of all bins in the digitized signal with a value over a specified limit. The limit must be set to ensure that it corresponds with the lesser of the ADC’s or the sensor’s dynamic range. A high range utilization indication is an essential part of full data quality coverage. Other approaches, such as peak to peak with a thresholded upper bound, can be used effectively. The algorithm, or algorithms, must ensure that saturation is not occurring at either the sensor or the ADC. Detailed Process A comprehensive approach must be taken to ensure DQ. A wide variety of issues can cause the data to be unreliable. It may be necessary to take a FMECAlike approach to the CBM system, to identify all possible DQ failures and a mitigating factor for each. This will provide confidence that risk will truly be avoided when going to on-condition status. Many of the rigorous processes defined in ADS-79C for CI/HI validation, including ground truth sample sizes and seeded fault testing, can be directly leveraged to DQ. Accuracy rates to be recommended will need to reflect the increased importance of false positive DQ indications.
This standardized process could ideally be incorporated into ADS-79. The specific data quality algorithms presented above are not comprehensive, but are limited examples of the Army’s DQ knowledge base. Table 1 gives three algorithms which have been very successful and which cover a number of common DQ failure modes. Name DC Offset Data Quality Amplifier Clipping
DQ Driver Connector, wiring issues Dead sensor
Detection Method Presence of peaks < 3Hz Inverse of energy in frequency domain
Sensor, ADC saturation
Number of frequency bins over limit Table 1: Suggested minimum DQ suite
While Table 1 is a helpful place to start, a complete algorithm set can only be identified once a systematic approach has been applied, and all relevant DQ failure modes are known.
REFERENCES 1.
Aeronautical Design Standard 79C-HDBK, Army Materiel Command Standardization Office, AMSAM-RA-IM-IS-IL, Redstone Arsenal, AL, 2012.
2.
Antolick, L., Branning, J., Wade, D., and Dempsey, P., “Evaluation of Gear Condition Indicator Performance on Rotorcraft Fleet,” AHS 66th Annual Forum Proceedings, Phoenix, AZ, May 11-13, 2010.
