The Right Stuff FOR AGING ELECTRONICS/ INTERMITTENCE/NO FAULT FOUND
For those in the avionics repair and maintenance business, the acronyms NFF (no fault found) and CND (cannot duplicate) are, unfortunately, all-too-familiar terms. After several decades of frustration with this elusive phenomenon, it continues to consume an enormous amount of test and diagnostic effort and is the source of considerable cost and discomfort within the multi-level avionics repair model.
In this article, we will outline the problem of intermittence and its testing difficulties and, more importantly, describe the unique equipment and process which has produced overwhelming success in intermittence/NFF resolution and mean time between failure (MTBF) extension. Working with Total Quality Systems (TQS) of Ogden, Utah, we implemented our team-developed overhaul system called Intermittent Fault Detection and Isolation System (IFDIS) which incorporates all the necessary testing procedures and technological capabilities that are proving to be critical to the resolution of the chronic intermittent/NFF problem.
There are undoubtedly many causes of NFF and all of them should be addressed. The question is: where do we start and which solution will be the most beneficial?
Our particular efforts have focused on the literal or statistical analysis of NFF, recognizing that if the system’s MTBF has decreased, or if the device’s NFF rate has increased with age and deterioration, a physical fault is most likely present. If it isn’t found during conventional testing, then it probably only fails intermittently. Similarly, having an intermittent failure, in all probability, cannot be detected or diagnosed at testing time because of known and demonstrated limitations in the conventional measurement equipment used to perform the tests.
The Testing Problem
Intermittence occurs randomly in time, place, amplitude and duration. The very nature of this type of failure suggests that the ability to detect and further isolate the intermittence root cause is based on detection SENSITIVITY and PROBABILITY, rather than conventional methods concentrating on ohmic measurement accuracy. Simply put, you can’t detect an intermittent event until it occurs, and then you might have limited opportunities to catch it on the specific circuit when it does. Trying to measure fractions of a milliohm, scanning one circuit at a time, is ineffective for this particular failure mode.
Through extensive hands-on failure analysis and repair of rogue/bad actor/NFF avionics and other aging electronics, our research revealed that nearly all NFF failures are caused by underlying intermittence in the circuit path interconnections, not the electrical components. The electrical components generally fail “hard” and are, by comparison, easy to troubleshoot and repair. In contrast, the interconnecting devices mostly fail intermittently. These types of “devices” are defined as the connectors, crimps, splices, circuit board traces and solder joints, bulkhead connectors, backplanes, switches, circuit breakers, fuse receptacles, etc. In short, it is all the electromechanical devices that tie the circuit components together mechanically.
Just like machinery, these particular devices wear gradually, or contamination builds-up over a period of time. Rarely, unless damaged, will they be working perfectly one minute and the next become a repeatable, testable, hard failure. Instead, the electromechanical devices go into a long and frustrating period of low-level intermittency as their mechanical tolerances change, depending on their age, wear and the current environmental conditions such as temperature, humidity, vibration and contamination.
When a particular circuit device’s electromechanical intermittence reaches sufficient magnitude, its overall electrical function begins to malfunction, resulting in increasing intermittent-type system failures, which when subsequently tested on the ground in a static environment, may perform sufficiently well as to avoid detection.
It is important to note here that an intermittence of sufficient amplitude and duration to cause a system malfunction during extremes in the operating environment is likely to manifest itself at a much smaller amplitude and duration during ground-based testing, unless environmental stimulus is applied. The amount of stimulus required to expose the intermittence is inversely proportional to the sensitivity of the testing equipment used to detect the intermittence.
It’s at this point that NFFs circular logic and confusion begins. When a malfunction is reported but is no longer evident or easily detectable with conventional test equipment, the maintenance specialist has only two expedient diagnostic choices: the intermittence is either in the aircraft or in the box. It’s unlikely that the pilot imagined or fabricated the original in-flight malfunction. Consequently, line technicians are often left to simply take a “shotgun” approach to the repair in an attempt to address the original write-up in a timely manner. Unfortunately, by removing system elements prior to locating the root of the intermittence, the potential exists that the removal was not necessarily the problem. Suggestions that the maintenance specialist simply pulled the wrong item due to inadequate training, tech orders, inexperience, etc., somewhat ignore the original reported malfunction and ensures that the defect remains undetected somewhere in the system. If it positively is not in the box, then it’s more than likely still in the aircraft.
Since intermittence occurs primarily in electromechanical devices, when the “most likely” opportunity is calculated, the line replaceable unit (LRU) becomes the most prominent suspect. There are hundreds (and in many cases thousands) of potential failure points in a typical avionics box, whereas the aircraft circuits and connections leading into the box may be just a few hundred.
The Testing Solution
Once intermittent failure modes are clearly understood, it becomes quite evident why the vast array of conventional test equipment cannot test for or isolate the root cause of this elusive problem efficiently or effectively.
In a typical avionics system, there are thousands of internal and external circuit paths moving electrons through thousands more physical interconnection points which are all aging to some degree, and will fail intermittently long before they fail permanently. It only takes one of these devices reaching this condition to render the unit unreliable. It is virtually impossible to manually probe such a system — and even if attempted, the probability that you would be measuring that specific path at just the right moment, looking for the right signal, would be infinitesimal and futile.
By any reasonable scientific explanation of the problem, in order to catch intermittents on the ground, you need to have phenomenal testing speed (sensitivity) and a 100-percent bandwidth. In other words, the proper technology for the task must be able to test all of the failing system’s paths all of the time, and in a simultaneous and continuous fashion. Conventional test equipment does just the opposite. Most testing devices employ digital sampling and averaging techniques to achieve higher levels of parametric accuracy, which will completely “average” a short-duration, ohmic, intermittent event right out of existence. Likewise, virtually all continuity testing devices also employ scanning methodology — while they may be physically connected to each circuit, they still only measure one circuit at a time and then only briefly. A continuity test ONLY verifies that the unit under test is wired correctly and is stable at that specific moment. These devices are typically limited to measurement speeds in the 100–200 millisecond range which add up to some rather massive holes in intermittence test coverage when testing just a single line. Event detection is nearly impossible on all of the interconnections found in typical avionics systems.
In order to address these testing limitations, the intermittent fault detector (IFD) was developed specifically with intermittence requirements in mind. It uses super-sensitive analog detection technology on the front end and digital reporting and data processing technology on the backend, and it does it all in an efficient, parallel circuitry manner. The IFD consistently detects any intermittent circuit event on any circuit simultaneously, at ohmic glitch durations as short as 50 nanoseconds. The numbers of simultaneous test points are scalable from 256 up to 16,384.
What does this mean in the overall scope of intermittence detection probabilities? It means everything! It means success or failure, reliability or unreliability, integrity of a test or no integrity whatsoever.
While certainly not comparing ourselves to Albert Einstein, his formula, E=MC2, which explained the force unleashed by the atomic bomb, is very similar to the probability gains derived from the IFD technology to catch random intermittents. In order to explain and demonstrate this enhanced capability in a system of simultaneous circuit paths under test we use a similar formula that we affectionately, with respect to Einstein, call:
Universal Synaptics’ Law of Intermittent Fault Detection Effectiveness:
In our formula, E is the Effectiveness that the IFD technology provides in detecting the most evasive of intermittent malfunctions (those causing NFF) in a given unit under test (UUT) device versus any other comparable piece of test equipment (measured in a ratio:1).
S is the single circuit intermittence detection Speed advantage that the IFD has over the single circuit intermittent detection speed capability of any comparable testing technology… for the IFD, use 50ns, 50 nanoseconds, 0.00000005 seconds.
Simply stated, what is the ratio of the shortest glitch detectable by any two pieces of test equipment on just a single circuit?
Example: 100us divided by 50ns = 2000:1 or 100ms divided by 50ns = 2,000,000:1.
C is the number of Circuits in the device that require simultaneous testing and this value is squared.
Note: The number one question that arises when explaining and using the IFD probability formula is:
“Why do you square the number of circuits to be tested (C) in the comparison formula?”
Since this is the key to the entire solution, let’s take a moment to fully understand it.
The reason the number of circuits under test is squared is that while every other single point or scanning-type testers are measuring one circuit at a time, the IFD is simultaneously testing all of the other circuits at the same time, for the same duration. As the conventional scanning continuity tester moves on to test a new circuit, the IFD continues to test all the other connected circuits at the same time and for the same period.
Intermittence by its very definition is random in time, place, amplitude and duration. Therefore, the detection of intermittence is a condition of probabilities and the ability to detect it is measured in test coverage.
The following is a simple explanation of the squaring effect of simultaneous and continuous testing for intermittence (see Table 1)
Using an easy example of a three-by-three matrix of circuits (nine total circuits to be tested,) like a simple nine-pin cable, let’s compare. Conventional scanning test equipment, while physically connected to all the circuits, still only measures one circuit at a time. While this technology might measure test point one for one second, the IFD’s all-lines, all-the-time technology, simultaneously and continuously tests all nine of the circuits for that same one second, for nine total seconds of intermittence test coverage. When conventional equipment then moves (scans) to measure test point two, also for one second, the IFD tests all nine circuits for another second, giving you nine more seconds of intermittence test coverage. Conventional equipment then moves on to test point three for one second, and the IFD again tests all nine circuits for that same one second. When conventional testers have finally completed testing each of the nine circuits for the one second each (nine seconds total), the IFD has just simultaneously tested all nine circuits for nine seconds each, or 81 total seconds.
It doesn’t matter if you have a nine-pin cable or a 10,000-test-point avionics box, with the IFD’s simultaneous and continuous test technology, you effectively square the number of circuits in total test coverage.
In fact, the IFD’s test coverage is actually even better than this. Scanning continuity testers take valuable test time to switch to the next circuit and in order to see ohmic glitches (rather than complete opens), they must also charge the new line sufficiently to see an ohmic change. Meanwhile, the IFD dutifully keeps watch for ohmic events on any line for the entire duration.
Using this simple-to-calculate formula (E=SC2) for test coverage or probability gain of the IFD technology, you can begin to see why the IFD works and other technologies simply don’t.
For example, consider a state-of-the-art scanning continuity tester that claims to test continuity at the rate of 3,500 test points per minute. The single-circuit intermittent discontinuity detection speed could then be computed to be approximately 17ms (0.017 seconds) (60/3500).
If you were testing just one wire or circuit, then the IFD at 50ns (nanoseconds) calculates to be 340,000 times more sensitive at catching intermittence on a single circuit.
S= 0.017 divided by 0.00000005 = 340,000 times more likely to detect NFF intermittence on a single circuit.
Now, take a 100-circuit chassis or cable.
Using the formula E=SC2:
E = 340,000 x 100 x 100 = 3,400,000,000
In this example, the IFD is 3.4 billion times more sensitive than the scanning continuity tester for detecting intermittent/NFF at 50ns on a 100-circuit chassis or cable.
Next, take a 1,000-test-point coverage requirement, such as the modular low-power radio frequency (MLPRF) LRU chassis in the AN/APG-68 radar used on the F-16 Fighting Falcon:
E = 340,000 x 1,000 x 1,000 = 340,000,000,000
In this example, the IFD is 340 billion times more sensitive than the scanning continuity tester for detecting intermittent/NFF at 50ns on a 1,000-circuit chassis or cable.
The demonstrated advantages in detection probability are why IFD technology is actively reducing the intermittent/NFF problem down to a five-minute test in a typical avionics system as outlined above. The simple-to-compute metrics also show conclusively why IFD technology works so well for resolving the intermittent/NFF problem. This technology sees real intermittent circuit occurrences that conventional test equipment cannot see and was not designed to detect. Given this “explosion” in test coverage, it becomes crystal clear why the IFD is the only applicable technology designed specifically for, and capable of, detecting, resolving, and gauging the overall problem and levels of intermittent/NFF.
IFDIS is the Right Stuff
IFDIS can best be described as a three-pillared approach to resolving NFF/intermittence.
These pillars consist of:
1.) The implementation and use of serialized data tracking to identify bad actors and repeat offender problems by aircraft and LRUs.
2.) The application of light environmental stimuli to duplicate the operational environment and rapidly expose even the “lowest amplitude and shortest duration” intermittent circuits during test time.
3) The use of precise intermittence testing technology. IFD developed by Universal Synaptics Corp. are specifically designed to detect and isolate the underlying intermittent causes at levels of sensitivity and probability never before possible, as well as form and fit interface test adaptation (ITA) to ensure that all of the potential failing circuit interconnects in the suspect devices are all tested simultaneously and continuously while closely simulating the aircrafts operational environment.
Last, but not least, IFDIS was voted as the “Great Ideas” winner at the Department of Defense Maintenance Symposium 2010 and the Tibbett’s Award Winner 2011, given by the U.S. Small Business Administration for advanced technological innovation.
Ken Anderson is Vice President of Sales & Business Development for Universal Synaptics Corporation, an industry leader in detecting and isolating intermittent faults in avionics systems. Anderson has 15 years of sales, marketing and business development experience.