A board that fails once every few hours, or once a month, is not “occasionally broken.” It’s failing under a specific combination of conditions that your test procedure doesn’t reproduce. Standard testing runs at nominal: nominal voltage, nominal temperature, nominal load, one pass, pass/fail. An intermittent fault doesn’t live at nominal. It lives in a corner of the operating envelope that nominal testing never visits, and confirming that much can already take days or weeks of undirected troubleshooting.
What standard testing misses
Start with margins. A part can pass every functional test at 25°C and still drop out intermittently at 45°C ambient, combined with a decoupling network that was always marginal and a signal integrity budget that had no headroom to begin with. None of that shows up when the test runs once, at nominal, in a controlled room.
Then there’s cumulative effect. A system validated over an eight-hour test run behaves differently after three weeks of continuous operation, and on 24/7 embedded systems this is where a large share of “unexplained” faults actually live. Thermal cycling fatigues solder joints and connectors in ways a single thermal chamber run doesn’t catch. Firmware that leaks a few kilobytes per hour, or fragments a heap slowly enough to pass any short test, can take days to turn into a visible hang. Flash wear, electrolytic capacitor drift, connector oxidation: none of it is present on day one, and none of it is present in a test that resets state every cycle.
Field conditions evolve too. The installation environment on day one of deployment is not the installation environment eighteen months later. Ambient temperature swings seasonally. Local EMI changes when a variable-frequency drive gets installed on the same line. Power quality degrades as the site’s electrical infrastructure ages or gets modified around the product. None of this exists at design time, so none of it exists in the design review either.
The factor that gets missed most often is usage itself. How an operator actually runs a device six months into deployment is rarely how it was used, or assumed to be used, when it shipped. Duty cycles shift, workflows adapt, features get combined in sequences nobody tested together. The failure report says “it broke”, but what actually changed is what the system was being asked to do, and that shift is invisible from a datasheet, invisible from a support ticket, and usually invisible to the operator too. Nobody flags a change that, from their side, looks like normal use.
Systematic, not lucky
None of this surfaces by repeating the same test and hoping the fault shows up again. It surfaces by treating the fault as an unknown variable: isolate one factor, change it deliberately, observe, record, move to the next. One variable, one combination of conditions, one clean result, every time. Not three changes stacked together to see if the problem “goes away”.
That discipline sounds obvious on paper but it seldom survives a deadline. Under pressure, the instinct is to swap the likely-looking component, retest for an hour, and move on if the fault doesn’t reappear. That’s not a fix, it’s an educated guess with a short observation window. If the fault comes back three weeks later, the team is back to zero, minus the original failure conditions and plus whatever the swap changed in the meantime.
Traceability carries as much weight as the test plan. Every stimulus applied and every result observed gets logged: parameter, value, outcome, position in the sequence. Not for paperwork. Because six hours into a diagnostic session, “did I already test this at this temperature with this load” is a question that needs a record, not a memory.
Capture and correlation
The other half of the problem is data. A fault lasting forty microseconds and appearing once every forty minutes doesn’t wait for anyone to be watching. What actually closes cases like this is correlating hardware, firmware and software data against the same timeline: a scope capture on a specific rail, lined up against a firmware log timestamp, lined up against what the application layer was doing at that exact moment. Each source read separately gives three disconnected stories. Correlated, they give the sequence that produced the fault. Correlation, an rather common word that hides a huge work behind the scenes.
Standard instrumentation covers most of this. It doesn’t cover all of it. On a fault that has already resisted the usual toolkit, the only way to get enough context around the failure window is sometimes a custom acquisition setup built for that specific application: a trigger condition tied to an unusual signal, a logger tapped into a bus the product was never designed to expose, a fixture built for a board with no test points where they’re actually needed. That’s not overengineering, it’s what’s left once the standard approach has already been tried and has already failed to produce the data that explains anything.
This is also the discipline where debugging and cybersecurity domains overlap, because the techniques used to probe, trace, log and analyse data are almost the same, with a different scope.
The report is part of the work
Finding the fault isn’t the end of it. A diagnosis that isn’t documented is a diagnosis that gets rediscovered the next time the same failure mode shows up on a different unit, or the next time someone else on the team runs into it with nothing to check against. Root cause, contributing conditions, what was ruled out and why: that belongs in the report as standard practice, not as something billed separately if the client remembers to ask.
What this changes
None of this is about owning better instruments than the next lab. It’s about not skipping steps on a fault that refuses to reproduce on demand. Symptom chasing produces a component swap and a fault that comes back later. Diagnosis produces the actual cause, documented, so it stays fixed.
If a fault has already survived a round or two of internal troubleshooting, that’s usually the point where the standard approach has run out, not where more time on the same approach helps.
If that’s where you are right now, Hard Probe Debug is where to start.
