IOActive Labs: Probing and Signal Integrity Fundamentals for the Hardware Hacker

by Andrew D. Zonenberg, Ph.D
Associate Principal Security Consultant

The latest new widget just showed up on your desk. You excitedly crack open the case, look around a bit, and find a signal that looks interesting. You fire up your oscilloscope, touch a probe to the signal, and... the widget won't boot! Or maybe it works fine, but you see garbage on the scope screen that looks nothing like a useful digital waveform.

It's a problem that's becoming all too familiar to hardware hackers. As technology advances, signals become faster, which makes them more sensitive and less tolerant to the sloppy wiring and probing techniques commonly used in reverse engineering. Even cheap SPI flash can run at 100+ MHz in modern designs.

This article will focus on practical problems and solutions for analyzing existing boards, without digging too deep into the more advanced electrical engineering theory needed for design of high speed logic. For a more in-depth treatment of the subject, High Speed Digital Design: A Handbook of Black Magic (1993) by Howard Johnson & Martin Graham is an excellent read, as well as its successor High Speed Signal Propagation: Advanced Black Magic (2003).

If your probing setups look like this, you're going to have a bad time reversing modern hardware!

At this level, the main thing to be aware of is that electrical signals move through wires at a finite speed, typically around 0.6-0.7x the speed of light (roughly six inches per nanosecond), and that changing cable characteristics (more specifically, impedance) causes an impact on the signal whose duration in time is proportional to their length.

Before we proceed any further I'd also like to introduce an abbreviation from the test equipment world that you may not be familiar with: DUT (Device Under Test). This is a fancy way of saying "the thing you're trying to probe".

Avoid Long Messes of Wires

In the photo above, a JTAG adapter was connected to the DUT with several feet of wire, without paying any attention to impedance or termination. The JTAG software was unable to detect the processor. When an oscilloscope was connected to the TCK signal on the DUT, the signal looked like this:

JTAG TCK signal with long wires

The rising edge of the clock signal climbs for a while, then dips, then continues climbing to its full value. These non-monotonic edges are a big problem! If the position of the dip happens to line up with the threshold of the device's input pin, it will see two rising edges instead of one, resulting in a bit of data being duplicated.

You might be tempted to reduce the clock frequency of the JTAG adapter in this case, but this won't help: all it will do is move the rising and falling edges further apart without changing their shape. Fast edges at 1 kHz are just as vulnerable as edges at many MHz.

Adding filtering to slow down the edge rate (at the cost of reducing the max operating speed), adding a ~33Ω series terminating resistor at the source end of the wire, or playing with ground geometry to match the impedance of the cable more precisely can all help. In most cases, however, the easiest solution is simply to shorten the wires.

No more spaghetti!

JTAG signal with shorter wires

If you look closely a small defect on the edge is still visible since the ribbon cable isn't impedance matched, but it's much smaller because the mismatched cable length is shorter. At this edge rate, this sort of defect shouldn't be a problem since it doesn't dip, it just stops rising momentarily. When dealing with sharper edges, you might want to shorten the cable even further. You should also...

Properly Secure Wires

This isn't strictly a signal integrity issue, but "my wires ripped off the DUT" will certainly lead to difficulty seeing the intended signal! Properly securing wires and probes is something overlooked far too often.

Don't rely on solder joints to provide mechanical support for a cable, which can be subject to bending and pulling forces as the DUT and cable move around your bench. If you're lucky the solder joint will fail and the probe will cleanly separate from the DUT. Worse yet, you might rip the copper pads/traces off the DUT or ruin your probe.

There are plenty of different methods of securing wires, so pick one that suits your specific scenario. I prefer to secure the wire first, then solder it, in order to keep it from flailing around while I'm soldering.

Hot glue: Cheap and easy, but hard to do neatly and difficult to rework if you get it somewhere you didn't intend. Requires rapid work to get things positioned right before it hardens and leaves annoying "strings" everywhere.

Kapton tape: Easily removable, but fairly low holding strength. You can buy it on big spools but I prefer using pre-cut dots to save lab time. Name-brand tapes generally have much better adhesive than cheap knockoffs.


Kapton tape securing a solder-in probe tip

Epoxy: Very strong, but almost impossible to remove once cured. Good choice for permanently attaching to something you plan to keep around the lab for an extended period of time.

UV cured wire tacking glue: My personal favorite (I'm a fan of Dymax #9-911, but there's many other options). It cleans up easily with alcohol in the uncured state, and solidifies in a minute or two under a cheap gel nail polish curing lamp. Most of these glues have intermediate holding strength: the wire is unlikely to come free by accident, but careful prying with a knife blade will cleanly separate it from the DUT without damage.


Wires secured with UV glue

Probe holders: There's a wide range of these available, ranging from DIY 3D printed bipods to $500+ three-axis micrometer jigs. More temporary in nature than any kind of adhesive, but useful for any time you want to look at more than a handful of signals while keeping your hands free. These get the probe tip right up against the board without any additional wire, which is great for probing faster signals.

Probe held in a bipod positioner

Avoid Long Probe Grounds

Yes,I'm talking about that big "tail" on your scope probe with an alligator clip at the end. Forget about using it on anything fast. It's very inductive, which when combined with the capacitance of a typical passive probe creates resonances.

10 MHz clock seen through a 10MΩ probe with a long ground wire

In the 10 MHz clock seen above, the ringing isn't enough to completely destroy the shape of the signal, but it doesn't take much imagination to see that with anything a little bit faster, the signal would be completely unreadable.

In the next few examples, we'll be looking at a 1.25 Gbps Ethernet signal. The "idle" sequence between packets contains a nice mix of fast 0-1-0-1 transitions and runs of several 0 or 1 bits, providing a nice idea of what a data bit at different rates might look like through a given probing setup.

Test signal with no probe

In this screenshot we see three different signals:

Blue: The signal as seen through the probe (flatlined, since the probe isn't on the board yet).
Pink: The signal internal to the DUT
Yellow: A snapshot of the pink waveform frozen in time. Right now they're identical, but if the probe alters behavior of the DUT you'll see them deviate.

To start, let's use a standard 10MΩ passive scope probe (a Teledyne LeCroy PP022) and an alligator clip ground.

Probing the easy, obvious way.

And here's what the scope sees

The signal is completely unrecognizable through the probe! In addition to causing ringing on edges, the inductance of the long ground lead destroys high frequency performance.

A spring ground is a bit trickier to use since you have to find a good ground point fairly close to the signal you're probing on the DUT, but gives a much nicer looking view of the signal.

Using a spring ground

Waveform seen with the spring ground

That's a lot better looking: the edges are rounded off and not as square looking as they should be, but that's an inherent limitation of this probe (500 MHz bandwidth). Individual data bits are clearly readable, so all is well... right?

Wrong. Take a look at the signal on the DUT (pink waveform)! There's huge dips in it. This might well be enough to corrupt data and render the DUT nonfunctional. This brings us to our next topic...

Beware of Probe Loading

You might be inclined to look at the "10MΩ" marking on a scope probe and assume that it imposes infinitesimal load on the DUT. That may be true at DC, but not at frequencies typical of modern electronics.

The problem is that typical 10MΩ scope probes also have on the order of 10 pF of capacitance. When a high frequency signal is applied to a capacitor it acts somewhat like a resistor, with an effective resistance of 1/(2*pi*f*C) for frequency of f Hz and capacitance of C farads. So the actual load the probe imposes is around 159Ω at 100 MHz and 31Ω at 500 MHz - a far cry from the 10MΩ seen by a DC signal! Since this loading is frequency dependent it will distort the signal as well as weakening it, potentially rendering it unreadable to the intended destination on the DUT.

Unfortunately, there's no trivial way to work around this issue using a standard passive scope probe, but there are several different types of probe designed for higher speed use which can help.

Active probes, which contain an amplifier in the probe head, can have much lower capacitive loading. A typical 1.5 GHz active probe might have an input capacitance of 0.9 pF, making it much less invasive. The price you pay for this is very expensive hardware - active probe price tags start in the low four figures (USD) and go up from there.

For looking at typical digital signals, a good low-cost option is the transmission line probe. This has a low-value resistor, typically 450Ω, at the tip and connects via coaxial cable to an oscilloscope input terminated with a 50Ω resistor to ground, for a total of 500Ω of loading as seen by the DUT. (Although this may present a problem when looking at signals with pull-up resistors, these are typically low enough speeds that a conventional 10MΩ probe can be used instead.)

Probing our test setup using a transmission line probe (Pico TA061)

Waveform seen through the transmission line probe

The benefit of a transmission line probe is that the input load is, in theory, purely resistive. Although there's always trace amounts of "parasitic" capacitance due to the physics of having the ground a finite distance away from the input, it's normally a fraction of a picofarad, which results in a high speed signal on the DUT being loaded much less heavily. As if that wasn't good enough this lower capacitance means that a high-inductance ground, like the standard alligator clip lead, will be much less likely to resonate - allowing this poor quality (but convenient) grounding method to be used with much faster signals.

In this screenshot the signal is clearly intelligible despite the long ground, and while the waveform as seen by the DUT has measurable loading it's far less severe than with the 10MΩ probe.

Waveform seen through transmission line probe with spring ground

Unsurprisingly, switching the transmission line probe to a spring ground improves the signal quality further - however, some degradation of the signal on the DUT is also visible. This is caused by the ~2 pF of parasitic capacitance of the probe. (The higher inductance ground lead concealed this loading in the previous test.)

If you don't want to spend $329 for a commercially made entry level transmission line probe (or $1K+ for a higher end transmission line probe like the Teledyne LeCroy PP066 or PicoConnect 900 series), you may wish to consider building your own. Simply solder a 450Ω resistor to the signal on the DUT you wish to probe, then solder the center terminal of a piece of 50Ω coax to the other end of the resistor and solder the ground braid to any convenient ground point nearby on the DUT. Attach the other end of the coax to your scope and you're in business.

Squeezing maximum performance out of this DIY probe design requires careful engineering, but building something that outperforms a standard 10MΩ passive probe for instrumenting fast digital logic isn't difficult. If you don't want to spend the time tweaking it, you might also want to consider...

Open Hardware High-Speed Probes

For the past year or two I've been working on two open-hardware transmission line probes, the AKL-PT1 (handheld, 6 GHz bandwidth) and AKL-PT2 (solder-in, 4 GHz bandwidth).

These probes target the OSHPark 4-layer and flex board stackups respectively. They're still being fine tuned but already deliver very nice results (outperforming every <$2K probe I've tested them against) so feel free to download the KiCAD files and try building one!

There will probably also be a group buy of assembled units with nice plastic enclosures and proper impedance control at some point in the near future once I'm finally satisfied with their performance.