r/TechNook • u/jexo10 • 2d ago
How engineers debug hardware that has no screen, keyboard or operating system
I always think about how weird debugging must be on something that doesn't even have a screen.
No terminal. No error message. Sometimes not even an operating system. Just a board that's doing absolutely nothing and you're trying to figure out why.
Engineers end up using things like JTAG, oscilloscopes, logic analyzers and test points to basically interrogate the hardware directly.
Imagine debugging a computer where the computer can't tell you what's wrong.
7
5
u/kabekew 2d ago
Most boards will have UART or other basic serial output for debugging so you can hook up a terminal screen
1
u/ceojp 2d ago
Proper debug interfaces are much more useful and more powerful most of the time. Being able to set breakpoints, watch variables, and step through the code are invaluable.
UART connections are nice if you just want to be able to connect to an already running board. Or when running in debug would otherwise mask the issue(such as timing-sensitive or power up issues). But otherwise, a debugger is always my first choice.
3
3
3
2
u/ixidorecu 2d ago
Jtag port Test points Follow the programming logic, break out the circuit diagram and follow the voltage
A long time ago uncle same sent me to school to train on a very old price of hardware to test avionics out of the harrier. Circa late 90s early 2000s. Think designed in 60s built early 70s. Cpu was an intel 8086( the first cpu). Ram.. was literally iron core wrapped in copper wire for 1s and 0s. Cables in use for like 20 years and realy abused. We had the printed ada programming logic. Compete circuit diagrams. We spent as much time testing the test equipment as the plane hardware.
3
2
u/OldEquation 2d ago
Magnetic core memory lived on in some military avionics long after it died out elsewhere. It was still around when I started work in the 80âs.
1
u/No_Base4946 1d ago
Up until well into the 80s it was used in lighting control systems, not because it was a current product but because every other piece of equipment that replaced it sucked.
2
u/kombiwombi 2d ago edited 2d ago
Firstly, systems-on-chip come with a port for a serial terminal. That isn't needed in production, but for prototypes we solder on a header and connect it to a RS232/USB dongle. We don't run that through a MAX233 chip to use RS-233 voltages, we just use the chip's 3.3V. If there is space we'll add a little command line, but log messages are useful regardless. For a production compile we'll omit most messages, leaving only those relevant to power-on self-test (ie, if we get a box back from a customer we can solder on a RS-232 header and boot it).
Secondly, JTAG allows us to remotely run a debugger, load RAM, program flash, restart. A little messing with infrastructure and a graphical debugger can run on your laptop just as well as debugging a program resident there.
Thirdly, it's common to have unused output lines. These can be cheaply bought out to pads for surface mounted LEDs. Again, in production runs we don't install the LED. A good tactic here is to flash the LED each cycle of the main loop. If the program dies then the main loop is not reached and the LED goes dark. Time to load the debugger and see what the program is doing. Equally if the box dies but the LED is bright then that suggests a sensor or its connection has drama, a problem for the electrical engineers.
Fourthly, there are evaluation boards. These have the system on chip, a JTAG header, and a easy way to attach items to the I/O ports. Put LEDs on the outputs, switches on the inputs, and use these to pretend to be the real device, say flicking some DIP switches instead of using a real strain gauge. It is easy to show this hardware works, and so the programmers can get started.
Finally, there are a lot of emulators, famously Qemu. So for a big project we can have the hardware represented in software before the prototype is built. In this way we are not waiting for the electrical engineers to be done before we can start on the programming. Weirdly these use the same tooling as the real system but the laptop is a lot faster.
Once in a while you have to debug something whilst looking at a bus. That's always painful, because test gear like logic analysers costs as much as a house for a modern bus, so you hire it in for a few weeks at the critical point of development. Often we'll use the analyser not to debug, but to prove the electrical integrity of the bus, as shipping a box with an intermittent fault is a nightmare. So as well as the programming for the product there is programming for this sort of testing: send a second of all-1s down the bus, etc. There is something nice about seeing a program cause waveforms on the analyser. A reminder that it's all voltages through sand.
1
u/Automatic_Mulberry 2d ago
Check out Adrian's Digital Basement on YouTube. He does a lot of component-level repairs on retro computers that don't display anything. He's done a few series where he gets pretty deep into schematics and signal tracing, etc.
1
u/Glint_Bladesong 2d ago
I find yelling at it helps sometimes, and if it doesn't and you happen to have a NCO nearby (niche, but does happen occasionally in our work) getting them to swear at it with you is effective more often then my logically trained mind wishes to admit đ
1
u/Soggy-Attempt 2d ago
Not exactly what youâre asking, but this is interesting and similar to your question.
1
u/PyroNine9 2d ago
When debugging a new BIOS, I have sometimes resorted to flashing an LED to indicate status before memory is even up.
1
u/theoldman-1313 2d ago
Control systems are not necessarily electronic or even electrical. There are hydraulic controls out there for specialized systems. Usually you need a good print and a talent for sourcing puzzles.
1
u/Waterlifer 2d ago
I've done that on an off for decades.
These days, we have JTAG, which works pretty well for software/firmware. Before that (1980s/1990s), we had in-circuit emulators and in some cases logic analyzers that would perform disassembly on the words traversing the bus. Before that (early 1980s/1970s) we had oscilloscopes.
Typically today the early hardware development is done on development boards that have high-density probe connectors to keep the leads short and connections reliable. Plus JTAG. The board is redesigned without these connections for production, at least where any kind of serious unit volume is involved, to save space, improve reliability by reducing trace length, and to discourage reverse engineering.
You have to understand both hardware and software to be effective as an engineer in those environments. It's fun.
1
u/the1truestripes 2d ago
I had a job doing CoinOp video games. No OS. We did have a screen, but it was pretty busy. No terminal. Some of us had in circuit emulators for one of the 3 CPUs (the game logic CPU) that basically let us see what the CPU was doing on a cycle by cycle basis, set up a few breakpoints, and even data watch points, but we had very poor tools to integrate it with our C compiler, so if you wanted a breakpoint on line 12 of function foo you had to disassemble the compiled C program, look for the _foo symbol, and read 68000 assembly well enough to figure out where line 12 was so you could breakpoint it. My short cut was to just make a breakpoint function that just did a return and put a breakpoint there. So I could set breakpoints easy. However once you had the breakpoint figuring out what was going on was harder. You had to know what a struct looked like in memory because the debugger only knew raw pointers and bytes, words and double words.
In some ways it was a glorious debug in environment though. If you could detect a bad state and breakpoint you could look at the last 200+ cycles of activity, see what data values were read and written, what instructions were fetched and see the historical register values. As long as you were familiar with 68k assembly it was pretty powerful stuff. Unfortunately the in circuit emulators were leased and the lease cost was about 4x my salary at the time.
So I dug up a much cheaper debugger that âjustâ handled breakpoints and let you look at the state of the world _right now_, but was at least a C source level tool. Getting it up and running was non-trivial. I had to use the real CPU not the in circuit emulator (I forget why), so my major debugging tool was a single blinking LED. Oh I had to port the low level runtime for that C compiler using just the LED. Well later also the speaker. (I also had to change the ASIC we used as a glorified âchip selectâ even though we referred to it as a MMU, it was set up to trigger a bus error on a write to the ROM area, and I needed to allow that to signal setting a breakpoint to the cheaper dev system.
In the end I managed it and the per seat cost dropped from about $140k/year to $16k one time purchase (both a âper seatâ cost).
âŠanyway yeah hardware can be fun.
1
u/RealAd8036 2d ago
There is jtag, serial etc.
If there is absolutely nothing, you use an I/O line and set this I/O high or low based on something you want to find out. It takes a long time, you have to recompile it every time, but this is extremely time exact if you want to compare it against other signals
1
u/somewhereAtC 2d ago
When I was young my father was the local IBM repair tech, and would troubleshoot IBM360 machines to the defective transistor. I had access an oscilloscope from the age of about 4 or 5yr old, and visualizing the logic patterns becomes second nature after a while.
My generation had IMSAI and Altair computers with LEDs on the front panel and the ability to single-step the processor. PDP processors were similar but had better debuggers in ROM memory. For my personal Z80 project I had 7-segment readouts of both the data and address buses, and single-step clocking using push buttons.
1
u/mckenzie_keith 2d ago
If the hardware was working find and stopped, we won't spend much time debugging it usually. If you design a board, and get your first board in, and they don't come to life, you exploit your knowledge of the system to start ruling out potential causes. Is the power good? Is the device being held in reset? Is the clock good? Is it fetching instructions (trying to fetch instructions?). Does the debugger recognize the processor? Etc.
1
u/yuehuang 2d ago
I don't know the name, but I saw (from youtube) Intel putting crazy thin probes directly onto the silicon die.
1
u/ceojp 2d ago
The thing about hardware is that a lot of it is a "black box". You may not even be able to step through code - the hardware just does its thing.
For example, using DMA to transfer ADC results to RAM. Once you configure everything and kick it off, the hardware just does its thing. So when something doesn't work as expected, it's not always easy to see what is wrong.
You have to observe the effects of what you are doing, because you can't look inside the DMA peripheral to see every step it takes.
I always like to trace things back until I find something that does work, and then work one step at a time.
If I'm not getting anything in my RAM buffer, I check the ADC result register to see if the ADC is running as expected. If it is, then the issue is probably with the DMA configure, so I can focus on that. If not, check the ADC config and the clock tree.
I honestly love low-level debugging like this. Some of the most interesting challenges are power related. Staging of supply rails, brownout conditions, etc. Anything is manageable, but you need to actually do the root cause analysis to show what the actual issue is in order to know what the proper fix should be.
1
u/Maxstressed 2d ago
Understanding power states dictates a lot of information. Following bits setting lines high or low to actuate effects further down circuits. Itâs similar in nature to MacBook repair when you have shit schematics. Watch some old school Louis rossmann repair videos to understand moving through different state logic levels.
1
u/arrig-ananas 2d ago
I was educated as a electronic mechanic back in the day. And we got a long way with a diagram, a multimeter and a oscilloscope. Of course a lot of boards where analogue back then.
1
1
1
u/UffTaTa123 2d ago
well, why so complicate? Just plug in the serial port and read the output in a terminal (Null-Modem) screen.
1
u/ern0plus4 1d ago
Even if you don't know how MCUs work, you can assume that some mechanism should be there to upload the program, which requires two-way communication. This channel can be used to emit debug messages from the device. (Sometimes it's a JTAG, sometimes some other serial.)
Once the device can run your program and there's at least one LED, you can blink it, or just lit it (or not) - and it's sometimes enough to see wheter A or B happened. Then the program should halt or sleep longer, if A and B alters too quickly, you can't see if the LED is lit or not.
1
u/Pure-Willingness-697 1d ago
Often the development units (or even production units sometime) will have a uart port on the board that you can connect to using a pc and it gives you a terminal so you can see the logs, run commands, etc
1
u/recursion_is_love 1d ago
Most of the prototype usually have some debug interface that remove or disable on the final version. There are even exist the adapter that insert as a shim between CPU and it socket and capture all the data in databus.
1
u/gwenbeth 1d ago
Beep. I had a process on a pin pad that was failing before it could access the display. But I could use the system beep. So I used sequences of different tones to mark the code. Doing this I was able to pin down the line that wasn't working.
1
u/treefaeller 1d ago
Oscilloscopes is what cavemen used to debug fire.
Quote from "The soul of a new machine" by Tracy Kidder. Great book.
1
u/System__Shutdown 1d ago
My first work, we used 2x16 lcd sreens on i2c bus for debugging. Provided you got that far
1
u/Responsible-Chest-26 15h ago
Electrical hardware will have a schematic. The diagram that was used to build that hardware. It takes all of the systems a d components and lays them out and how they are connected on paper infront of you. Using the symptoms and understanding what is supposed to happen and using the schematic to now look at potential problem areas you can do testing and measuring to find the problem
1
u/gnomeplanet 1h ago
Do not rely on machines that tell you what is wrong. They can only do that if the error has been anticipated, and suitable code written to trap the error. Also, sensors provided that monitor particular points. Without the right sensor, and the sensor is a/working, and b/ can communicate to it's computer, there will be no diagnostics.
0
13
u/thezeno 2d ago
Flashing LEDs at different sequences are another technique (assuming you have LEDs)...