The complete example program:

Here’s a screenshot of the MegunoLink Pro monitor window after the program has been running for a minute or two showing the program output:

x2scrap2

 

You can see it has saved 9 crash reports so far, and the next one will be saved at location 9 (of 10). The interesting part, the location in our program where the lockup occurred, is included too. For crash 8, the program was executing instructions at program memory address 0x398. For tracking down the offending line of code in the program, we need the byte address though. In this case: 0x730 (the byte address is simply twice the program memory address).

The hunting of the source code

By itself, the program memory address is not very useful. It isn’t a line number of file name; it is simply the location in program memory of the instruction that the Arduino was executing when the lock-up occurred. To get from the program memory address, to the offending line of source code we need to use the disassembler. To upload your program to an Arduino, the IDE first compiles the code in each file into instructions the microcontroller can understand—assembly code—creating a separate object file for each source file. A linker joins these together into an executable and linking format, or ELF, file. The ELF file contains the instructions at each program address and the disassembler lets us connect those addresses back to the original program code using this magical incantation:

Here, CrashTracking.elf is the linker output for the crashing program and avr-objdump is part of the Arduino installation that you’ll find in “arduino\hardware\tools\avr\bin”. You’ll need to add this folder to your system path to run the object dump program. Running this command will create a text file named “Disassembly.txt”. The disassembly file contains the original source code mixed in with the assembly instructions, as in the excerpt below. Lines with assembly instructions start with the program counter address. Typically, you’ll see a line of source code followed by the assembly instructions that the microcontroller executes to implement the source line.

So to find the offending lock-up, search through the disassembly until you find an assembly line with the same address as the byte address in the application monitor output. In this case, the lock-up occurs at address 0x730, or line 13 in the listing above. Ah-ha! The problem is obvious: address 0x730 simply jumps back to itself in an endless loop that will end only with the collapse of the Universe (or the power plug is pulled, whichever comes first). But you don’t need to read the assembly code to make use of the application monitor. For line 14 and 15 have the corresponding program code (sometimes the program code comes after the program address): an endless while loop keeping the Arduino busy doing absolutely nothing. Remove this line and the program will cheerfully, like all soothsayers, predict a doom that never arises.

The end (of all bugs)

Okay, perhaps this won’t track down every bug in your program, but if you think this will be a useful tool in your quest for the perfect program, post a comment below. You can download the Arduino lockup monitor from github. And download MegunoLink Pro too. Once you’ve fixed the bugs in your program, you can use MegunoLink Pro to monitor, plot or log your data in real-time or build a simple user interface to control your Arduino program from your PC.

 

Recommended Posts
Showing 25 comments
  • Christophe
    Reply

    Great stuff, thanks!

  • Carlos
    Reply

    Very good recommendations. I inserted it in my home automation. Thanks!

  • Emre
    Reply

    I’m using Due SAM3X on 1.6.7 ide and I have a bug which made me crazy for days… I’ve found this very helpful topic but when I try to implement the files, I get the error while compiling…

    In file included from F:\# All docs ever\Documents\# Arduino\AirPlatform_GND_V4.3-bmpdebug_WD\AirPlatform_GND_V4.3-bmpdebug_WD.ino:47:0:

    sketch\ApplicationMonitor.h:10:21: fatal error: avr/wdt.h: No such file or directory

    I’ve copied the files from github to C:\Program Files (x86)\Arduino\libraries\CrashTracking as a folder and after one try and getting the error, I’ve also copied ApplicationMonitor.h and .cpp files inside my sketch folder too but still no use… PS. I did restart the ide after the process and also reboot the pc. A conflict with SAM3X uC ?

    • Philip Rowe
      Reply

      Hi Emre, unfortunately it looks like the Due is quite a different device from traditional AVR (uno etc). It cant find wdt.h because it doesnt exist for the Due in that location and judging from this
      http://forum.arduino.cc/index.php?topic=358228.0

      its probably not compatible anyway. You would need to figure out how to port the crash detection from AVR to the due device.

      Good luck
      Phil

  • braeden
    Reply

    Can this be used to dump the memory addresses into a string rather than to Console / serial? Thanks

    • Philip Rowe
      Reply

      It should be possible but is not part of the current version. You would need to modify our application monitor to add the functionality.
      Cheers
      Phil

      • Bob
        Reply

        Excellent – This dug me out of a bad situation very quickly 🙂

        As it stands this will loop forever – resetting the Arduino each time it freezes.

        For very rare events how could it be made to stop resetting the Arduino after 1st trap??
        ie so that the report dump is on the terminal screen without having to scroll back 3days.

        Obviously this stops the application running but this is more convenient in a debug/test environment.

        TIA
        Bob

  • Cian
    Reply

    Does it overwrite oldest entry after DEFAULT_ENTRIES number of resets has happened? I have reset my micro many times and the addresses of the 10 entries is not changing which is suspicious

    Thanks

    • Paul Martinsen
      Reply

      It will overwrite older entries. The address is only saved during watchdog faults however. A normal reset won’t do it.

  • Martin Bergman
    Reply

    I am running your Program.cpp “sketch” above on a Arduino Nano, using Arduino IDE 1.6.12. Everything seem to run correctly as far as to line 41 (“The end is here. Goodbye cruel world.”), and then the micro goes into the while(1), loop evidently. The problem is that it stays there and the watchdog timer does not manage to reset. While in this “spin” the built-in LED (13) flashes very rapidly (like about 10 on/off cycles per seconds), for some reason. There should be no LED flashing in the simulated lock-up loop. If that rapid flashing is somehow caused by repeated resetting, it still doesn’t explain why it happens at approximately 10 Hz, since the watchdog timer should fire at an intervall of 4 seconds.

    I can’t understand what is going wrong here. The only thing I have changed in your original crashTracking code is delay(200) to delay(1000) in lines 35 and 37, to slow the loop down a little.

    Grateful for any tips,
    Martin

  • Martin Bergman
    Reply

    I found a post from 2012 by guru Nick Gammon on the Arduino forum addressing a similar issue with a Nano.
    I quote: “Sounds like … have one of the bootloaders that doesn’t reset the watchdog timer. The flash you see is the bootloader initial flash of pin 13. Then it resets again and flashes it again.”
    So now I’ve burned the Optiboot bootloader to my board, and everything is hunky-dory.

  • JL
    Reply

    Great great code ! Thank you !

  • Sener
    Reply

    This looks a nice wrapping up the Watchdog usage with Arduino.

    I couldn’t get a benefit from this yet. I am using bare metal Atmega2560 without bootloader
    and using Visual Studio with Visual Micro. My own application works as expected.
    So, I wanted to give a test your sample application on the same chip.

    I build and flashed your sample application safe and sound onto the chip. For uploading I am using AtmelStudio.

    Eventually, all I get a continuous reset and on the terminal window, only prints “Application�Ready” and LED 13 rapidly blinking.

    I know the issue while you are using old bootloader. But, in my case there is no bootloader.
    Any idea about that?

    Thank you for your inputs from now.

    • Philip Rowe
      Reply

      Hi Sener, not sure about this one. Did you have any luck getting it going?

  • Matt
    Reply

    How would you suggest clearing the wdtLog in eeprom? I’m thinking only the headers need clearing. Any suggestions how?

    • Philip Rowe
      Reply

      Easiest way would be something like:
      void CCrashReport::Clear()
      {
      CApplicationMonitorHeader Header;
      Header.m_uSavedReports = 0;
      Header.m_uNextReport = 0;
      SaveHeader(Header);
      }

  • Allan
    Reply

    I dont understant this line
    if (g_nEndOfTheWorld == 0)

    If g_nEndOfTheWorld is set at 15, where is it dynamically decremented to reach zero.

    • Philip Rowe
      Reply

      Hi Allan, line 46 “–g_nEndOfTheWorld;”

  • Florida-Rob
    Reply

    looks like 6 Bytes stored for a UNO, example: 0B 03 header, then record 1 = 0A 48 25 00 00 00
    byte address is not stored, simple 2x ‘word address’
    0B = total records saved (weird, should be 0A)
    03 = next position to save
    then each record x 10 = 0A 48 25 00 00 00
    0A 48 = 2 bytes for word address
    25 00 00 00 ?? what is the uData that is stored ? where does it come from ? and easiest way to add an additional byte to store context if triggered for a manual reset ?

  • Rob Walker
    Reply

    Any advice why integrating other code using USART Interrupts, that the combo of this code causes a death of the micro – I guess its stuck in bootloader
    The program actually locked in downloading new code over USB-Serial on Leonardo, so assume the code download was longer that the WDT period (set at 4s) ? so the code half-loaded ? now the bootloader is damaged and cannot load new code ? or stuck in an infinite loop ?
    appreciate your advice, stuck…

pingbacks / trackbacks

Leave a Comment

Start typing and press Enter to search

Create a user interface for your Arduino!

1. Drag and drop controls to build your user interface
2. Send serial comands to your Arduino program
3. Use our free library to process serial commands
Learn More