"No board running" - fixing broken ArduSub install

Thank you for the help. I am now having a new problem. I went through the steps you listed and made sure to get right parameters like we talked about before. And when I got to the part where a few of the parameters didn’t appear like BARO_PROBE_EXT, VISO_TYPE, EK3_SRC1_POSXY, and EK3_SRC1_VELXY. I followed the step where I needed to make sure I had ardusub beta (4.1.0). For clarification I am running BlueOS version 1.0.1. I updated the firmware through BlueOS and rebooted vehicle. But now whenever I turn on vehicle, the ardusub board doesn’t connect. So I have no connection to QGroundControl. And since no board is running I can’t go back to the 4.0.3 firmware I was running on before. Do you know how I can get the board to get back running?


Hi @Jnyberg,

I believe I’ve run into this issue before (documented here), which we suspect is caused by an un-caught failure in ArduPilot’s firmware updater tool (which is used in BlueOS). I was able to fix that at the time by flashing a fresh ArduSub firmware onto the Pixhawk via a direct (USB) connection to QGroundControl.

Unfortunately we haven’t yet developed / replicated that QGC functionality in BlueOS, although we would like to and are planning to, since it would add a robust fallback option. I’ve asked internally for any alternatives (and will get back to you with any responses), but assuming no new ones have been found you’ll likely need to open the enclosure and fix the autopilot with QGC.

As some context, BlueOS is super robust at updating itself (and recovering from any errors that may occur), but external flight controllers like the Pixhawk don’t have the same kind of modular architecture, so errors are harder to detect and unfortunately can’t be rolled back in the same way. Improving the firmware updating robustness is definitely something we’re working on.

2 Likes

Following up on this, @rafael.lehmkuhl has asked that if possible you get the ardupilot-manager log, in case it’s caused by some other issue. You should hopefully be able to find it in var/logs/blueos/services/ardupilot-manager/ via the File Browser :slight_smile:

1 Like

One more thing that I would like to add in top of what @EliotBR just said:

We’ve also seen this problem of the Pixhawk not appearing after a failed firmware flash. It’s a problem with the board detection routine on the official uploader script, which we are trying to fix.

Usually one can get around the board-recognition failure by hard-rebooting the companion computer (turning power on and off). The pixhawk board should then appear, so you can try to flash it again with the frontend interface normally. If the flash with the interface does not succeed or the board is not recognized, you can still try to flash the board through the bultin-terminal:

  1. Open the terminal
  2. Run ardupilot_fw_uploader.py --identify (to check if the board is being recognized)
  3. Put your firmware file in the companion computer
    1. via the FileBrowser, OR
    2. download via wget, e.g.
      wget -O ardusub.apj https://firmware.ardupilot.org/Sub/stable-4.0.3/Pixhawk1/ardusub.apj
      
  4. Run ardupilot_fw_uploader.py /path/to/firm/file
2 Likes

@rafael.lehmkuhl and @EliotBR thank you so much for the info and tips! I ended up doing the first option and reflashing the firmware directly from QGC via usb and got pixhawk back. And when I retried the firmware update it worked. But what @rafael.lehmkuhl mentioned seemed to be exactly the case too so if I run into problem again I will try to use that efficient method so I don’t have to open up the enclosure :slight_smile: Thanks again!

1 Like

I’m glad you got it working again! Would you mind sharing your last logs for this service, including the one from the session where you had the firmware flashing problem? We want to make sure we are solving the problem for all failure cases, and the service logs will help with that.

1 Like

I finally managed to replicate this issue at a time when I was able to readily access the logs, and do some debugging and testing with @rafael.lehmkuhl. We believe we’ve found the cause (firmware flashing was reporting as successful even when it wasn’t), and will be taking steps to detect such failures going forwards.

Unfortunately the root cause of why flashing fails sometimes is still unknown (and there could be several reasons), but at least being able to consistently detect when it occurs means we can notify about it properly.

The “hard reboot” will likely still be required to fix it (since the Pixhawk needs to be physically shut down, which can’t be done in software), but we should be able to simplify the process to avoid needing to run terminal commands, and it shouldn’t require opening the electronics enclosure, which is a definite win :slight_smile:

There are some more details in my comment on the GitHub Issue, where further developments can be tracked as they occur.

It’s great to be making some genuine progress towards having this resolved - intermittent problems are really challenging to find and handle.

1 Like