Diagnosing dead Pixhawk?

Blue Robotics Vehicles
,
1

We have a BlueROV2 from 2017, with a Pixhawk 3DR. I recently installed blueos 1.0.1 on the Raspberry Pi, and updated the Pixhawk to Ardusub 4.1.0, after not having used it for a couple of years. We had the system running and were able to dive, but were getting frequent “EKF3 IMU1 forced reset” messages, pre-arm check failures and warnings about yaw inconsistent, even after recalibrating all the sensors. We then lost the connection. On power up, we can see video in QGroundControl, and we can ping 192.168.2.2, but there is no heartbeat in blueos.local, and the Firmware link says that no board is running. On the Pixhawk, the FMU lights are:
PWR: flashing green
B/E: off

and the IO lights are:
PWR: solid green
B/E: solid orange
ACT: flashing blue

Something (the Pixhawk?) is beeping every few seconds, one long tone and two shorter ones, rising in pitch.

I tried the restart autopilot button in blueos.local, but it doesn’t seem to change anything. I can’t see anything disconnected between the Pi and the Pixhawk, although I will open up the electronics tube to check. Is the Pixhawk dead, or are there more things I should check? And if it is dead, should I just buy a Navigator instead?

3 Likes
2

Hi Matthew,

Did you solve this issue? I am having the exact same problem after attempt to install blueos firmware.

Thanks!

3

Hi @Kristen_M -
The issue @mspencer describes here seems like a hardware problem with the pixhawk. The video system is handled by the Pi, and typically the IMU forced reset messages indicate a poor quality compass calibration. However once you here beeps from the ESCs (via the thrusters, they are acting as speakers!), this indicates the pixhawk isn’t sending a valid PWM control signal.

Double checking your ESC to Pixhawk connections, making sure there is no evidence of a leak, and that your USB cable (between Pi and Pixhawk) is ok would be good first steps. Plugging the pixhawk into the computer and flashing it with ArduSub from QGC is a good thing to try to.

Upgrading to a Navigator is definitely worth it in my opinion!

@Kristen_M - if you already have a Navigator, what version of BlueOS did you flash to the Pi SD card?

1 Like
4

We bought a new Pixhawk, and it’s been fine with BlueOS since then. I think we didn’t change to a Navigator because it would also have needed a Raspberry Pi upgrade.

5

Hey Tony,

Thanks for your help! When plugged into a 14.8V 10,000MaH battery, the ROV turns on, the subsea lights turn on (and stay on, slowly fade out after ~3 min, instead of the usually quick on and off), there are the 3 initialising beeps, but not the two longer beeps that usually come next.

The Power light is green and B/E light blinks yellow/orange on the FMU side of the pixhawk, and the power light is green, B/E is a solid yellow/orange (no blinking) and the ACT is blinking blue. The main LED light in the middle of the pixhawk has no light displayed. There are green and yellow lights active on the raspberry pi3 and fathom tether board.

On the Fathom-X topside interface, the power and link active lights are on.

When connected to QGC, only the video stream is active, and the vehicle is disconnected.

All of this started after a failed installation of blueos firmware, the first was a timeout error, and a second attempt gave these errors:
‘failed to install firmware’ and ‘error 500’.

The Pixhawk to ESC connection is good.

I haven’t done much maintenance of the hardware & software side of things. The only available micro usb port that I can see to connect the pixhawk to the computer currently has the ESC cord connected. Can this be unplugged so the port can be used to plug into the computer and flash ardusub?

The raspberry pi 3 board underneath has a spare micro usb, will this have the same outcome? I have tried using this port and connecting via QGC, but no device is detected.

Previously in the field we were having intermittent connection issues, and thought this issue may have been caused by an old and roughed up slip ring on the tether. Perhaps this was actually due to a dying pixhawk or raspberry pi board?

Thanks again!

1 Like
6

Hi @Kristen_M -
It definitely sounds like your Pixhawk has failed, or is not running valid firmware. If the ESCs are connected to it and don’t initialize with that final beep, that’s the key piece of evidence.

The Raspberry Pi only has a green and red LED indicator - not sure how you’re seeing yellow?

The Fathom-X has both green LEDs illuminated, since you can get a video stream ok.

To be clear - all this started when you attempted to install the autopilot firmware on the pixhawk, from BlueOS - BlueOS itself is not firmware! This may be recoverable, again if you use QGC and directly connect the Pixhawk to the computer running it via USB.

You will indeed need to disconnect the USB from the Raspberry Pi, and connect it to the computer to flash. If no device is connected, try another USB cable. If it still doesn’t detect, the pixhawk is indeed dead!

When you say intermittent connection issues - were you losing video connection or just vehicle connection? If just the vehicle connection, that would be likely due to the failing pixhawk, yes. I’ve never encountered a slip ring causing connection issues for the Fathom-X.

7

Thanks Tony. Yes all of this started after I put the microSD card into the raspberry pi and attempted to install the firmware onto the pixhawk, through blueos interface.

I have flashed the pixhawk by connecting it to the computer, and gone through system setup calibration steps. Everything is working, including video feed, however I have no control of the camera tilt. The subtext file reads that the tilt is at 238%… I have only ever seen it between 0-100%.

I have calibrated the joystick and made sure all buttons are assigned to appropriate controls, and have the camera setting on channel 9.

I am at a loss of what to try next! Thanks so much for your help!

8

Hi @Kristen_M -
So after flashing, things are working in Cockpit or QGC, except for your camera tilt?
Typically camera tilt is controlled via channel 16?

9

That’s correct. Everything other than the camera tilt and the lights are working. When I access camera settings or light setting in the vehicle setup a pop up box appears saying ‘parameters are missing from firmware . You may be running a version of firmware which is not fully supported or has a bug in it. Missing params: 1:BRD_PWM_COUNT”

This is the notification that pops up for the lights. The camera is much longer, I’ll attach a photo.

IMG_2873
IMG_28733024×4032 2.83 MB

10

Hi @tony-white I have run through the flash process again for the pixhawk and have the exact same results. The ROV works fine in every regard, except the camera won’t tilt or respond to tilt controls, and still sits at the high tilt % in the subtext file, and the lights don’t respond and state they are disabled in settings with no option to turn on. The subtext file also states that the lights are at level 50, but they are not on.

I am at a loss for what to try next. Thank you!

11

@tony-white I have switched the pixhawk from one of our other ROVs that has been out of action, and having no problems at all now. Thank you again for your help!!