Flight Controller Firmware Reflash and Unresponsiveness Issue

Blue Robotics Software ArduSub
,
1

After reflashing the firmware, the flight controller operates smoothly. However, attempts to arm and disarm the vehicle using the PyMavLink Python code have proven unsuccessful. The vehicle becomes unresponsive, ceasing to accept further commands from either QGround Control or the Python Code. At this stage, the heartbeat connection is lost. As a remedy, we are compelled to repeat the entire reflashing process to restore the functionality of QGround Control.

The vehicle does not arm using the PyMavlink Python Code provided in the Official Documentation.

2

Hi @Sathyashree, welcome to the forum :slight_smile:

You’re seemingly describing a very unexpected error type and without much context, so I’m both unsure whether I’m misunderstanding you and struggling to provide very meaningful advice.

As some questions to clarify the situation:

  1. Which software versions are you using (ArduSub, BlueOS, Pymavlink, QGroundControl)?
  2. Which hardware are you using on your vehicle (onboard computer, flight controller board)?
  3. What python code are you running to connect to and arm/disarm the vehicle?
  4. Are you trying to connect to the vehicle using both your Python program and QGroundControl at the same time? And if so, have you set up separate MAVLink endpoints to enable doing so?
  5. When the vehicle “becomes unresponsive”, do the video stream and telemetry data continue being received at the topside, or do you entirely lose connection to the flight controller and/or the onboard computer?
  6. Once the vehicle has become unresponsive, is functionality restored when you reboot / power cycle it, or do you actually need to re-flash the flight controller firmware or the SD card on the onboard computer (and if so, what are you “reflashing”)?
  7. Does arming and disarming (and general vehicle control) work as expected when done through QGroundControl?
3

Hello,
To address each of the questions:

  1. Software Versions:
    We are currently using the following software versions:
    ArduSub: 4.0
    BlueOS: 1.0.1
    Pymavlink: 1.8.62
    QGroundControl: 4.2.3

  2. Hardware Configuration:
    The hardware setup on our vehicle includes the BlueROV2 bundled Raspberry Pi 4 Model B and the navigator flight controller

  3. Python Code for Vehicle Connection and Control:
    To connect to and arm/disarm the vehicle, we are using the official Pymavlink documentation.
    We are using the following Python code:

    from pymavlink import mavutil

master = mavutil.mavlink_connection('udpin:0.0.0.0:14550')
master.wait_heartbeat()

master.mav.command_long_send(
    master.target_system,
    master.target_component,
    mavutil.mavlink.MAV_CMD_COMPONENT_ARM_DISARM,
    0,
    1, 0, 0, 0, 0, 0, 0)

print("Waiting for the vehicle to arm")
master.motors_armed_wait()
print('Armed!')

master.mav.command_long_send(
    master.target_system,
    master.target_component,
    mavutil.mavlink.MAV_CMD_COMPONENT_ARM_DISARM,
    0,
    0, 0, 0, 0, 0, 0, 0)

master.motors_disarmed_wait()

  4. Simultaneous Connections:
    We have tried both simultaneous and non-simultaneous connections using our Python program and QGroundControl, but unfortunately, neither approach worked as expected. Additionally, we attempted to create a new MAVLink endpoint for connecting, but this attempt also failed to establish a connection(Arm/Disarm).

  5. Unresponsiveness and Data Transmission:
    When the vehicle becomes unresponsive, we lose communication with the flight controller, which results in the loss of telemetry data. However, the Raspberry Pi onboard computer continues to function without any issues during this unresponsive state.

  6. Recovery Process:
    Rebooting the vehicle does not solve the issue of unresponsiveness. To restore functionality, we have found that we need to reflash the BlueOS version 1.0.1 on the flight controller. After this reflash process, the vehicle resumes normal operation.

  7. Arming, Disarming, and Vehicle Control through QGroundControl:
    Yes, arming, disarming, and general vehicle control work as expected when done through QGroundControl.

Please let us know if you need any further information or if there’s anything else we can provide to help with our current setup. Your assistance is greatly appreciated!

4

Dear Eliot, is the issue addressable? We are not able to figure out the rootcause of the issue. We are thinking it could be either in the software configuration of the navigator controller board. It would be great if we understand what could be the potential rootcause of the issue so we can work on it going forward.

Thanks and Regards,
Sathyashree

5

Apologies for the delay on getting to this.

Thanks for responding to the different questions - that’s helpful for narrowing down the issue :slight_smile:

Navigator support was first added in ArduSub 4.1.0, so most likely one of these pieces of information is incorrect.

I tried running this code snippet on my laptop and it worked fine for me, both with and without QGC open, although I did need to configure a separate MAVLink endpoint for Pymavlink to connect to to be able to use QGC at the same time. When both were connected at once I could see the vehicle armed/disarmed state changing in QGC as it was modified by Pymavlink.

I was testing using our latest recommended software versions, but on a Raspberry Pi 3B with a Pixhawk, and with BlueOS 1.1 beta and Pymavlink 2.4.39.

I’ll try again tomorrow with a Navigator setup and BlueOS 1.0.1 (stable) to see whether that makes a difference, and I’d recommend you try updating your software to recommended versions in case that helps resolve your issue.

This is strange, but seems like at least one of the firmware or the ArduPilot Manager service is crashing. It could be useful to see a system log from ArduPilot Manager, but I’d recommend trying BlueOS 1.1.0-beta first (which will also make it easier to access system logs if you need them).

This is the most unusual part - my best guess is that the ArduPilot Manager is crashing and then failing to resume because of some bad settings or something. I’d be curious whether resetting the settings helps.

My only other idea here is that there’s a problem with your Raspberry Pi’s SD card, in which case trying a different one might help, but I would expect that to cause issues with BlueOS running at all, not just a limited and consistent part of it.

6

I tried this and had some issues with the Navigator failing to run correctly, which is likely due to a difference in settings since I downgraded from a more recent BlueOS version. If it’s important I can try a settings reset or a fresh install, but it was working fine when I tried with BlueOS 1.1.0-beta.28 so I’d recommend you try that first if possible.

7

Thank you so much Eliot!! We’ll try with the beta version and get back to you as soon as possible.