QGC Says "Disconnected" but Video OK after Companion/Pixhawk update

Blue Robotics Software
1

@williangalvani @jwalser
We recently pulled up our 900m ROV and I am taking advantage of the short time window before redeployment to update SW and make a few other improvements. After updating the companion software and the Pixhawk software to the current stable versions, I am unable to get telemetry from the ROV. I can get video from the ROV but it shows “Disconnected” in QGC and I cannot download parameters from the ROV. Both Companion & Pixhawk updated successfully.

Something similar happened after a previous update a year or so ago and I resolved it by resetting the Mavproxy to default settings @ 192.168.2.2:2770/mavproxy
I tried that this time but it did not resolve the issue.

I have this week to resolve the issue before redeployment to 900m again next week. Any quick solutions you can provide would be much appreciated!

2

Hi @hube268,

Can you confirm that the Pixhawk is being detected, and that the versions are in fact the latest stable ones for Companion and ArduSub? You should see all of the following on the System page (http://192.168.2.2:2770/system):

companion-stable-versions
companion-stable-versions746×888 19.2 KB

If the ArduSub version can’t be detected then most likely the update was not in fact successful, in which case you may need to connect the Pixhawk to the topside computer directly, and flash it via QGC’s Firmware page.

If it is being detected then our troubleshooting docs may help :slight_smile:

Are you aware that Companion is no longer being developed?

BlueOS (its replacement) is into stable releases now, and we’re in the process of making a proper post about that which recommends switching to it, but that post may not be complete for a few more days (and I figured this is better to flag for you sooner rather than later).

3

Thank you Eliot for the info. In fact, for the ArduSub version, it says “unknown”. I tried to “revert back to default firmware” but that didn’t work either. Is there no other way to recover besides plugging in the Pixhawk directly? I was hoping to not have to open up the top enclosure as its been validated to 900m but if thats the only way I may have to.

Thank you for the heads up about the companion software. I may have to call my contacts at BR to discuss that option. I appreciate the heads up!

4

You can try the “Stable” update button again, but I believe Companion in general has issues trying to interact with the Pixhawk if it doesn’t already have a valid ArduSub firmware installed. I’ve asked the software team in case there’s some other approach I don’t know about, and will get back to you :slight_smile:

Note that upgrading to BlueOS will require opening the enclosure to flash the SD card, so if you do need to open it you can do that update at the same time instead of waiting until the next vehicle surfacing. BlueOS can also run on a Raspberry Pi 4B if you want some extra processing power, but that may not be feasible in your current timeline if you don’t already have one on hand.

5

@EliotBR So here’s an update:

I opened up the enclosure and reflashed the Pixhawk with it directly connected to my windows computer via USB and following the instructions from the Firmware Page you linked in your first reply. It all seemed to go smoothly and the Pixhawk restarted and downloaded parameters while still connected via USB.

I plugged it back into the Pi and connected to it again and am having the same issue where it shows the vehicle is “Disconnected” but I am able to see video and I am able to access the web interface. On the web interface, it still says “Not Found” for Ardusub version. The LED on the Pixhawk is flashing blue when the ROV is on.

The settings for Mavproxy found at 192.168.2.2:2770/mavproxy are:
–master=/dev/autopilot,115200
–load-module=‘GPSInput,DepthOutput’
–source-system=200
–cmd=“set heartbeat 0”
–cmd=“param forceload /home/pi/companion/params/serial0.param”
–logfile=/tmp/telemetry.tlog
–out udpin:localhost:9000
–out udpin:0.0.0.0:14660
–out udpbcast:192.168.2.255:14550
–mav20
–streamrate 10
–default-modules=output,param

What else can I try?

Thanks for the help!

6

@EliotBR I also was following the troubleshooting guide and there does seem to be an issue between the communication of Mavproxy and autopilot. When I logged into the Pi and ran the command “screen -r mavproxy” I got the below. Is this an issue with the Companion computer, pixhawk firmware or both? Are the latest stable versions of Pixhawk and Companion firmware compatible?

File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 2876, in
working_set = WorkingSet._build_master()
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 449, in _build_master
ws.require(requires)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 745, in require
needed = self.resolve(parse_requirements(requirements))
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 639, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: future
/dev/autopilot
Traceback (most recent call last):
File “/usr/local/bin/mavproxy.py”, line 4, in
import(‘pkg_resources’).run_script(‘MAVProxy==1.6.1’, ‘mavproxy.py’)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 2876, in
working_set = WorkingSet._build_master()
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 449, in _build_master
ws.require(requires)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 745, in require
needed = self.resolve(parse_requirements(requirements))
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 639, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: future
/dev/autopilot
Traceback (most recent call last):
File “/usr/local/bin/mavproxy.py”, line 4, in
import(‘pkg_resources’).run_script(‘MAVProxy==1.6.1’, ‘mavproxy.py’)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 2876, in
working_set = WorkingSet._build_master()
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 449, in _build_master
ws.require(requires)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 745, in require
needed = self.resolve(parse_requirements(requirements))
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 639, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: future
/dev/autopilot
Traceback (most recent call last):
File “/usr/local/bin/mavproxy.py”, line 4, in
import(‘pkg_resources’).run_script(‘MAVProxy==1.6.1’, ‘mavproxy.py’)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 2876, in
working_set = WorkingSet._build_master()
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 449, in _build_master
ws.require(requires)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 745, in require
needed = self.resolve(parse_requirements(requirements))
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 639, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: future
/dev/autopilot
Traceback (most recent call last):
File “/usr/local/bin/mavproxy.py”, line 4, in
import(‘pkg_resources’).run_script(‘MAVProxy==1.6.1’, ‘mavproxy.py’)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 2876, in
working_set = WorkingSet._build_master()
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 449, in _build_master
ws.require(requires)
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 745, in require
needed = self.resolve(parse_requirements(requirements))
File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 639, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: future

7

@EliotBR I think we figured it out. I pulled the SD card from the Pi and reflashed it on my computer and it seems to connect now as well as show the Pixhawk version on the web UI. I guess this is a good reminder not to trust the update feature when the ROV is 900m below the ocean :slight_smile:

We’re re configuring everything now so hopefully that goes smoother than the update.

Thank you for your time!

8

Glad you managed to get it sorted :slight_smile:

I believe that’s caused by an undetected failure during a Companion update, which is corroborated by you flashing a fresh Companion image and solving the issue.

While Companion’s updating process attempts to detect and roll back any failures that occur, there are occasional failures that don’t get picked up during an update. When that happens the result is a messed up installation in an unknown state, and because of Companion’s software architecture that’s challenging to get out of without a freshly flashed image.

That lack of robustness was one of the driving forces behind the containerised architecture of BlueOS. Not only is the new updater less likely to fail, and better at detecting failures, if an undetected failure somehow occurs (or if a broken version is somehow installed) it’s possible to easily roll back to a working version from on the device, manual upload, or downloaded from the internet. In addition, the BlueOS updater runs independently, so it will still be available even in an extreme case where the whole main web interface gets broken.

Updating was definitely a potential pain point in Companion, and a lot of effort has gone into making version management instead be a robust and versatile feature in BlueOS, which should benefit both users and developers :slight_smile: