I updated the firmware on my companion computer to 0.0.26 and the Pixhawk to 4.0.3 when I installed the new newton gripper last weekend. Ever since then my ROV has been unusable due to horrible connection speeds. Even QGroundControl constantly loses connection to the vehicle. The ROV was working fine until I updated the software. I’m not sure how to go back to how it was before. I tried the different Pixhawk firmware options (even default firmware and settings) with no real difference, and I don’t see a way to roll back the companion computer.
Do I need to remove the microSD card and manually flash a older version (like 0.0.25), or is there a simple fix?
But sometimes it briefly goes back to normal before the network speeds crash again. I only used the “Manual” network settings. I just clicked the “DHCP Server” option before I took the screenshot because I was considering trying it.
I finally got around to trying to manually flash the 0.0.25 companion computer image since obviously “updating” the ROV to 0.0.26 and 4.0.3 was a huge mistake (for the second time in a row), but I don’t see a download link for it since it only has the latest version. Rolling back ArduSub was super easy since I can manually upload the ‘ardusub-402-depth-hold-fix’ image. I just need to roll back the companion computer. Downloads · GitBook (ardusub.com)
Thanks. I haven’t tried that yet. I just need to get the companion computer back to 0.0.25 since that was the last known good configuration for my ROV. I have a known good ArduSub image that I can upload on the web interface page, so that isn’t a problem. I still appreciate having a link to all of the versions.
Hi, I can also confirm network issues with companion 0.0.26 and ardusub 4.0.3.
The network test shows very low download rates and unrelastic high upload rates of 1000 mb/s.
In our last test outside we also experienced some latency issues after 1-2 h of operation.
I think I am also going to downgrade to 0.0.25
and ardusub firmware 4.0.2. depthhold fix …
@almathisen It says that I need a username and password. It isn’t showing anything under branches or tags on the origin and it needs to login when I click on the key symbol to the right of the address, so I assume that the origin is locked without being logged in. I did follow that guide and clicked on the Fetch button first.
That only works on the local tab (/home/pi/companion). There’s nothing on the github origin tabs for Branches or Tags. I did follow the directions and Fetch it first to ‘get the last changes of the remote repositories’, and I obviously do have it connected to the Wifi with an internet connection. I think that lock symbol has something to do with why they aren’t showing up.
Update if anyone cares: I was forced to pull the electronics tube and manually flashed 0.0.25 on the MicroSD card. I was less than impressed with the location of the card, but luckily I quickly realized that it was 2 screws to pull the end off (with the camera assembly) and then I had easy access to pull it out. A quick Ping and inspection in QGroundControl shows that the network is back to normal. I will leave 4.0.3 installed for now to give it a fair test. I guess 0.0.26 or something in the upgrade process really was the problem.
I might do a test dive in the next few days if it’s not too hot or windy after work.
Further update: I tried going back to 0.0.25 but it has the same 1,000ms+ pings that 0.0.26 had, so there is a possibility that I may have been too quick to judge 0.0.26. The problem is definitely network based and might even be an intermittent problem. I just got 0.0.24 working so theoretically I finally have a working ROV as long as it reliably connects to my computer. It initially didn’t connect at all, but then I noticed that the “Link” light on the ROV tether interface board wasn’t lit, but it did light after I switched USB ports on my laptop and QGroundControl connected and seems responsive.
Right now I’m very hesitant to touch anything because I was in this same position last week and then the ROV stopped working when I re-tried to use 0.0.26. I had planned on looking at a shipwreck on Lake Coeur d’Alene last weekend and I missed out on that adventure due to the ROV being down. I’m planning on going back and doing it this weekend so I might still have an interesting video to share next week (if the ROV works).
I just took it to a dive site and it definitely does not work. Looks like I wasted most of the limited free time that I have after work driving out there and back, and I’m probably going to miss out on my planned adventure this weekend (again). I don’t understand why the ROV either seems to work fine and have 5ms pings, or it doesn’t work and then has 1,000+ms pings. The ROV has been down for 68 days now, though I haven’t been troubleshooting it for the whole time. It’s getting annoying how I’m really starting to miss out of the things that I was planning on doing with it.
I think some of my network connection problems was caused by the FXTi not being given 192.168.2.1 even though it is manually set to that in the network options. When I get it connected the pings are still in the 500-700ms range so the transfer speeds are obviously still too low to use the ROV. QGroundControl looks like a slideshow.
I’m really not sure how to troubleshoot the horrible network speeds after it’s connected. It used to be fine before I installed the Newton gripper and upgraded the companion to 0.0.26. I need to get it working in the next few hours or else I won’t be able to use it this weekend.
If the FXTI isn’t assigned the correct IP address then I believe your computer won’t receive any video packets at all, and if it’s not on the 192.168.2.xxx subnet then it wouldn’t receive any mavlink packets either. That’s more of an ‘infinite latency’ situation than a ‘high latency’ one.
From what we’ve seen some people have high latency from mavproxy using a lot of CPU, but I don’t believe that’s a new phenomenon to 0.0.26 (although it is something we’re hoping to fix in 0.0.27, which I believe is releasing next week).
It also doesn’t seem plausible the gripper could be causing your issues, because even if it was capable of drawing enough power that the RPi got under-powered that should show up in the companion web interface, and also could only happen when the gripper motor was doing work, which it shouldn’t be unless it’s actively holding something or changing from open to closed.
Have you tried bypassing the tether and fathom-X boards with an ethernet cable directly from the RPi to your computer (or if necessary via the ethernet to USB converter in the FXTI)? I have some recollection you mentioned trying that somewhere, but I might be getting confused between other people I’ve been helping.
We have had the same problem with high latency sometimes.
You can try to disable the waterlinked gps service on the raspberry pi. I dont know if it was that the cause of the problem, but i have not had latency problems after i disabled the waterlinked driver. The problem also stopped when I typed the correct IP for the waterlinked in companion when UGPS G2 was online.
But we are using Ardusub Beta, DVL companion and QGC 4.1.1.
In companion/.companion.rc you can disable some of the services so they not start up when you start the ROV. Just use # in front of a line.
That’s correct. It’s won’t connect at all if the FXTi doesn’t have the correct IP. I’m not sure if it was just a 1 time thing caused by my troubleshooting process going between Windows and Ubuntu, or if it’s part of the main problem. It used to flawlessly work when switching between multiple computers.
That’s what I was thinking since it’s just a signal out of one of the Pixhawk Aux channels. The Newton seems to be working perfectly. I though that I would mention it anyways since that is when the problem appeared. The only things that changed are: I installed the Newton, I upgraded the software to 0.0.26 and 4.0.3, or maybe I accidentally unplugged something or damaged the tether without knowing.
I haven’t. As I have mentioned, Sometimes the ROV has 4ms pings and 70Mb transfers, so I’m not sure if it’s the tether since it occasionally goes back to normal performance. I haven’t figured out why that happens yet. It seems to be usually right after I flash a new companion image. Twice now I have gotten the ROV working in my bedroom and have thought that I was back in business since it reliably re-connects after power cycling it, but then the next day the problem is back.
What it the best way to troubleshoot the tether other than bypassing it with a ethernet cable?
Edit: I suspect it could be high CPU usage causing the lag. The companion computer usually has more than 60% CPU usage.
@almathisen Thanks for the tip. I’m not using any GPS yet. I don’t want to disable services since I don’t know what any of them do. Mavproxy was using the most CPU and I can’t disable that.
I doubt you’ve unknowingly damaged the tether, but might be worth making sure the connections are all physically stable (e.g. Ethernet cables fully plugged in, fathom X wires screwed in properly, etc) for the full set of connections between the RPi and the surface computer (RPI → Ethernet cable → fathom-X → twisted pair → tether → binder plug → FXTI twisted pair → fathom-X → Ethernet to USB converter → USB-A to B converter → USB B to A cable → computer)
The main other thing that can be tried is swapping to a different twisted pair in your tether if you’ve got a spare one, in case the original one is damaged/corroded, but it’s pretty rare that that’s the problem.
This is another thing that could do with some clearer documentation
The different services have names after the screen -dm -S, which are hopefully relatively intuitive for the most part. I’m meant to be sleeping, so trying to give enough detail without taking too long:
pingproxy allows ping1D to connect
pingmav allows ping1D to send distance measurements to QGC
mavproxy connects QGC to the Pixhawk
video is video stream
webui is the companion web interface
webterminal is the terminal you’re using (probably good to keep, but not essential if you know how to use ssh, user:pi, pass: companion )
commrouter is for forwarding connections, at the /routing page of the web ui (not essential if you’re not using any such connections)
audio is saved as part of video recordings, although QGC doesn’t play it live at the moment
file-manager allows you to get files off the companion computer (can comment if not using, also possible to do directly with scp instead)
nmearx is for NMEA GPS (can comment if not using)
wldriver is for water linked underwater GPS (can comment)
bridgemanager enables ping360 connection
mavlink2rest provides a mavlink terminal in the webui (can comment out, always possible to restore later if you for some reason need it for something)
network-service I think handles wifi connections, probably leave it
Yeah, don’t disable mavproxy, although as an interim measure you can try reducing the --streamrate parameter at http://192.168.2.2:2770/mavproxy, which may help with CPU usage as well.
Hope that’s helpful, and really hope it’s not too late for your weekend fun!
Edit: if it’s helpful, I believe the biggest differences would be from reducing the mavproxy streamrate and commenting out mavlink2rest - the others you can comment if you want but they’re unlikely to change much (they do very little when they’re just waiting for a device to connect).
Edit 2: .companion.rc is a startup file so changes won’t apply until you reboot the RPi through the web interface or power cycle the ROV. Can’t remember if mavproxy settings updates are instant or also require a reboot
@EliotBR Thanks. You are very helpful as always. Unfortunately it didn’t solve the problem. I reduced the Mavproxy stream rate to 1 and now the pings are less than 500ms and the QGroundControl video has about a 3 second delay with most of the frames being dropped. It’s still not usable, but it’s getting close. The network connection seem stable but it’s just transferring the data too slowly for some reason.
As of right now I’m back in business and I just completed a 20 minute dive with no real issues other than some interesting acrobatics before I setup the thruster rotations.
I had disassembled the FXTi to switch tether pairs and accidentally pulled the whole board out and disconnected the brown (main) tether pair from the tether interface board. I then plugged the brown pair back in where I thought it went and looked on my computer to verify that it had a connection, but I was surprised to see that the pings were back to normal.
Looks like tomorrow’s shipwreck dive is back on schedule as long as the ROV stays the way that it is right now.
