Editing this whole post to reflect what I learned on a long goose chase: terrible comms to a BR2 with the OTPS system were due to some sort of incompatibility with or just plain poor performance of a fancy new laptop I got expressly for the purpose of running my ROVs. Everything went back to normal when I switched to an older, less-powerful laptop. That older laptop had an older version of QGC on it (4.3) so I tried that version on the newer laptop, still saw terrible performance on the newer one.
1 Like
…and then randomly I’ll have some periods of time where the download speed isn’t too bad:
Followed by times where the video feed drops and the vehicle disarms itself while nothing else has changed. Frustrating!
Enclosure tube off and back on the bench, same results. Both images below from the same speed test. Download starts off normal, looking like it’s heading to the 65mbps range, but then plummets to almost zero. I took the BR ethernet switch out of the loop to see if it was causing the trouble, no change. Swapped interface USB cables in case the previous (extra long) one was bad, no change.
So it has something to do with video and/or QGC. I started with a fresh BlueOS 1.3.1 installation this morning. Same results as before when QGC was running. Shut QGC down and voila, back to normal:
I’ve used tether setups with as little as ~15mbps bandwidth with video and a high-def sonar with no problems, so something is clearly weird here for the video stream coming from the vehicle to be crippling throughput with more than 70mbps capacity. Interestingly, when I connect to the vehicle with Cockpit instead of QGC (after multiple reboots of the vehicle) I get no video stream (and no decrease in speed/bandwidth). I haven’t changed any settings in Cockpit and it did work early on in my setup attempts with this OTPS system.
Hi @rperkins -
We recommend using QGC version 4.2.8 - that might be worth a try on the newer laptop? You can find recommended software versions on the Technical Reference.
1 Like
Thanks Tony, I’ll give that a try.
One other thing @rperkins -
If you continue to have camera issues, have you verified your camera is not affected by Technical Bulletin 12 ?
How is the video performance in Cockpit?
Is the ROV running BlueOS 1.3.1 and the latest Cockpit?
I didn’t know about tech bulletin 12, and it turns out that my camera and mount are indeed affected! I will chisel away a bit from the mount so that it doesn’t squish those resistors.
Video performance in Cockpit seemed fine until it went away entirely, at least on my new laptop. No video shows up there at all anymore! I will try with the old laptop shortly. ROV is running 1.3.1 and the latest Cockpit as far as I can tell. I’m still having the problem where BlueOS can’t fetch updates on this vehicle. Raspi and Ardusub have no problem doing updates and BlueOS can see them, it just fails when trying to actually fetch anything.
The camera stream cutting out means you likely have already damaged at least one of the resistors, the camera can be replaced. Please reach out via the support form with order #!
As for the connectivity issue, are you saying you can’t pull updates or install extensions? Or that you’re getting nuisance warnings the connection to BlueOS has been lost? What is the behavior of the heart-beat icon in the upper right during this if so?
I had noticed Cockpit appearing and disappearing from the menu on the left side of the BlueOS UI. When I went to look at the installed extensions, initially I think it said I had Cockpit 1.0.2. I clicked the update button and it failed just like the BlueOS updates do. I did another post here about the docker updates failing. I’ll put another screenshot of that here:
Same behavior with different laptops and different internet access methods (home router versus tethered to cell phone). The Pi and Ardusub can update themselves, just not BlueOS things. Internet access on the Pi works fine. Vehicle heartbeat is steady, QGC has no issues while these update attempts are happening.
A weird thing about the video issues is that I can see the stream coming in at its usual steady 10-12mbps when looking at ethernet traffic via task manager whether QGC is running or not. I see ethernet traffic jump when I do a speed test, and it goes right back to normal afterwards. My new laptop has an Nvidia A500 GPU, which is pretty much the same as the laptop RTX 3050 I have in my older laptop. When I tell QGC to force video decoding to the Nvidia GPU I can see it working and the performance is a little better than when using the integrated Intel GPU or forcing it to do software decode. The Nvidia GPU normally hums along at less than 10% usage when it’s decoding which is what I would expect. Sometimes, however, usage spikes to around 50% for no apparent reason and stays that way for awhile. Sometimes I notice this happening when I’m doing a speed test, but not always, and it will stay that way long after the speed test is over. All in all the behavior is much like I would see with my oldest laptop when it wasn’t plugged in to AC power, which is much less powerful than the two I’ve been talking about here. It has a very basic discrete GPU which it would not use at all unless plugged in to AC power. My newest laptop with the A500 acts like that ancient old laptop in its unplugged feeble state all the time! What I really don’t understand is why communications from the ROV to my laptop would be affected no matter what’s happening on the laptop when the ROV is just blithely sending its UDP video stream regardless of who is “listening”.
I can’t download a fresh version of Cockpit, naturally. I was looking at the other extension shown there- something named major tom and identified as a cloud agent. Its container isn’t running either. Perhaps that’s the piece that’s supposed to fetch updates?
Now I don’t have any video coming from the vehicle. Looks like maybe fixing the camera mount took the pressure off of those squished resistors and now they are disconnected, killing the camera? Sounds about right.
Swapped out camera and video has returned.
Hi Richard, about that behaviour on your speed test,
It is very familiar to me also.
Have you found that it’s caused by QGroundControl being open on your machine?
Yes, the slowness is at least partially caused by QGC running. The weird part is that it didn’t used to be that way-- I know in the past I have run speed tests with QGC running and while the throughput might have been a bit lower it never did the start-bog down-slowly speed back up pattern it’s doing now.
Hi @rperkins -
I wouldn’t read too much into the behavior of the line graph illustrating bandwidth - in some portions of the test it is affected by the read/write speed of the Raspberry Pi SD card! Looking at the overall average provided at the end of the test should be satisfactory when diagnosing tether bandwidth.
Have you installed the tether diagnostic extension, and checked if anything strange is going on? This is a newer extension, that actually talks to the homeplug modules on tha Fathom-X. If you can share screenshots from this extension, they may help us diagnose your issue!
I can FINALLY update BlueOS and get extensions. I guess my home and cell phone IP addresses are banned by docker.io or something. I’m using a different access point in SoCal at the moment. Interestingly, I only see ~10mbps on my PC coming in over the ethernet connection to the vehicle.
Here are my video settings, H264 as it should be:
Here is a look at the network activity from system monitor in BlueOS:
I wouldn’t be bugged by the look of my speed test results by themselves, but it does bother because:
- they never looked that way before
- it matches up with the generally poor performance I’m seeing, as if I’m operating with a really tiny bandwidth connection, even though I’m not and no matter how burly the cotrol PC I’m using
Possibly unrelated question: does the BR power sense module correctly sense power output when using the OTPS? Mine reports only a bit above 500 watts output at the point where the power supply starts to reduce voltage. I thought this was a 1kw system?
Hi @rperkins
One thing to remove some of the variables is to use iperf3 to test your bandwidth instead of the speed test in blueOS.
It gives you options to control how the speed test is done and more detailed info on what’s happening.
Try running iPerf3 in normal mode, Reverse mode and bidirectional mode.
I believe iperf3 is installed on blueOS by default so it’s easy to run it in the blueOS terminal window.
You have to also install it on your topside control machine, and designate one as server, and the other as client. e.g. topside as server and RPi as client.
I installed it on my topside machine via WSL2 (windows subsystem for linux) but I’m sure you can install it directly on the windows machine also.