I am having issues with bandwidth on a 300m tether. The tether is new and has never been used.
Upload speed at surface is between 20Mbps and 27Mbps quite similar for download speed.
The problem:
Going down to 20m the video starts to lag and freeze severely and trying to run the bandwidth test the upload speed is around 0.3-0.7Mbps. Which I think might indicate a broken cable. I still have information from pressure, heading etc. running smoothly.
What I have done:
Plugged the cable end directly to the fathom-X topside interface
Disconnected altimeter from the RaspberryPi onboard.
Swapped cable pair onboard the ROV and topside.
Measured the resistance over the tether. Found around 39 Ohms over all the connections which is within range .
Will also try to swap the tether for a shorter one and see if this helps.
Does anyone have any previous experience with this or a similar problem?
Incredibly thankful for any answers or suggestions
Cheers
Those speeds aren’t great, but should still be usable given they represent the remaining bandwidth while the telemetry and video stream are already being transmitted.
If the cable was broken I’d expect the telemetry wouldn’t be working either, so most likely this is an issue with the companion computer getting overworked (e.g. too high CPU usage, or throttling from overheating or undervoltage) such that it can’t manage to keep running the relevant software while also streaming video+telemetry at the required speed.
A few questions:
If you’re able to reproduce the issue, could you check and report the ‘Companion Computer Status’ info on the companion web interface System page?
Which versions of ArduSub and companion are you running? (also from the System page)
Probably unrelated, but out of interest did you have the lights on when the bandwidth reduced, or was the camera stream quite dark?
If this didn’t help then it definitely seems like the tether isn’t the issue
We will try to replicate the problem later and then check the Companion Computer Status at the Systems page. With the ROV just powered on and on land the CPU load varies between 40-50% and the CoS temp is around 62 Celsius degrees
I am currently running version 0.0.26 for the companion and ArduSub 4.0.3
The camera stream was quite dark close to pitch black when the bandwidth reduced. I did not put the lights on during the decent.
Those values seem fine, which is good to have as a baseline. Issues are more likely to arise when the electronics have been running for a while and have potentially warmed up the enclosure quite a bit, so interested to see how your further testing goes.
Those are our latest stable options, so at least can’t be an issue with outdated software/firmware
I’ve had an issue like this at a previous company, which is why I asked about it. If you’re able to reproduce the issue I’d be curious if turning the lights on restores the camera feed, and whether reducing the framerate and/or setting a manual exposure (through the companion web interface Camera page) is helpful.
My thought process on this is that
either the camera or gstreamer could be struggling to adapt appropriately when auto-exposure significantly increases the exposure time to handle the darkness, since the streaming settings will still be trying/expecting to stream at 30fps, which may not be physically possible if the exposure time is too long, AND/OR
on longer exposure times the camera could use too much power for the USB port/RPi to support - unlikely but not impossible, albeit only with multiple other quite high power USB peripherals connected, and/or if the more general 5V supply has a too low current capacity. Blue Robotics used to use a 5V 3A BEC, but we now recommend and use a 5V 6A BEC instead, because the 3A one had too many capacity issues.
Unfortunately I can’t replicate the issue on my setup, so it seems like both gstreamer and the camera are capable of handling the automatic lower framerate from longer exposure times when the camera is in low light/complete darkness, which makes it more likely to be a 5V or RPi USB power issue, but still isn’t by any means conclusive.
Interested to see how/where this goes, and hopefully we can get your issue sorted out soon
The issue was in the tether! I have no idea as to what was going on there. I changed from a 300m four pair tether to a 100m one pair tether. The bandwidth increased from 20Mbps to just below 80Mbps. and everything is working as it should.
This kind of change isn’t too unexpected, since a 3x longer tether can definitely have some signal degradation, and more wires along the way means more possibility for capacitive noise from the high frequency signal leaking across to other pairs.
In saying that, that doesn’t seem necessarily linked to the ‘working and then suddenly not’ issue you were having, so while we can hope the tether switch fixes your problems do feel free to add any follow-ups if you find it stops working again at some point
