Lost communication, multiple cameras

Hi,

I am having some issues with lost communication after a few minutes of operation.

I have connected two Low-light HD USB kamera and one DW exploreHD to my BlueROV2.
This works for a while and gives me three pictures. I have tried to connected directly to the Raspberry and connection through a powered USB hub.

But, when I have all three cameras connected I loose communication to the sub after 5-10minutes. After this I’m not able to communicate with the companion computer at all. This happens when the cameras are connected either way (directly or through a powered USB hub). If I only use two cameras in total everything works fine!

I have tested with BlueOS beta 9 and 11. And I am using DWQGroundControl.

Anybody else who have experienced this and have any tips on how to solve this?

Hi @Stihus,

The main reasons I can think of for this are:

  • too much bandwidth usage (may be able to be reduced by changing the camera resolutions, or the encoding bitrate of the exploreHD if that’s possible; if you want you can try bypassing the Fathom-Xs and tether with an ethernet cable directly from the Raspberry Pi to your topside computer to see if that helps at all)
  • insufficient power supply for the Onboard Computer (less likely when using a powered hub, although it may be worth trying to power the hub with a different regulator/supply to the Raspberry Pi, if you’re not already)
  • the Raspberry Pi could be overheating (are you testing in water, or in air? do you have an aluminium enclosure, or acrylic? are you on a Raspberry Pi 3 or 4? on Beta 11 you can check the CPU usage and temperature in Tools / System Information / System Condition)
  • there could be some kind of bug in the camera manager that’s causing it to have trouble when streaming more than 2 videos (that’s something we can look into if the other options don’t seem to be the case)

Assuming the web interface is open when you lose connection, are you getting any warning notifications before it shuts off?

Thanks for the tips! I will check all details on monday when back at the workshop.

I have not tested with ethernet cable directly between Raspberry and topside computer. But I have done a quick test with fiber connection between the sub and topside computer (fathom-x cards removed). Did loose communication then also.

When testing inn air I have powered the hub from the wall socket (not the sub). I have not installed the hub in the ROV for test under water, could this be worth a shot maybe?

I get the fault both inn water and in air. I have acrylic enclosure. Running Raspberry pi 3 and have tried both beta 9, 10 and 11.
The highest CPU temperature I have seen is 65 degrees celcius, this shuldnt be to high i guess? CPU usage is around 50% average but occasionally peaks to 100%.
Is there a possibility that I am above the processing capasity of the Pi? That it actually can’t handle three video stress? Dies the Raspberry pi 4 have better specs, and is ut compatible to do a direct swap to replace the Raspberry pi 3?

I guess this could still be a possibility yes.

I have not noticed any alerts yet, but I will study the warnings closer when back in the workshop.

Im also planing to do a complete reinstall of BlueOS. Do you recomend formating the sd-card bedre flashing the image again ? Is there a special format you recomend to choose when formating the SD-card?

Maybe a power issue? I have a vague memory of my Pi failing when testing multiple cameras plus 2 sonars on USB. I don’t remember exactly if I only lost the USBs or all connectivity to the Pi…

Next time this happens, try to ping it and see if at least that responds.

If you connect it to wifi, it works as an extra connection method in case ethernet goes down =]

My gues was also power issue, but I still have problems when I power the USB devices from a seperate powered USB hub.

Did you not find a solution om your case?

Tried to ping the Raspberry, no response.
But I will try do connect/ping it over Wi-Fi!

Im now considering to move over to Raspberry pi 4 to see if this can handle it better.

Ok, it’s likely not a bandwidth issue then.

The electronics aren’t intended to run for extended periods in air because they can overheat - the design expectation is that the enclosure will be cooled by the water around it when operating. If you’re testing in air then I’d suggest doing so with the enclosure off/open if possible, and ideally a fan to push air over the electronics.

That should be ok, that’s still below the throttling point

That may be ok, but it depends how long those peaks are for, and what’s causing them. If something requires more processing than the pi can provide then it could conceivably crash while trying to handle that, which may be what you’re seeing.

Maybe, although H264-encoding is designed to put minimal stress on hardware that’s just passing it along, so if it’s a processing capacity issue then I expect it’s caused by something other than the video streams (or 3 video streams + everything else could be too much in total).

The Raspberry Pi 4 definitely has better specs, and if you’re using the BlueOS Beta (which you are) then yes, it can be directly swapped in. It may be difficult to get one though - there’s a shortage in several places at the moment.

Flashing on a new image should handle that by itself - formatting beforehand shouldn’t make a particular difference :slight_smile:

Hi,

Finaly got the chance to do some more testing.

I changed out the USB hub I was using because the first one was not stable at all. Cameras where moved around to different UDP addresses after reboots, and often had to be physically disconnected and connected to show up after a reboot.
The hub I’m using now is a Delock 62650 USB hub. This is much more stable, but still I’m loosing communication after a while when streaming three cameras.

I’m awaiting a new Raspberry PI 4 B, hopefully this will get me better results.

After loose of communication over ethernet, wifi connection is still available and I can connect to the Companion through a web-browser, and ping the companion to it’s WiFi IP address. But, the video manager page is not accessible, no USB devices show up.
So I gues this is proof that the Raspberry is not completely shutting of.

Been keeping a close eye to this, and the load on the processor is mainly around 40%. Only for max 10% of the time it peaks to 90-100% load. So i guess this is not to heavy for the PI.

Been keeping a close eye to this, and no it no warnings before it shuts off.

I now have tested the sub submerged with a separate power supply to the USB-hub. Still loosing communication.

Flashed a new Beta 5 to the SD card and I also tested BlueOS B.E.T.A 12, still not stable with three cameras.

Good! check http://companion.local:6030 and click the kernel tab. Check if there are any obvious messages in there.

2 Likes