Hi All,
We have 10 ROVs running BlueOS. They are each attached via an ethernet tether to their own Vonets Wifi Bridge / Buoy, which connects wirelessly to a router on the side of the pool. The router is connected via Ethernet to the local network. In another room, several laptops are wired to the same local network.
Individually, they all work as expected with a smooth camera stream and hardly any dropouts. Even using around 4 or 5 ROVs simultaneously remains pretty consistent.
When we have 6 or more ROVs going, they all start to drop out and have severe connection issues that makes most of them unusable.
We measured the traffic on each ROV, and found that depending on the camera resolution the traffic was between 5-20 Mbps.
All 10 of these ROVs we’re flashed from the same image (copied from one master ROV). Could there be network settings that need to be unique, but are identical because of the image being copied? We did change the MDNS hostname for each one from inside BlueOS. This is the only BlueOS related thing I can think of.
Has anyone else had success in having several ROVs online at the same time?
Does anyone know of good tools to measure each individual network connection, or maybe more importantly the traffic on the router as a whole?
Is there anything on the BlueOS side of things that could be causing issues with multiple BlueOS machines on the same network?
Is it maybe an issue of RF interference? How could I test for this?
Hi @MarkFromCanada -
Wow! What a crazy cool project! Can you share what you’re using the multiple ROVs operated simultaneously to try and accomplish?
You are likely running into a bandwidth limit as you approach 5+ ROVs, assuming you’re using 100 Mbit rated switching components.
If you ensure all your video streams are using RTSP and not UDP (configurable on Video Streams page) this may help?
A gigabit supporting Mikrotik switch can provide an interface that lets you monitor the traffic on each port, there are many similar options out there, but the terrible UI of Mikrotik does represent an incredibly powerful feature set and configurability.
Hi Tony!
Thanks for the pointers! And yes, they are for a school’s STEM program, we set them up so the students to drive around the outdoor pool, navigate obstacles, scan QR codes etc. The Students really enjoy it!
You are likely running into a bandwidth limit as you approach 5+ ROVs, assuming you’re using 100 Mbit rated switching components.
We looked into this, and did some more testing. We had 9 ROVs online today, and we found that the Bandwidth on our router was ok if we lowered the camera resolution to 800x600 (instead of 1920x1080). We are using a gigabit Meraki router, but still ran into issues at the higher resolution.
We switched them all over to RTSP, and it did seem to help a little. Thanks.
A more serious problem occurred with the cameras though. Sometimes they would get in this state where it would be a smooth stream for a couple seconds, and then give out for a few more seconds. Occationaly if we reloaded cockpit in this state, it would crash the video service completely and we’d have to restart the core container to get it back up and running. We also noticed BlueOS in a state where the video stream was smooth, but the thumbnail service / video streams page was down. We thought this was a network issue at first, but we measured the network traffic and the network (at least at low resolutions) seems to be just fine.
Maybe I should make a different topic for this issue? It’s hard to know if It’s an issue with Mavlink-Camera-Manager, BlueOS, or Cockpit
Hi @MarkFromCanada -
Are you aware of the technical bulletin related to cameras in the ROVs? I wonder if this is what could be causing your issue…
Hi Tony,
Thanks for directing me to the bulletin, although we are using ov5467 cameras with a custom mount, so unfortunately it can’t be that. I’ll keep searching.
Hi @MarkFromCanada -
What version of BlueOS are you using? 1.3 Beta 7 is about to become stable, and contains many fixes - it may be worth trying it out and seeing if it improves your video performance!
Oh that’s great news! I’ll give it a try. Right now we’re using 1.2.6