Hi BlueRobotics Team and User Community,
We’re re-deploying our custom BlueROV2 to the bottom of the Monterey Bay (900m deep!) for our Ocean Discovery program. This time around, we really wanted to figure out the Tele-operations piece so we can distribute the connection to students around the world. We have successfully figured it all out except for one little bug. The video seems to have a compression bug that we can’t figure out. I have included a video link below that shows our issue. The same bug does not happen when we are directly/locally connected to the ROV. We’re hoping that someone here can offer some suggestions.
The way we are connecting is through setting up a VPN using ZeroTier. We’ve successfully used ZT for land based robots that also utilized a Raspberry Pi but have always used web sockets for the video and not UDP. UDP should prove to be much better if we can figure out this video issue.
Is there some settings we should try to adjust? We’re open to all suggestions!
After reading your post i went and did some research into UDP, VPN’s and ZeroTier.
900m = 1400 psi…Ouch! Are you sure your WTC can handle that pressure?
Not being a computer guy, more of an engineer/fabricator actually, i tend to do things differently.
My WTC is an 8" diameter, 3/8" wall, 3’ long aluminum pipe, with a custom-blown acrylic camera dome (1/2" minimum wall thickness at the apex of the dome). I put 8 cameras in my chest-freezer-sized ROV, connected them to a standard security-system DVR, then access all 8 realtime video feeds by signing in over my LAN. My main pan/ tilt camera is a Panasonic SDR-T55 camcorder ($50 used off ebay) with only 704x480 resolution but 78x optical zoom. All the other cameras are cheapo Eachine $20 FPV micro-cameras from Gearbest. The DVR in the ROV is plugged into a 4-port router, as is the R Pi that controls the Pixhawk. Several people can sign into the DVR feed at once, and a computer guy could probably figure out how to stream the feed from the topside computer to students at a University.
The Pixhawk is a great autopilot but doesn’t have good aux control outputs, so i connected an SSC32U servo controller to the R Pi via I2C. This gives me control of 32 proportional analog ‘servo’ channels. Those control my lights, manipulators, camera pan/ tilt, ballast adjustment/ drop, adjustable buoyancy, etc.
The router connects to an eKL ethernet over coax adapter, which allows communication over RG59 coax cable, with speeds of 100Mbps at 300m, to 30Mbps at 2000m, plenty enough for realtime video feeds.
I don’t know how to do it the way you are asking help with, but my way works fine for me.
Thanks @williangalvani That did mostly fix it. I’m now only getting the glitch once every couple of minutes. Turning down resolution seems to also help in some cases. We’ve been testing this with multiple people in different countries/locations. My speed according to Fast.com is 210Mbps with a latency of 6ms unloaded and 13ms loaded and an upload speed of 260Mbps. Its pretty fast.
My colleague tested with his 20Mbps connection and had the same results.
If there is anything else we can do let me know. Should we try a different type of compression?
Thanks for the feedback @Oddmar . Our Blue Robotics 6" Aluminum WTC can handle the pressure. We’ve already had it down there once for about 8 months continuous without any leaks.
I guess those are lost packets, there’s not a log we can do to fix it. the only other thing that could be done is not show the bad frames (freeze the video instead). But that would require rebuilding QGC.
My colleague did some research and believes that its having an issue with some I-frames… We’re going to move the ROV to the school tomorrow where the internet connection is really fast and see if the issue is just with my home internet. I have a fast download speed here but only a 15Mbps upload speed. The speed at the school should be similar to when it is deployed.
So to conclude this thread, the removing “low latency” mode in the QGC preferences did mostly fix the issue. To resolve the last bit, we moved the ROV from my home network to a internet connection with a much better upload speed and that resolved the rest of the issues assuming the remote client has a decent download speed. When deployed subsea, the ROV will have a fiber connection all the way back to land so I suspect that remote users should have a good experience. Will keep you posted!
Nice! I’ve operated ROV’s remotely frequently, and this issue is always a bear. It crops up occasionally still, and you’ve found all the right fixes. The only thing not mentioned is that sometimes increasing the power of the hardware running your vpn, or not sending video over the vpn (obv.) can help as well.
Looking forward to future releases with new video methods!