My colleagues and I are trying to replace divers with a BlueROV2 for tracking fish against a blue background, which removes diver-related limitations (depth, rapid depth changes, long tracking periods, and night tracking using red light).
Our first module will automatically detect the fish when the ROV is switched on, and the fish is released in front of it. Detection will rely on a pre-trained deep-learning model (YOLO) trained on many hours of GoPro footage. Then, we initialize a CSRT tracker using the YOLO bounding box to maintain geometric tracking and reduce the risk of losing the fish.
For dynamic, automatic positioning, our current idea is to keep the bounding-box size constant in the video, assuming this will maintain a constant distance to the fish. Stereo or acoustic ranging seems too complex for BlueROV2 and may not be necessary. However, this approach means the ROV will end up closer to smaller fish (1 cm) than larger ones (5 cm), which may disturb them.
We are seeking feedback on similar tracking projects with an ROV, and generally on how to program the autopilot (e.g., via ArduSub). We would also appreciate ideas for distance-keeping that do not depend on fish size.
We also have a few technical questions:
• Is the electrical power available through the tether sufficient to run an on-board NVIDIA Jetson (1–2 A) for real-time detection?
• Would adding a tilt camera below the ROV help reacquire fish that move out of frame?
• Does the DeepWater Exploration camera provide autofocus performance comparable to a GoPro and a suitable lens to prevent losing the fish due to blur?
Hi @eruiz -
Welcome to the forums!
It sounds like you have an interesting, and challenging, goal for your application. In answer to your questions -
Electrical power is not send via the BlueROV2 tether - only data! Suficient battery power is available to run a Jetson for sure, however that hardware won’t fit in the standard BlueROV2 enclosure - you would need to add it in another enclosure, like a water-tight box? However, I’d recommend running your YOLO model on a more powerful computer located at the surface, which could then send commands to control the vehicle. Before even attempting automated control, I would think having a human pilot verify they can actually follow the target fish successfully is a good sanity check for your approach…and gather some more model training data! I’m a bit skeptical that an ROV would be able to keep up with the fish - they can likely swim far faster than1 m/s, and as more tether is spooled out and drag is increased, this top speed is reduced…
The standard camera in the BlueROV2 is on a tilt servo, and can look nearly straight up and down! This tilt angle could be controlled programmatically if desired. You could add an additional camera angle, but if external and wanting to move it, you’ll need a subsea rated servo. Blue Trails makes some, but Blue Robotics will be launching one by Q1/Q2 next year! A downward facing camera is often quite useful for situational awareness.
The DWE exploreHD USB camera does not have autofocus - it is a fixed focus camera! The picture is only 1080p, which is lower resolution than most modern action cameras, however it does have pretty good automatic white-balance. Motion blur is going to be an issue for any camera in low lighting, without a global shutter…
When it comes to controlling the autopilot, you can simulate pilot input with commands via pymavlink, as the examples illustrate here. This positioning would only be as “automatic” and performant as your code can accomplish…
Your bounding box approach may work, but I would think a stereo camera would be best suited, as true depth information regardless of fish orientation/size with respect to the camera could be collected. The Marine Situ C3 camera will be launching on the Blue Robotics Reef before the end of the year! It’s stereo B&W cameras are global shutter…
Thanks a lot for your very informative and detailed answer!
We will follow early-stage fish, so a speed of 1 m/s and less will probably be enough for most species. This is something we have already done diving, which enabled us to gather a lot of training data. I think we will first try to run YOLO on a computer, before trying to add a Jetson for on-board computing.
When the stereo camera will be released in a few months, that might be a very good solution to use the one you indicate, but for the moment, we will first try with two independent camera channels, or just stick to the size-dependent tracking with one camera.
Thanks for the details about the on-board camera… Fixed focus might be a problem if the fish turns back, but otherwise, I think that if focusing on the infinite, we will avoid most problems of wrong autofocus, which could be highly beneficial in the end, as we plan always to keep the same distance to the subject!
Finally, simulating the autopilot will be very useful for our project, we will do so before testing the ROV in the field!
We will probably come back with some more questions in the future, when we will face problems or when we will have to make critical decisions.
Hi @eruiz -
The C3 stereocamera will be launched in ~ 1 week, not months from now!
The fixed focus of both standard and exploreHD camera is really only blurry at extremely close range…
To be clear, pymavlink allows you to simulate pilot input. Having the vehicle navigate automatically requires a position source from an underwater localization system. You can definitely test your code in SITL to verify the expected behavior before taking it to the field though!
My team and I advanced a bit further on this project to test pymavlink and choose the components we will need for our project.
We would like to know what the difference is in terms of vertical speed between the classic ROV with 2 vertical propellers and the heavy kit with 4 propellers. Is the vertical speed greater than 2 cm/s in the first case, and does using two central propellers cause instability problems? Does the greatest weight of the ROV + heavy kit slow down the horizontal speed of the ROV, and if so do you have any estimate?
We were also asking ourselves what the autonomy of the usual battery pack was on the ROV, with and without the heavy kit, if we don’t use external lights (only a camera and the propellers)? Do you think we need to add an additional battery pack, and if so, does it fit in the battery enclosure of the heavy kit: Lithium-ion Battery for the BlueROV2 and Other Subsea Applications. If we need to add a battery pack, what would be the equivalent of the one evoked before near the Maldives, as we need it to be shipped there? I saw on the webpage given before that they mention a reseller in India, but I could not see any battery on their website: https://www.banergy.com/marine
Hi @eruiz -
The standard BlueROV2 with only 2 vertical thrusters can certainly achieve 2 cm/s in vertical speed - did you mean meters per second? In both configurations, vertical speed is similar, but a bit faster with 4x vs. 2X vertical thrusters. Their benefit is slightly lost due to the additional drag of the fairing that goes around them. We don’t list a maximum vertical speed in the datasheet, but I would estimate at least 0.5 m/s is typically achievable.
Weight doesn’t slow down an ROV - drag does! Because the 4 vertical thrusters allow the vehicle to automatically maintain stability in both roll and pitch (instead of just roll as with 2x vertical) faster horizontal speeds are achievable in most conditions - albeit at a slightly higher power consumption. The standard BROV2 with 6 thrusters can sometimes pitch when accelerating, which can make maneuvering a bit trickier.
Battery life for the BlueROV2 depends on conditions - at minimum, in strong currents it may only be 20-30 minutes at 100% gain! At 50% gain, with moderate maneuvering, an hour is typically achievable. When testing with long periods of the vehicle idle, I typically go for almost 2 hours on a battery.
The battery enclosure on the BlueROV2 only accommodates one battery… You could add an additional enclosure and wire the second battery in parallel, but this would be entirely your own user modification, something like this:
Thank you very much for your very useful answer! Do you think an additional 1.15 kg battery pack in a 3’’ watertight enclosure could fit on the 6-thrusters BlueROV, or do you advise to go for the heavy kit as adding that below the ROV will probably largely increase drag?
Do you have any advice or material to make sure the connection between the additional battery and the main battery pack is waterproof? Why did you represent 3 battery packs, do you think one would be sufficient?
Hi @eruiz -
Users have installed a second battery on a standard ROV frame, but it’s tough to do so without impinging on the water path of the two vertical thrusters within the frame. You may require a different bottom frame plate to provide mounting points for the additional clamp and 3" enclosure.
To connect the enclosures electrically, WetLink Penetrators and power cable can be used - you’ll need to source XT90 connectors - these can be a pain to solder as they require a lot of heat!
The graphic was made previously for someone requesting information on connecting 3 batteries. Without knowing more about your application, its tough to say whether 2 batteries are sufficient - even one could be? It’s fairly fast and easy to swap batteries in the field, and certainly less complicated. When connecting batteries in parallel, they MUST be within ~ 0.1V of each other to avoid potential damage.
Thanks @tony-white for this additional insight! This is exactly what I was thinking as the two vertical thrusters are in the middle of the ROV.
The ROV needs to perform at least 10 continuous tracking for 10 to 20 min in the open ocean until -100m, so using at least 2 batteries will probably be necessary. Changing batteries on the boat with an oceanic swell might be a bit tricky, even though it’s good to know this can be done fairly fast, thanks for this advice!
Hi @eruiz -
I don’t think two batteries are necessary for such a mission, especially if only using a 6 thruster ROV. I you’re in really strong current, and fighting it at 100% gain, the battery should last 20-30 minutes, plenty of time to dive to 100m and drive around. I’d guess 40 minutes would be possible in most conditions to a dive to 100m, with an 18AH 4S battery - swapping on deck is common, and the same “motions” as connecting the battery to power the system up (you’ll be doing this already “in the swell…”)!
Thank you very much for your feedback on this question! We will probably do as you said, as I was not thinking we would need to switch the connector manually (I thought both batteries could be connected to the ROV in the meantime to avoid resurfacing for exchange).
Hi @eruiz -
You could have both batteries connected in parallel before starting your dive. Connecting them, or a single default battery requires opening the battery enclosure(s) and connecting the xt90. My point is that when doing so, the container is open do swapping a battery is no more complex then turning on the vehicle…
My colleagues and I come with another question: do I you think it would it be possible to perform a live treatment of the two video channels obtained from two DeepWater Exploration exploreHD USB cameras on a computer on the boat above, to perform the distance estimation of the bounding box? If so, are you aware of any algorithm that would allow us to achieve that (we cannot afford buying the 4000$ stereocamera)?
Hi @eruiz -
Unfortunately it’s not as simple as hooking up two cameras to determine stereo-derived range information!
First, both cameras need to be mounted a fixed, known distance apart, very securely. They need to be as parallel as possible. They then need to be calibrated - see more on that here.
The bigger issue is capturing images from the cameras in a time-sync’d fashion. DWE offers a stereo camera, more expensive than the MarineSitu option just launched, that lets a user capture synchronized pictures/video. Simply using USB camera devices to capture images at the exact same instant (necessary for stereo depth to work!) can be nigh impossible…
I’ve not experimented with the software side of capturing syn’c images - I would say some simple experiments with both cameras pointed at a stopwatch readout on a display would let you determine how close to the same instant an image can be captured from both cameras! If close enough in time, you may have good results- please share if you attempt!
Hi @tony-white ! Thank you for this very honest reply! I did not imagine that synchronizing cameras would be that complicated, we might do the test?
Another unrelated question: do you think the BlueRov could carry a 350 * 76 cylinder weighing 3.5 kg (Valeport 803) without losing too much buoyancy or speed (increased drag?), or should we use the heavy kit?
For the camera synchronization, what if we perform live distance estimation? We would treat both video channels in the meantime, therefore ensuring they are synchronized? I don’t understand why it would be complex to do it this way?
Hi @eruiz -
I’d definitely recommend a test capturing both video streams. Any distance estimation, live or otherwise, relies on taking the image from both cameras, taken at the same instant, and comparing the pixel location of the same point in both. This difference in x,y coordinate of a fish nose for instance, is how the distance from the cameras is calculated. If the pictures were not taken at the same instant, and the fish was moving relative to the ROV, that calculated distance will be based on pixel coordinates from one image that are from a different moment, and so you’ll get a wacky result!
As for your payload - that water current sensor is interesting - it’s like a DVL, but it tracks the water rather than the bottom it would seem? Since it weighs 3.5kg in water, and the ROV payload capacity is less than that, you’ll need to add additional buoyancy to keep your vehicle neutral or slightly positively buoyant. It needs to be mounted pointed up or down (?) so I don’t expect the drag impact to be large. A standard ROV could likely move it well, but a Heavy configuration will keep the vehicle perfectly level with active stabilization while doing so… and so you’ll likely be able to travel faster in this way!
Hi @tony-white ! Thank you again for your very clear and precise answer. We now have all the elements in mind to order the right elements for the BlueRov, and to correctly start our tests. I will keep you informed if my colleagues and I test a stereo system with DeepWater cameras. Thanks a lot for your help in general!
