My colleagues and I are trying to replace divers with a BlueROV2 for tracking fish against a blue background, which removes diver-related limitations (depth, rapid depth changes, long tracking periods, and night tracking using red light).
Our first module will automatically detect the fish when the ROV is switched on, and the fish is released in front of it. Detection will rely on a pre-trained deep-learning model (YOLO) trained on many hours of GoPro footage. Then, we initialize a CSRT tracker using the YOLO bounding box to maintain geometric tracking and reduce the risk of losing the fish.
For dynamic, automatic positioning, our current idea is to keep the bounding-box size constant in the video, assuming this will maintain a constant distance to the fish. Stereo or acoustic ranging seems too complex for BlueROV2 and may not be necessary. However, this approach means the ROV will end up closer to smaller fish (1 cm) than larger ones (5 cm), which may disturb them.
We are seeking feedback on similar tracking projects with an ROV, and generally on how to program the autopilot (e.g., via ArduSub). We would also appreciate ideas for distance-keeping that do not depend on fish size.
We also have a few technical questions:
• Is the electrical power available through the tether sufficient to run an on-board NVIDIA Jetson (1–2 A) for real-time detection?
• Would adding a tilt camera below the ROV help reacquire fish that move out of frame?
• Does the DeepWater Exploration camera provide autofocus performance comparable to a GoPro and a suitable lens to prevent losing the fish due to blur?
Hi @eruiz -
Welcome to the forums!
It sounds like you have an interesting, and challenging, goal for your application. In answer to your questions -
Electrical power is not send via the BlueROV2 tether - only data! Suficient battery power is available to run a Jetson for sure, however that hardware won’t fit in the standard BlueROV2 enclosure - you would need to add it in another enclosure, like a water-tight box? However, I’d recommend running your YOLO model on a more powerful computer located at the surface, which could then send commands to control the vehicle. Before even attempting automated control, I would think having a human pilot verify they can actually follow the target fish successfully is a good sanity check for your approach…and gather some more model training data! I’m a bit skeptical that an ROV would be able to keep up with the fish - they can likely swim far faster than1 m/s, and as more tether is spooled out and drag is increased, this top speed is reduced…
The standard camera in the BlueROV2 is on a tilt servo, and can look nearly straight up and down! This tilt angle could be controlled programmatically if desired. You could add an additional camera angle, but if external and wanting to move it, you’ll need a subsea rated servo. Blue Trails makes some, but Blue Robotics will be launching one by Q1/Q2 next year! A downward facing camera is often quite useful for situational awareness.
The DWE exploreHD USB camera does not have autofocus - it is a fixed focus camera! The picture is only 1080p, which is lower resolution than most modern action cameras, however it does have pretty good automatic white-balance. Motion blur is going to be an issue for any camera in low lighting, without a global shutter…
When it comes to controlling the autopilot, you can simulate pilot input with commands via pymavlink, as the examples illustrate here. This positioning would only be as “automatic” and performant as your code can accomplish…
Your bounding box approach may work, but I would think a stereo camera would be best suited, as true depth information regardless of fish orientation/size with respect to the camera could be collected. The Marine Situ C3 camera will be launching on the Blue Robotics Reef before the end of the year! It’s stereo B&W cameras are global shutter…
Thanks a lot for your very informative and detailed answer!
We will follow early-stage fish, so a speed of 1 m/s and less will probably be enough for most species. This is something we have already done diving, which enabled us to gather a lot of training data. I think we will first try to run YOLO on a computer, before trying to add a Jetson for on-board computing.
When the stereo camera will be released in a few months, that might be a very good solution to use the one you indicate, but for the moment, we will first try with two independent camera channels, or just stick to the size-dependent tracking with one camera.
Thanks for the details about the on-board camera… Fixed focus might be a problem if the fish turns back, but otherwise, I think that if focusing on the infinite, we will avoid most problems of wrong autofocus, which could be highly beneficial in the end, as we plan always to keep the same distance to the subject!
Finally, simulating the autopilot will be very useful for our project, we will do so before testing the ROV in the field!
We will probably come back with some more questions in the future, when we will face problems or when we will have to make critical decisions.
Hi @eruiz -
The C3 stereocamera will be launched in ~ 1 week, not months from now!
The fixed focus of both standard and exploreHD camera is really only blurry at extremely close range…
To be clear, pymavlink allows you to simulate pilot input. Having the vehicle navigate automatically requires a position source from an underwater localization system. You can definitely test your code in SITL to verify the expected behavior before taking it to the field though!