BlueROV2 with visual navigation system

Hi All,

we have been working on automating our BlueROV2 using a visual navigation system. Our prototype system has been tested in a pool and we are working toward some field tests soon. The hardware and software components are diagrammed below, and some videos demonstrating early trials are available at (https://www.youtube.com/watch?v=tOlrNdwt2I4, BlueROV_2 - YouTube).

The key hardware components are a Stereolabs ZED mini interfaced to an NVIDIA Jetson TX1, which communicates to the BlueROV2 Raspberry Pi via Ethernet (substituting for the topside computer). All of the visual navigation software is run on the TX1. We’re using a slightly modified version of ORB-SLAM2 for localization and mapping (achieving ~10 fps with VGA images) and a basic trajectory controller to navigate predefined routes. The camera/computer enclosure was made by Sexton (http://www.thesextonco.com). The videos above show the ROV completing a ‘lawn mower’ pattern as our ultimate goal is to use the system for mapping marine ecosystems.

Brian

9 Likes

Well done! Do you think it is possible to fuse in a USBL position while navigating to increase the accuracy of the mower track and starting point?

Fantastic work!

1 Like

Looks good. I am aiming to do the same thing.

Fantastic :+1:. I definitely want one when testing is completed please

Rgs
Grant
Oz

Great work

A few questions for either / both @brianh and @deepsouth to get their options for my own (all be it poor coding knowledge) benefit

How well do you think this style system would work as a monocular system (with the camera down facing and assuming it is getting fed reasonable bottom images)

In your opinions is the IMU utilised in standard BlueROV2 Pixhawk (the Invensense MPU 6000 sample rate up to 8kHz angular rate ±250, ±500, ±1000, and ±2000°/sec) near suitable if not what sort of range of accuracy and sample rate do you believe is necessary to get usable data to determine the camera pose (I couldn’t find what IMU is running in the ZED mini but it looks like it is reasonably up specced from a twitter quote “The IMU of the ZED mini is running at a fixed 500Hz, both the RAW and the fused data are available from the SDK” )

Not taking anything away from either of your fantastic work (as it makes great sense in driving the ArduSub project towards AUV navigation and is a huge plus in general) But given the BlueROV2 is a ROV and that it has a tether back to the surface and is typically driven by a laptop (and yes Andrew I understand you are running the control as a web server down on the ROV so in your case it may be lacking this topside computing grunt) Would it be a reasonable workable solution to have the ORB-SLAM2 for localization and mapping running on the topside computer and even if there was not complete loop closure these flight tracks would be of assistance to any operator

Regards

Scott

Hi Scott,
Most SLAM systems will work with monocular systems (ORBSLAM does and I’ve tested it with some underwater video sequences and it performed reasonably). Using only a monocular camera the path/map will lack an absolute scale, but depending on your needs that may not be a major issue.
The IMUs on both the BlueROV and ZEDmini are more than capable of supplying fairly accurate camera pose, and the IMU update rates far exceed the camera frame rates. More recent SLAM systems (VINS-Fusion, OKVIS, etc) make use of the IMU data to partially infer position, velocity, and acceleration changes and I imagine these systems would be more sensitive to the quality of the IMU (but we haven’t tried them yet).
Finally, you could definitely run a SLAM system on the topside computer to aid navigation. you’d have to get the video, IMU, etc data up the tether but the topside computer would typically be more powerful and capable of running SLAM at high frame rates (which is critical) than an embedded computer.
thanks, Brian

From the mapping side of things with a camera looking straight down, not sure if you have seen this? Its very cool


Thanks Brian and Andrew for the replies

Thanks for the link Andrew I have been playing around with underwater photogrammetry for 5 year (see below) and is the main driver for the BlueROV2 so as not to be as limited as a diver with depth and duration

Scott

3 Likes

Very impressive Scott!

  • What did you use for the image above?
  • How far have you got augmenting the technical capabilities of the BlueRov2?

Hi Andrew
That image was done with a standard GoPro but not from a ROV (just me as a free swimming diver)

I have done some photogrammetry from the ROV and good coverage is the hardest thing to get it working (hence the intrest in visual navigation ) I have a couple of projects in deeper water coming up very soon (110m and 145m) so outside the Waterlinked system

And some reference system of coverage at those depths would be great

Re augmenting the base unit at this stage just upgraded housing and 200m detachable tether

Looking for a ethernet hub and decent camera (considering a sony qx1 camera mainly for form factor )

Scott

Thanks Scott

The camera looks good. Just checking you have seen this produce. Gigabit Ethernet switch powered by USB
not sure if its what you are looking for


Thanks Andrew

Looks a nice little unit (2 x 6.2 x 7.9 cm) so easy to find a place to put it

Still tossing around different configurations a bit as I need to get more battery life into the unit hence push “stuff” around a bit in the frame on a recent dive to 118m I used 14.8Ah of the 18Ah battery for just under 10 minutes on the bottom (seriously looking at a burn wire system (similar to Building the world's first consumer deep-sea dropcam) to add weight to the unit and jettison it when at or near the bottom (may take out about 4Ah of consumption and we can skull drag the unit by it’s tether back to the surface)

Ideally I would still love to get a decent camera (say at least greater than 12MP stills with good Iso) that would fit in a shortened 2" housing rather than a cut down 3" (think some of the Drone cameras [ZENMUSE X5 or X7 if you can get rid of the gimbals]) if I have to use a 3" housing then the Sony RX0 would also be in the mix

Scott

Hi Scott,

Really cool photogrametry.

I am able to squeeze my HD cameras into a 2 ich housing with acrylic flat cover but GigE cameras are too big.

Also at that resolution, optics are going to be really important. It would require a proper underwater lens I think.

I am also completing an upgrade of my GigE / Fiber - Ethernet 4 port and 8 port system so I will be testing/building GigE cameras later this year.

Would a surface supply system not be preferable instead of using a battery?

Cheers,
Etienne

The Ethernet switch is even smaller when you take the cover off :slight_smile:

Thanks Etienne

HD is still pretty low resolution it is always best to shoot stills (and always straight down)

  • 720p (1280×720 px) = 0.92 MP
  • 1080p (1920×1080 px) =2.07 MP
  • (4K UHD) 2160p (3840 × 2160 px) = 8.29 MP
  • Shoot stills with an old GoPro (Hero 4 @ image every 0.5 seconds) 12 MP

Optics are important the better the base image the more likely you are to get decent results although this

produced

The software (I use Agisoft) is pretty good at removing barrel distortion (have a look at a standard GoPro and how much it puts in http://img1.skisoo.com/2012/09/fisheyeCorrectionFeatured.jpg)

The biggest issue I have found is ISO and blur from motion hence better sensors and more light (sort of) as a solution

Etienne my biggest issue is $'s

I’m not a commercial venture just an individual looking for and documenting shipwrecks so 's always come into it the need to spend money for fuel and time on the water searching for new wrecks V spending money on the SSS or the MB or the ROV they all compete for the few small 's and what will give the biggest overall bang for buck your

A second battery housing and something like your battery holder (sorry mate most likely a home build) are the most likely the 1st port of call (as well as the burn wire system)

Thanks

Scott

Finally Sorry @brianh I don’t wish to hijack you thread again I really like what you are doing

lol no worries at all, I post these things on here so those who want to home build have ideas from what is being done and those who want to buy can :wink: