Recently had a fun time with @tony-white and a crew from the Sexton corporation in Kailua-Kona, Hawaii, assisting them with their cameras. The calibration they had wasn’t working, so I built some software to assist them in monitoring and recording the BlueROV while it was underwater and calibrating the camera. Really fun underwater computer vision project. I definitely think it’s the start of a larger set of projects I’ll work on in the future.
Thanks @jackmead515 !
We’re currently evaluating the ZedHead for the BlueRobotics 3rd party product “Reef” - if this is something you’d be interested in, please share your application and any questions you may have!
I have always been aware of the direct application of stereoscopic cameras for underwater photogrammetry, although it can also be done with conventional 4k cameras. It would be interesting to know if it offers advantages for this purpose
Hi @juanjepalomeke -
The primary advantages are twofold - a reduction in processing time when generating the 3D photogrammetry model, and that model being generated too-scale. When done with a single camera, the size of objects captured is unknown!
The code shown is generalized yes, but each camera may have a different calibration matrix and raw video format / resolution, as well as a different baseline camera separation.
Yes to back Tony up, the intrinsic and extrinsic properties of each stereographic camera are highly unique. And given even slight adjustments to resolution, baseline, recording medium (air, fresh water, salt water, etc), will greatly adjust the quality factor.
In addition, the calibration images may have to be collected several times over as getting a quality set of images is an art-form rather than a very applied approach. This is due to the need for the estimation model to have a generalized set of points distributed over the image plane, at all different angles, and depths from the camera, the lighting conditions as they influence the template matching algorithm and how it approximates the the positions of the checkerboard image, and even the checkerboard geometry itself as that has to be configured correctly. (9x6 or 11x8 etc)
In a controlled environment this can be easy! But if you notice in the tutorial, some of the checkerboard points in the video are distorted due to medium noise and lighting conditions. There are various denoising and equalization filters that could be applied to minimize this, but they just introduce more tunable options.
Our checkerboard template we used comes with a high precision and the plane is very smooth. Any slight distortions or warping of the checkerboard plane (for instance say due to printing out a checkerboard pattern from a printer) will also affect the calibration accuracy.
When looking through your tutorial you mentioned that you used an embedded NVIDIA GPU, it can do real-time depth estimation and point-cloud generation… Which GPU did you use?
Thanks for reading! The Zed 2i Stereo Camera was connected to a NVIDIA ORIN NX 8GB that COULD be used to do stereo processing. OpenCV provides a cuda module with block matching that can be executed on an Nvidia and that could run natively on that device.
However, for this tutorial, I only ran this on CPU. With the resolution I was using (can’t remember now) I was getting around 15FPS on my AMD Ryzen 5 3600 6-Core Processor.
Hi @c.pick -
Welcome to the forums!
Just to clarify, the hardware used was indeed a Zed2i camera, but it was connected to a ORIN NX 8GB inside the Zed Head
Hi @tony-white
I’m definitely interested in ZedHead. A potential application I’d like to explore is using ZED 2i camera for real-time 3D underwater mapping and photogrammetry
Hi @ryan354 -
That should be possible! The camera can be used with a topside laptop that has an nvidia graphics card with the Zed API, and @jackmead515 developed this code that can be run on the ZedHead to provide a simple web page with live video stream and recording functionality. The onboard nvidia CPU is ready to run your code - maybe even real time SLAM ?
Yes, exactly! The ZED camera web page @jackmead515 could potentially be developed as a BlueOS extension, making it easier to integrate, right?
My goal is to turn the ZED camera into a local video streaming system which requires hardware acceleration to handle real-time encoding and decoding efficiently. The onboard NVIDIA Jetson can take care of encoding, while a high-end NVIDIA GPU on the topside laptop can handle decoding and finally, with ZED API—depth sensing, positional tracking, and spatial mapping—plus the addition of high-power lumen lights, I think it could potentially achieve VSLAM, similar to what Voyis and Uvision 3D Scanner offer. This could make surveying much more effective and precise.
Hi @ryan354 -
That code is better suited to running on the hardware in the Zed - it is easy to embed the web page in Cockpit via the iframe, and have full screen stereo vision with record controls while driving the vehicle normally, or picture in picture.
I’d expect achievable results to be quite similar, yes! I haven’t been able to connect an nvidia GPU machine to the hardware, but would be happy to share recordings if they’re useful to you.
Sharing the recordings here. Please share any results!
Associated with that footage is calibration data, although it doesn’t match the formatting that zed-head uses for their calibration files that come with the camera (and are intended for use in-air.)
The stereo_parameters.json is the tuning configuration for the cv2 block matching, points.parquet is the parquet file with all the points of each detected checkerboard, and calibration.pkl is the parquet pickle file with the stereo camera calibration: you can read with pandas.read_pickle()
