Efficient video options?

General Discussion
1

BR2 users question :

Has anyone found an efficient hardware/software solution to simultaneously stream a low-resolution H.264 video feed from a Raspberry Pi to a surface computer, while also recording high-resolution video or capturing raw images locally on the Raspberry Pi ?"

More Context:

  • I only need a resolution of 448p for the video feed displayed on my 7-inch screen / low spec computer, which helps conserve bandwidth for an additional network component using the tether.
  • I am not looking to transmit real-time 4K video through the tether; instead, I want the Raspberry Pi to locally record the high-resolution video.
  • I also need the capability to take RAW-quality photos to get the most of the sensor’s performance and save them directly on the Raspberry Pi. My tests have shown that H.264 encoding significantly reduces image quality in low-light conditions. A RAW frame produces far better results.
  • I know that the low-light USB BR2 camera I am currently using cannot output a YUY2 RAW feed or support simultaneous dual-viewing channels.

Progress So Far:

I have discovered a high-resolution camera with dual outputs: one USB UVC feed and one HDMI output. The HDMI signal can be connected to a video acquisition device to generate another USB UVC stream.

Does anyone has a better hardware/software idea using only one camera on board and not exhausing the raspberry Pi 4 processing capabilities ?

Thanks ! :smiley:

2

Hi @Lyra -
As far as I know, it is not possible to capture “RAW” images from the standard USB low-light camera - this is because the camera itself streams H264 encoded onboard the device. You can also access the mjpeg or YUV stream as you mention, but this is not likely to provide different results in terms of quality…

Sharing the camera you’re referring to would be helpful! It may be possible to capture the HDMI stream with a USB capture card, but I don’t expect this to be that high-resolution…

Blue Robotics is getting quite close to launching an upgrade kit for the BlueROV2 that provides a significantly better, 4K camera, with amazing low light performance. Stay tuned!

2 Likes
3

As I understand it direct onboard recording is not too crazy on the processing front, but can be taxing on I/O bandwidth and is best done with high speed storage that can support many read and write cycles (e.g. a high class, high capacity SD card, or an SSD via a HAT module).

That said, there are also cameras that support onboard recording internally (so the Pi could process only the preview stream), which could be controlled via MAVLink camera controls, or whatever API/control signal they happen to support.

Onboard recording is something we’re hoping to provide decent support and guides for at some point, but there are currently higher priorities in our development and testing pipelines.

2 Likes
4

Hi @tony-white

Thank you very much for your answer !

Some ideas playing with the BR2 low light USB camera :

You can also access the mjpeg or YUV stream as you mention

My apologies, I completely missed the point. You’re right, the BR USB Low light camera does indeed offer a YUV2 stream.

ffmpeg.exe -list_options true -f dshow -i "video=USB Camera"

display the best YUV2 resolution as :

pixel_format=yuyv422  min s=640x480 fps=15 max s=640x480 fps=30

If I’m not mistaken, YUV2 is a “RAW” image feed( Y′UV - Wikipedia)

While 640x480 isn’t a high-resolution format, it’s still quite manageable. For live feeds on small screens, it’s actually ideal—after all, TV hasn’t always been 4K or HDTV.

The bandwidth required for a YUV2 stream at 640x480 resolution and 30 fps is approximately 147.456 Mbps. Since the tether can only support around 30 to 80 Mbps, sending the YUV2 feed directly to the surface isn’t feasible.

A possible solution would be to capture the feed in the YUV2 format (640x480), then create a dual virtual video feed using a tool like v4l2loopback. One stream could be encoded by the Raspberry Pi as 640x480 30 fps H.264, while the other remains in the YUV2 format for easy screenshot extraction and saving in the PPM format ( Portable pixmap — Wikipédia).

Once this setup is complete, I will be able to receive a 640x480 30 fps H.264 video feed while simultaneously capturing PPM screenshots from the YUV stream by sending a capture command (via SSH or Telnet).

You can also access the mjpeg or YUV stream as you mention, but this is not likely to provide different results in terms of quality…

An interesting test using the BR2 low light USB camera with poor lighting:

frame to frame - PPM screenshots
frame to frame - PPM screenshots909Ă—368 205 KB

I find that the YUV2 frame offers sharper edges, fewer fuzzy areas, and less noise. This is expected, as the H.264 stream is essentially the YUV2 stream processed and compressed by the camera’s internal hardware encoder. For capturing still images, the YUV2 frame seems ideal as it provides a lossless stream.

Some thoughts about the dual output camera :

Sharing the camera you’re referring to would be helpful! It may be possible to capture the HDMI stream with a USB capture card, but I don’t expect this to be that high-resolution…

ELP 4K USB Webcam IMX415 Color CMOS Sensor 3840X2160 30fps H.265 H.264 HDMI USB C Camera With No Distortion Lens [ELP-USB4KCAM01H-H120] - $97.34 : ELP USB Webcam

Definitely not a low-light camera, but it offers dual output!

One possible setup would be to use the HDMI stream through a USB capture card to output a 640x480 30 fps H.264 stream via the Raspberry then the tether. Meanwhile, the USB stream could be used for local recording by the Raspberry, either capturing 4K H.264 video or taking 1920x1080 YUV2 snapshots.

With this configuration, if I’m not mistaken, you get the best of both worlds while offloading all encoding tasks from the Raspberry Pi.

Other things :

Blue Robotics is getting quite close to launching an upgrade kit for the BlueROV2 that provides a significantly better, 4K camera, with amazing low light performance. Stay tuned!

That’s fantastic ! Do you know if the 1920x1080 H264 stream could be better than the existing low light USB camera ?

5

Hi @Lyra -
The RadCam outputs a clearer picture at 1920x1080 and 3840x2160 (4k) , and provides user-controlled manual focus and 4x (optical) zoom. It uses a much larger sensor, and thus has improved low light sensitivity - if the scene is low in motion or ghosting can be tolerated, it can see shockingly more detail in even lower light than the low-light USB camera.

I had no idea the YUV2 stream would provide a difference in quality - although I’m not sure how much of a difference the average user would perceive, I’m sure machine vision / photogrammetry applications could benefit.

Generally using cameras with HDMI outputs seem a poor solution for the Raspberry Pi, but if another SBC like a Orange Pi Pro 5 was used perhaps it would be more realistic to record high-resolution files onboard? Faster storage as Eliot mentioned would definitely help.

6

Hi,
We can recommend this solution for streaming video with very low latency and several simultaneously streams.
A very known technology is a HDMI to set encoder like this one:

We use this for our drones sending live low latency video streams over an ip radio link.

You can setup various resolution and bit rates.
Also audio channel is available.
In this way you dont need to use the rpi for this task at all.
It’s widely used for video streaming events.
Latency can be as low as 16 frames

Pob747

7

Hi @pob747
That’s a cool device! I can see it being useful in situations where an hdmi output from the camera is standard, like a DSLR in a subsea enclosure (5" with dome!)

Since Blue Robotics vehicles already have the Pi, and Cockpit is setup to receive video from it, it may take some work to integrate and would be active in parallel with the Pi.