Ubuntu 20.04, QGC 4.2, cannot connect raw video

Good afternoon, all,

I was previously using QGC 3.5, and my system provided video to QGC after code I wrote modified the video stream. I was sending TCP, 8 bit, Gray, 640x480.

I have since attempted to upgrade to QGC 4.2-stable.
I can no longer get the raw video connection between my code and QGC to work.
I keep getting a ‘Broken Pipe’ error every time I try to send data.
It does seem to connect, but breaks the connection each time I send a packet of data.

Should I be using a newer or older version of QGC?

Thanks for the help,

Mark

Hi @mark,

We’re not sure what’s going on here, but it’s also hard to give advice on a custom setup without much information about it.

Are you able to specify where you’re modifying the video stream (e.g. on the onboard computer (if so, which Companion/BlueOS version are you using) or on the topside computer), which encoding you’re using, and also provide a gstreamer pipeline so we can try to replicate the issue?

Eliot,

Thanks for getting back to me.

I could not find in the forums any specific help for this.

I am grabbing data in RAW video format, 640X480, 8 bit gray

I have some proprietary work to perform per frame.

Then I encode it with GST.

For testing, I am using a recorded RAW file and running it through the following:

gst-launch-1.0 filesrc location=file.raw ! videoparse width=640 height=480 format=GST_VIDEO_FORMAT_GRAY8 framerate=24/1 ! videoconvert ! mpeg2enc format=3 ! tcpserversink host=xxx.xxx.xxx.xxx port=yyyy

I have set QGC source to:

TCP-MPEG2

My server IP:port

I don’t appear to be getting any logging outputs but still receive the Waiting for Input on the video

I know that the gst pipe works, but not sure that it is the correct pipe.

Also not sure that I am choosing the correct settings for QGC

Thanks for the help,

Mark

Elliot,

I have changed my code (gstreamer pipeline) to more closely match what I find in BlueOS companion:

Companion pipe looks like:

gst-launch-1.0 -v v4l2src device=$DEVICE do-timestamp=true ! video/x-h264, width=$WIDTH, height=$HEIGHT, framerate=$FRAMERATE/1 ! h264parse ! queue ! rtph264pay config-interval=10 pt=96 ! udpsink host=192.168.2.1 port=5600

My new pipeline looks like the following. For testing purposes, I am sending raw video from a file:

gst-launch-1.0 filesrc location=.raw ! videoparse width=640 height=480 format=GST_VIDEO_FORMAT_GRAY8 framerate=24/1 ! videoconvert ! x264enc bitrate=1024 ref=4 key-int-max=20 ! queue ! rtph264pay config-interval=10 pt=96 ! udpsink host=xxx.xxx.xxx.xxx port=yyyy

I have QGC set to UDP, and port YYYY

As I look at the logging from QGC, I can see that it is recognizing that I am sending data, but it doesn’t actually parse the data for display.

I only get the ‘Waiting for Video’ in the display.

I am not a gstreamer expert. In fact, I am quite new at it.

Have you any thoughts as to how I should modify my pipeline to get the video to QGC?

Thanks again.

Mark

~WRD000.jpg

Ok, Eliot,

I am getting closer.

I got this pipeline from example on internet:

gst-launch-1.0 -v videotestsrc ! video/x-raw,framerate=20/1 ! videoscale ! videoconvert ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay config-interval=10 pt=96 ! udpsink host=127.0.0.1 port=5000

This shows the test pattern in the QGC Video display.

I guess the raw conversion I am using is seriously wrong

So I will work from here.

I don’t think I need your help any longer.

Mark

~WRD000.jpg

I finally got the pipeline running.

In case it helps anyone, here is my situation

Raw 8 bit recorded video – 640x480

Need to process it before sending to QGC

My pipeline now looks like:

gst-launch-1.0 -v filesrc location=/home/grumpy/Desktop/AnglerRec-640x480-8Bit-00024.raw ! videoparse width=640 height=480 format=GST_VIDEO_FORMAT_GRAY8 framerate=10/1 ! videoconvert ! video/x-raw,format=I420 ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay config-interval=10 pt=96 ! udpsink host=192.168.2.42 port=5650

This now works sending data via UDP to QGC

Mark

1 Like

It’s possible the TCP-MPEG2 source option has some issues - if I remember correctly the video pipelines have changed somewhat recently, and I’m unsure whether that’s something that would have been tested to ensure was still working.

All our standard pipelines use H264-encoded video over UDP, which is what you seemingly ended up getting working in the end. Glad you managed to resolve your issue! :slight_smile:

Note that in the H264 encoding that you’re doing (with the x264enc element) you may be able to get some additional quality from the stream (/avoid some compression loss) by increasing the bitrate and/or using a lower speed-preset value.

Eliot,

First, thank you for all of the help you have provided. I have learned a lot.

I have, I hope, one last issue. this pipeline worked in the previous version I was testing, QGC 4.2

I am running QGC 4.2.4 - master. Compiles and runs. But when I supply video via the following gstreamer pipeline for testing I get the following odd video in QGC.
I am hoping you can provide some clarity for this issue. I have not been able to find any questions
here or the Internet.
This is on a fresh install ubuntu 22.04.

Thanks for your help,

Mark

gst-launch-1.0 -v filesrc location=~/Desktop/640x480-8Bit-00024.raw ! videoparse width=640 height=480 format=GST_VIDEO_FORMAT_GRAY8 framerate=10/1 ! videoconvert ! video/x-raw,format=BGRA ! rsvgoverlay location=“/xxxxxx/configuration/targetBox.svg” x=1 y=1 width=100 height=100 ! videoconvert ! video/x-raw,format=I420 ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay config-interval=10 pt=96 ! udpsink host=192.168.2.1 port=5600

If a pipeline works when received by the stable version of QGC but not master then that’s an issue you’ll need to report and discuss on the QGC repo - we don’t actively support QGC development. Alternatively you could check out the tag for the relevant stable release and try to build off that, if you’re only making temporary changes to test things.

At a guess the QGC video integration code has likely changed somewhat between the versions you’ve tested, and there may be some kind of issues with the non-standard options you’re testing (normally video streams are in colour, not greyscale, and we also don’t usually have svg overlays or onboard encoding as part of the video stream pipeline).

Another thing to try (assuming the normal video settings are available in the blurred out section of your window) would be changing QGC’s video fit options.

OK, anyone viewing this thread, I have discovered a mistake in my pipeline.
I was converting 8bit B/W video to BGRA in order to add an SVG overlay.
then I was changed it back to I420 (don’t know why).
Well, I took out the conversion to I420 and added a queue after the BGRA conversion and another before the x264enc.

the new pipeline looks like:
gst-launch-1.0 -v filesrc location=~/Desktop/640x480-8Bit-00024.raw ! videoparse width=640 height=480 format=GST_VIDEO_FORMAT_GRAY8 framerate=10/1 ! videoconvert ! video/x-raw,format=BGRA ! queue ! rsvgoverlay location=“/xxxxxx/configuration/targetBox.svg” x=1 y=1 width=100 height=100 ! videoconvert ! queue ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay config-interval=10 pt=96 ! udpsink host=192.168.2.1 port=5600

1 Like