I was out testing Cockpit Beta 4 last week (Windows 11, Chrome browser) and struck a problem with downloads failing. The first recording worked fine and downloaded a massive 500MB file (I have a 12Mpx IP camera stream setup via Mavlink Camera Manager). From then on there was a trickle of *.ass files and then everything got choked.
Questions for you:
Are there any fundamental limitations on the stream recording, buffering and download approach being used with Cockpit? (files size, duration etc.).
Is the intention for it to be used for long-ish (5-10min) video captures or just occasional bursts of recording?
Are there any known issues you’re aware of that could be causing these download failures. I had a look on Github but couldn’t see any (apologies for not doing a screen capture).
I’d also love your take on any concerns around video quality. *.MP4 vs. *.webm.
Thanks heaps for your great work on this. I wish I could contribute more meaningfully to the effort myself.
Hey Peter! Thanks again for the report! Answering the questions:
Currently, the only limitation is that we are storing the video both on the memory and on the disk, and browsers limit the memory usage on 4GB per tab, so if the video reach about 3,5GB (accounting for 500MB for Cockpit itself), the browser should kill Cockpit’s tab (I didn’t test to that point yet). That does not seem to be the problem you’re having, because you said it failed during the download, where the video Blob is already finished. We have an issue open to fix that, and I will solve it this week.
It is intended to be used for long videos, yes. We are even storing the video on the disk (using the IndexedDB of the browser), so there shouldn’t be any limitation to the video size as long as you have disk space, and we fix the bug I previously mentioned. @tony-white has already made videos as long as 30 minutes with Cockpit if I’m not mistaken.
I can’t see any. Could you open the configuration menu and download Cockpit’s logs for when that happened? I will take a look at those. They are in the Development page.
I’m not the best to answer around video encodings and file types, but I think @joaoantoniocardoso can help you with this one.
Also, there’s a chance we can recover your failed videos. Could you try opening the IndexedDB of your browser to see if the videos are there? You can do it by opening the browser inspector (right click the screen>inspector) and following the steps on the following image. If they are there, don’t close the Cockpit tab, as there’s a chance we can recover them.
@PeterM even if the download problem didn’t come from Cockpit itself (it could be a problem in the network, in BlueOS, in the computer, in the browser, etc), as this is a very critical situation, I’ve opened an issue around that with the highest priority, to be solved in the next days. The idea is to stop auto-removing recordings, and always let to the user the decision over it, as they are the only one that really knows if the recording should be removed or not. Feel free to add there anything you find relevant.
Thanks for jumping all over this @rafael.lehmkuhl . I checked the indexedDB and the videos are long gone… We have GoPro footage as back-up and will organise another day on the water soon.
I think what’s suggesting in the ticket sounds great. Also good to know that the software is designed for capturing videos of good duration.
Good to hear you had a backup. I will try to make sure this does not happen again
Also Peter, do you recall how long was the footage, more or less? Maybe it got near an hour?
I ask that because the currently implementation is likely going to fail for vídeos around 60 minutes or more, because of the memory usage. With the patch that I’m currently working on that shouldn’t be a problem anymore.
No worries @rafael.lehmkuhl . Fully expect shaky behaviour in beta. We were recording 15-20min videos (200m transects with boat following) but at 4k with a camera we’re currently evaluating.
I had calculated that for the FullHD camera we would have a maximum of 70min before the memory went off. With the 4K camera that should go to a maximum of about 17 minutes.
You’ve probably experienced that before everyone because we are still not testing it with 4K cameras
I have one here and will do some tests after the patch to see if everything goes well.
Just adding an update here that in the latest Cockpit version (1.0.0-beta8) the video patch was merged, and I have successfully recorded a 2h (7.4GB) video without problems (I interrupted it, so this is not the actual limit).
Just tested our 4k camera and recorded a 2.5hr video (approx 12gb) all worked really well. Webm file and .ass file saved no problems.
Only issue we have is not being able to scrub through large webm videos to quickly check quality / completeness before a move to next location etc. Scrubbing through just jumps back to beginning of file in both vlc and wmp - anyone found a better viewer?
Wow! You probably recorded the longest and biggest video on Cockpit till now haha
I was wondering how long we could go with 4k footage. Theoretically Chrome raised the per-tab memory limit to 16GB, so you went almost as far as possible
Some questions:
Which version of Cockpit were you running?
Did the stop command took long? We saw issue around that for those long recordings.
And after clicking stop, did it take long to process and appear on the video download table?
About the problem on scrubbing, it’s a bug on the chrome video generation (it does not add duration to the video) that we are theoretically fixing on the processing phase, but I’ve noticed that it only really fixes the problem on smaller videos. The longest ones are always suffering with it.
Matthew, on top of that, while we don’t fix the duration/seek/scrub problem for the long videos, @tony-white said you can fix the video outside Cockpit by reprocessing it with a video convert tool like Handbrake.
@PeterM Hi Peter, we are currently experimenting with this camera. It seems to perform quite well once we dialed in the best settings. We get a good quality 4k stream at a bit rate of near 14k. Currently appears to utilise about 15mbs bandwidth.
@rafael.lehmkuhl thanks for the explanation ref the video scrubbing issue. Our main problem is regarding time on a vessel for surveys. We need to be able to do a quick quality / sanity test on the acquired videos before we can move to another site / location. Whilst converting via handbrake is an option - that takes up significant vessel time and therefore $$$.
Regarding the video recording time, it was a bench test to check how far we could push it. We’d never need that length of time in reality but was a good test. There wasn’t any appreciable delay when the video was stopped etc. All seemed very smooth.
We were using that latest version of Cockpit, think beta 8
And I think even better than allowing the user to scrub the video in Cockpit itself (which is actually a feature we want to have), we should have some kind of video analyzer that just do this automatically for the user (analyze the entire video after it was saved to see if there are dropped frames or something like that).
Opened an issue to track that. I think once we reach stable, those quality-of-life features related to video should be worked on.
Hi @tony-white it works fine in vlc for short videos, but won’t scrub through long videos on any machines we’ve tried it on - even on our main video processing work station. Not sure at what point of duration the cut off is.
We can work around it but looks like the suggestion by @rafael.lehmkuhl will be a great addition in the future.
I wonder if this is a RAM issue. If you’re trying to pre-load the entire decoded video into memory for efficient scrubbing then it will take significantly more space than the encoded form that’s stored in the recorded file, so it might be worth checking whether your viewing software supports reading/loading the video in chunks.
I’ve opened a complementary issue that’s focused on live status tracking and warnings/notifications, since Rafael’s issue is focused on providing information after a recording is complete.
@joaoantoniocardoso was investigating this issue (seeking/scrubbing/duration) last week and the found out the culprit. We already merged the fix on BlueOS master, and it will be available on BlueOS 1.2.0-beta9.
Let me know if everything works fine after you update BlueOS. I’ve tested here with a 30 min video and both seeking and total video duration were perfect.
