I was out testing Cockpit Beta 4 last week (Windows 11, Chrome browser) and struck a problem with downloads failing. The first recording worked fine and downloaded a massive 500MB file (I have a 12Mpx IP camera stream setup via Mavlink Camera Manager). From then on there was a trickle of *.ass files and then everything got choked.
Questions for you:
- Are there any fundamental limitations on the stream recording, buffering and download approach being used with Cockpit? (files size, duration etc.).
- Is the intention for it to be used for long-ish (5-10min) video captures or just occasional bursts of recording?
- Are there any known issues you’re aware of that could be causing these download failures. I had a look on Github but couldn’t see any (apologies for not doing a screen capture).
I’d also love your take on any concerns around video quality. *.MP4 vs. *.webm.
Thanks heaps for your great work on this. I wish I could contribute more meaningfully to the effort myself.
Hey Peter! Thanks again for the report! Answering the questions:
- Currently, the only limitation is that we are storing the video both on the memory and on the disk, and browsers limit the memory usage on 4GB per tab, so if the video reach about 3,5GB (accounting for 500MB for Cockpit itself), the browser should kill Cockpit’s tab (I didn’t test to that point yet). That does not seem to be the problem you’re having, because you said it failed during the download, where the video Blob is already finished. We have an issue open to fix that, and I will solve it this week.
- It is intended to be used for long videos, yes. We are even storing the video on the disk (using the IndexedDB of the browser), so there shouldn’t be any limitation to the video size as long as you have disk space, and we fix the bug I previously mentioned. @tony-white has already made videos as long as 30 minutes with Cockpit if I’m not mistaken.
- I can’t see any. Could you open the configuration menu and download Cockpit’s logs for when that happened? I will take a look at those. They are in the Development page.
I’m not the best to answer around video encodings and file types, but I think @joaoantoniocardoso can help you with this one.
Also, there’s a chance we can recover your failed videos. Could you try opening the IndexedDB of your browser to see if the videos are there? You can do it by opening the browser inspector (right click the screen>inspector) and following the steps on the following image. If they are there, don’t close the Cockpit tab, as there’s a chance we can recover them.
And I’m the one who needs to thank you. Your feedback is being very important for our development, specially in this Beta phase. Thank you very much
@PeterM even if the download problem didn’t come from Cockpit itself (it could be a problem in the network, in BlueOS, in the computer, in the browser, etc), as this is a very critical situation, I’ve opened an issue around that with the highest priority, to be solved in the next days. The idea is to stop auto-removing recordings, and always let to the user the decision over it, as they are the only one that really knows if the recording should be removed or not. Feel free to add there anything you find relevant.
Thanks for jumping all over this @rafael.lehmkuhl . I checked the indexedDB and the videos are long gone… We have GoPro footage as back-up and will organise another day on the water soon.
I think what’s suggesting in the ticket sounds great. Also good to know that the software is designed for capturing videos of good duration.
Good to hear you had a backup. I will try to make sure this does not happen again
Also Peter, do you recall how long was the footage, more or less? Maybe it got near an hour?
I ask that because the currently implementation is likely going to fail for vídeos around 60 minutes or more, because of the memory usage. With the patch that I’m currently working on that shouldn’t be a problem anymore.
No worries @rafael.lehmkuhl . Fully expect shaky behaviour in beta. We were recording 15-20min videos (200m transects with boat following) but at 4k with a camera we’re currently evaluating.
Haaaaa, there’s the problem!
I had calculated that for the FullHD camera we would have a maximum of 70min before the memory went off. With the 4K camera that should go to a maximum of about 17 minutes.
You’ve probably experienced that before everyone because we are still not testing it with 4K cameras
I have one here and will do some tests after the patch to see if everything goes well.
Just adding an update here that in the latest Cockpit version (1.0.0-beta8) the video patch was merged, and I have successfully recorded a 2h (7.4GB) video without problems (I interrupted it, so this is not the actual limit).
Just tested our 4k camera and recorded a 2.5hr video (approx 12gb) all worked really well. Webm file and .ass file saved no problems.
Only issue we have is not being able to scrub through large webm videos to quickly check quality / completeness before a move to next location etc. Scrubbing through just jumps back to beginning of file in both vlc and wmp - anyone found a better viewer?
Wow! You probably recorded the longest and biggest video on Cockpit till now haha
I was wondering how long we could go with 4k footage. Theoretically Chrome raised the per-tab memory limit to 16GB, so you went almost as far as possible
- Which version of Cockpit were you running?
- Did the stop command took long? We saw issue around that for those long recordings.
- And after clicking stop, did it take long to process and appear on the video download table?
About the problem on scrubbing, it’s a bug on the chrome video generation (it does not add duration to the video) that we are theoretically fixing on the processing phase, but I’ve noticed that it only really fixes the problem on smaller videos. The longest ones are always suffering with it.
I will open an issue to better track that.
Matthew, on top of that, while we don’t fix the duration/seek/scrub problem for the long videos, @tony-white said you can fix the video outside Cockpit by reprocessing it with a video convert tool like Handbrake.
Awesome @mattcmgb. Do you mind sharing what 4k camera you’re using? We’re still evaluating solutions in this area.
@PeterM Hi Peter, we are currently experimenting with this camera. It seems to perform quite well once we dialed in the best settings. We get a good quality 4k stream at a bit rate of near 14k. Currently appears to utilise about 15mbs bandwidth.
@rafael.lehmkuhl thanks for the explanation ref the video scrubbing issue. Our main problem is regarding time on a vessel for surveys. We need to be able to do a quick quality / sanity test on the acquired videos before we can move to another site / location. Whilst converting via handbrake is an option - that takes up significant vessel time and therefore $$$.
Regarding the video recording time, it was a bench test to check how far we could push it. We’d never need that length of time in reality but was a good test. There wasn’t any appreciable delay when the video was stopped etc. All seemed very smooth.
We were using that latest version of Cockpit, think beta 8
Hi @mattcmgb -
I have no issue scrubbing through recordings, without having to transcode, using VLC media player…