Thank you. Let’s fix this as soon as today.
thanks @rafael.lehmkuhl
Marcus, can you reproduce the bug on a bench test? If so, I would ask you to change the interval from 1 to 0.5 and test. I’m curious if the bug is time related or quantity related. If I’m correct, the problem is quantity related and will happen with around 8 minutes.
From your logs, the bug always happen around 17min/1000 captures.
I’ll do an in-water test in about 5 mins
That makes it unusable due to cockpit blinking and being very laggy on controls - but test wise, it seems to be quantity related
Thanks Marcus! I have confirmed what the bug is. It’s a memory leak in the snapshot feature. We are not completely destroying the capture elements, and they top at 1000. This is also causing a resource usage problem.
I’m going to release a fix today.
@3dMB I was able to reproduce and fix the bug. It was indeed a leak in the snapshot system.
Please download and test a build from here and tell me how it goes.
If you want, I’ve also prepared this build, which, besides the leak fix, also removes the screen blink and success snackbar for timed snapshots and allows sub 1s timed snapshots (tested here 10Hz snapshots without problems).
@rafael.lehmkuhl You are THE MAN!!! This seems to work beautifully with the standard IP Camera - not tried with ExploreHD yet as using different ROV today, but assume that camera shouldnt make any difference.
The screen blink is actually kinda reassuring and a reminder that capture is occurring, however at less that 1s, it becomes unusable. As a thought, is it possible to automatically disable it for sub-second intervals, or make the blink no more frequent than 1s when capturing sub-second?
Hey Marcus! So happy to hear that! I know you’re using this feature a lot, so it’s important for us that it works flawlessly!
And thanks a lot also for reporting the problems. It’s the kind of thing we don’t catch in our bench tests, and we depend on the feedback from you pilots!
About the screen blinking, it does make sense. I will put it as optional but enabled by default on sub 1Hz captures. It is also already always enabled for manual snapshots.
I’m also happy that with those changes the snapshot system is much more performative. It used to cause severe frame-rate drop.
Hi @rafael.lehmkuhl - Today I tried the test build on my other ROV, not because of need for snapshot, it was still installed on my laptop from earlier in the week. That ROV really did not like the test build. The ROV control went really laggy, control inputs would occasionally do nothing, other times brief inputs would result in the thrusters ‘sticking’ for unto 10 seconds, arm/disarm did not play nicely either, small taps to the camera tilt resulted in the servo running beyond to its limits. Initially I thought it may have been my topside wifi, so I went back to FXTi box, the issues persisted. Remembering that I was using the test build, I looked in downloads folder for previous versions of Cockpit. I re-installed 1.18.2 and normality returned. Incidentally, the ROV that this happened on is the same one that I changed the Pi on previously - my other ROV that has never skipped a beat ran all day happily on the test Cockpit build.
That’s very curious. Can you send me the logs from that session?
absolutely! will do tomorrow when I have some bench time
Nice, thanks!
And please use version 1.19.0-beta.2. It includes the full snapshot fixes from that dedicated build. This way it’s easier for us to track the report against the implementations.
@rafael.lehmkuhl here are the logs for yesterday and today. When I fired the ROV up this morning, the same issues were experienced - video link to follow, then I installed 1.17.0, and all was good, I then re-installed the test version and I was unable to recreate the fault. I have since gone back and forth between the versions, and still cannot replicate the fault - gremlins somewhere…
Cockpit (Jun 04, 2026 - 10꞉46꞉54 GMT+1).syslog (4.4 MB)
Cockpit (Jun 05, 2026 - 08꞉57꞉21 GMT+1).syslog (5.8 MB)
Cockpit (Jun 05, 2026 - 08꞉53꞉45 GMT+1).syslog (5.8 MB)
Cockpit (Jun 04, 2026 - 13꞉26꞉42 GMT+1).syslog (9.2 MB)
Cockpit (Jun 04, 2026 - 13꞉21꞉58 GMT+1).syslog (5.5 MB)
Cockpit (Jun 04, 2026 - 11꞉07꞉30 GMT+1).syslog (4.5 MB)
Cockpit (Jun 04, 2026 - 11꞉01꞉30 GMT+1).syslog (4.4 MB)
Cockpit (Jun 04, 2026 - 10꞉59꞉51 GMT+1).syslog (8.8 MB)
Thanks Marcus.
Yeah, those kind of bugs are the hardest to find. Damn gremlins haha
I will take a look at the logs to see if there’s anything there, and try to investigate the codebase as well to see if there’s any obvious problem.
If you see it happening again, try to identify if there’s anything specific to that situation (hardware-wise, software-wise, even order of things). Those are usually harder to recognize but sometimes help in the debugging.
Anyway, thank you again for the help with the snapshot bug, and let us know if you see any other problem around it.
Apologies on the delay uploading these, I recorded another but somehow it corrupted. Either way you’ll see what is happening in the different versions:
Test version - https://youtu.be/RmGaBk9N4dA
V1.7.0 - https://youtu.be/DuqDyH3KeyU
I noticed that the test version puts the ROV into Alt-hold at random intervals - that seems to be what is causing the thrusters to spool up, but then the disarm key will not work. This may suggest that it is a communications issue, as IIRC the failsafe for loss of comms is Alt-hold - through this may be a rabbit hole.
No worries.
Can you add four Plotter widgets to your view and perform again this same test you just recorded? This will help understanding if the application is stalling or if the communication is. Please point the Plotters to:
- Cockpit App Frame Rate
- BlueOS network ‘eth0’ download speed (Mbps)
- BlueOS network ‘eth0’ upload speed (Mbps)
- BlueOS CPU Usage (average)
@rafael.lehmkuhl is that the same as running the following as mentioned above?
Almost! That one is only missing the BlueOS Upload Speed plotter.