@rafael.lehmkuhl video with plotters on screen. Half way through the ROV lost signal, disarmed etc
For reference, this is running 1.17.0 https://youtube.com/shorts/4ZXyMvslkFo?si=bahoLwrqUHeCvPqk
Thanks for the test Marcus!
It’s very clear now that the problem is on Cockpit’s side. More specifically, it’s a performance problem causing all the rest. You can see that from the App Frame Rate plot. It should be an almost steady-line around 30/60/120 FPS (depending on your monitor’s refresh-rate). It’s clear in the plot that we never go near that. It varies a lot and at the moment you lost the control of the ROV, we can see the frame rate dropping to something close to 0 for a long time.
Some questions:
- Do you have a Snapshot widget running in another view?
- Does it always lost signal after around the same usage time?
- Restarting Cockpit (by actually closing and reopening) resets the problem?
- Can you try v1.19.0-beta.2 and see if it’s causing the same problem?
- No snapshot was being used.
- No - yesterday was after around 2 hours, Thursday was on boot up
- I didn’t try this yesterday
- Will do tomorrow on the bench
@rafael.lehmkuhl I can confirm that the issues remain in v1.19.0-beta.2. I have done a few videos on the bench today, mostly just letting the ROV sit there, armed in manual mode, to watch the telemetry. Following are some videos, one recorded on phone showing some of the control responses, the other two screen recordings.
Video #1 - Recorded on phone showing control inputs https://youtu.be/07juBtZIJrg
Video #2 - Screen recording where at around 30s in, the thruster spooled up having had no control input https://youtu.be/F7LBzKqHK_c
Video #3 - Screen recording with audio (forgot to enable audio on first screen record), the thrusters spool up at around 40s having had no control input. I use cursor to disarm and re-arm as x-box controller seems to stop the snipping tool. I then leave the recording going for around 10 mins after with no control inputs. https://youtu.be/Zru7q7hQ9ew?si=uAKFWtjOI3rpXtV5
Around a minute after the third recording, the thrusters spooled up again under their own control.
Final additional screen grab of the same ROV, tether, network, laptop but running 1.17.0 https://youtu.be/s_vuUv-7hNg
Thanks again for the tests Marcus.
It is becoming more clear to me what is probably happening. From your tests and my investigations, I’m leaning to 3 culprits:
- Data lake update fan-out is being too naive and updating more than the application frame-rate
- Joystick pipeline have some heavy and unnecessary update logics
- Some canvas widgets (e.g.: Virtual Horizon and Compass) have some heavy and unnecessary update logics as well
While i figure them out, could you test this other build for me? More specifically I want you to use the “capture 10s profile” button under /menu/dev/performance-monitoring. It will get us a profile file that tells us which functions in the app are wasting the most computation time. That build does not have any fixes to the problems, just instrumentation for us to figure out where they are coming from.
I would ask you to capture two of those profiles, both recording the screen using the phone, without any screen recording app (since those can also put some extra performance pressure in the machine and we want to leave this out of the math for now):
- One profile without a joystick connected
- Another profile with a joystick connected and being used (both the motor control and camera tilt being changed)
In the meantime I’m developing fixes for the three items I mentioned earlier.
@3dMB after you perform the tests mentioned in the previous post, I would ask you to try this new build. This one is build on top of the previous (instrumentation) one, but it includes several performance improvements. I’ve tested here and was able to see a very clear gain. I’m curious on what you’re going to see there.
If you feel like testing it, please also use the profile tool with this one so we can see what’s the before and after 
Thanks @rafael.lehmkuhl - Alas I have a run away on some non-ROV jobs, so will not be able to run these until my return. I shall come back to you with some results ASAP
No problem! We can get back to this on your return.