Hi @3dMB -
Sorry I haven’t had a chance to compare your parameters to my own that work fine with Cerulean DVL.
It is possible to reset to default parameters from the BlueOS Vehicle Setup page (configure → parmaters) - you may want to backup to preserve your thruster directions and sensor calibrations first!
Today I put the ROV back in. I was running the most current version of cockpit. I have restored default parameters - except the parameters that Cerulean suggest in the Tracker 650 setup. I enabled position hold mode and the ROV went charging into a quayside - certainly not a position holding pattern…
Hi @3dMB -
Without a log file / knowledge of what versions of software you’re using it’s hard to provide much in the way of troubleshooting advice!! The .BIN log file, found under Autopilot Logs in BlueOS will contain the parameters you’re using.
If you reset to default parameters, you may need to calibrate your compass/accelerometer/gyro before any automatic navigation will work. If you’ve setup all your autopilot parameters per Cerulean’s documentation, and calibrated the ROV, things should just work! Have you made any change to default IP addresses?
Hi @tony-white, today I put the ROV back in for DVL testing (which I am consulting Cerulean), running cockpit. Before the dive I updated to the latest version of cockpit 1.15.2. Part way into the dive I put the ROV in manual control mode and just left it be - to show the DVL messages. Shortly into the manual control period, cockpit lost control “MYGCS 255 Heartbeat lost”. I was unable to regain control using Cockpit and the ROV sank. Failing the ability to regain control I opened QGC which happily took control.
I am also using the most up to date BlueOS and Ardusub - all checked before the dive.
I happened to be screen recording for the dive HERE - at just after 7mins in, the control loss detailed above occurred.
The parameters are all as stock due to new install - except the parameters stated on the Cerulean page - all sensors have been recalibrated since new install.
Hey Marcus! Could you get us the BlueOS logs from that dive? They will probably explain what happened.
Hi @rafael.lehmkuhl - according to the modified times it should be one of these two.
00059-2025-05-24_09-03-51.tlog (690.7 KB)
00061-2025-05-24_09-05-45.tlog (4.6 MB)
@rafael.lehmkuhl my bad - try this:
system_logs.zip (8.9 MB)
Thanks @3dMB! I can see an automatic restart on the autopilot manager happening at 14:30 UTC. Do you remember if the time matches your tests? It indicated something went wrong with the autopilot process and it tried to recover.
@rafael.lehmkuhl - no, according to the screen record, the issue happened at 10:13 BST (09:13 GMT/UTC)
@3dMB I couldn’t find any clear problem in the logs. I will ask for help from @tony-white and @williangalvani on that since they have DVLs and know better about their behavior.
In the meantime, I’m adding some new debug tools in Cockpit so you can know what Cockpit is sending/receiving, and also what the mavlink router is exchanging. This way we can better know where the problem is.
@rafael.lehmkuhl Just a thought, how does the Pi/Navigator update time for the log? Does it set it automatically down the tether from the topside computer or does it need an internet connection? If the latter, the restart you spoke of at 14:30 could be the same event. If the former, then the time displayed on the topside computer is definitely the time the event occurred.
Yes, there are inconsistencies regarding time sync in the way BlueOS logs the different services. I had a discussion with the team and they are working on improving that so we can make the debugs easier. Todat it only updates the clock when connected to the internet, and different services are using different timezone references, without making it explicit.
In the case that restart at 14:30 is indeed the same event, that explains what happened (a fail on the mavlink router), but not why it happened, although we have found a very similar issue in the last week, and I believe it could be the same you’re having.
Would it be possible for you to test the latest Beta of BlueOS? It includes an update on the mavlink router that solves the problem I mentioned (and maybe yours as well).
1 Like
Thanks @rafael.lehmkuhl - I’ll give it a go and report back, it won’t be for a couple of weeks though.
1 Like
Hi @rafael.lehmkuhl, the same happened again today with BlueOS v1.5.0beta6. I did 2 dives, both dives started with Cockpit running, then error saying “[SOMETHING I DIDNT CATCH] 255 heartbeat lost”, ROV disarmed and wouldn’t re-arm, cockpit manually quit & QGC opened, ROV carried on happily. Logs attached.
system_logs.zip (11.4 MB)
Hey @3dMB!
Just to be sure, this was also with the DVL enabled, right?
I received another report yesterday from a user that is having problems of heartbeat lost as well, also with a positioning system enabled.
And does restarting Cockpit make it work again, or it won’t regain control? I’m trying to narrow down what could be happening. I had no problems with 2h+ sessions in my office tests, but I don’t have a DVL connected and my Pi is not in an enclosure, so if it was one of those two causes, I would indeed not see here.
Hi @rafael.lehmkuhl,
Correct, the Cerulean Tracker 650 DVL was active and the dive started with geolocation being inputted into Cerulean Tracker program.
I have not tried restarting cockpit in recent months, my priority is to regain control of the ROV and QGC achieves that without fail.
Totally fair.
One of my colleagues has a Cerulean DVL here. We will setup it and try to reproduce the problem.
If you happen to be able to reproduce it again, and can just give a restart on Cockpit to see if it regains control, that can already tell us if the problem is on Cockpit or in one of the components of our mavlink pipeline (mavlink2rest and mavlink-server).
I also asked my colleagues in BlueOS to take a look at your logs to see if they find any clear problem.
@rafael.lehmkuhl IF I am in a suitable location when it happens again, I shall try to reboot Cockpit.
Would the Mavlink logs be of any use to you?
Thanks.
And it does, yes.