Ah yes - present in Cockpit status
2 dives today with UDP Fix and MAVLink Server set. Lots of “EKF Bad” warnings again. The Cockpit widget showed red, and it was always the compass. Upon looking at the compass calibration in BlueOS, the two compasses were pretty much identical, while the EKF was fixed at 0 degrees. EKF would then rectify and follow the 2 compasses.
Hi @3dMB -
If you could share a .BIN log from the autopilot from that session, it may contain useful clues beyond “EKF Bad” for us to dig into!
@tony-white Latest logs added to the existing dropbox folder. Direct link here: Dropbox
Hi @3dMB -
I see a BlueOS system log from the 26th there, but no .BIN log from your recent dive?
@tony-white Do you not see the ‘logs’ subfolder in there?
I do, but the dates there are only through 9-15, and you posted more recently than that? If you could narrow it down to just the single .BIN file from the dive in question, that would be great!
I am guessing that the ROV date/time is not current - does it not automatically update from topside computer? The previous set of logs (from last week) went up to 244, the ones downloaded yesterday go up to 266. Is it fair to assume that the dives (as opposed to the bench tests etc) will be the larger log files between 245 & 266? If the date is actually correct, then the logs do not seem to have been saved 
Hi @3dMB -
The logs do not seem to show the EKF Bad message you were seeing:
Additionally, the switch to Mavlink Server may not have stuck, as the .BIN logs are empty files and you only have .tlogs. Mavlink Server does not create .tlogs!
Double check this under Mavlink Endpoints. If the compass continues to seem like the culprit for your EKF, consider calibrating again. Is there any chance large magnetic fields are present in your test area? These can confuse things in most situations…
The frequent heartbeat lost message suggests your tether is having issues? Or you’re not using the standalone version of Cockpit, and switching away from the browser tab running Cockpit lite?
Thanks @tony-white - I am currently away, but will double check the MavLink endpoints again on Wednesday.
The site is a granite quarry - so no theoretical interference. The quayside is made up of steel cofferdam pilings, so I have not been calibrating the compass directly there, however the compass issues were occurring while the ROV was 30m+ away from the quay, in fact up to just shy of 200m away, so interference should not be an issue.
I use standalone Cockpit - this was 1.19.0-beta13 or beta14, bandwidth is good (for FXTi) at around 70mbps (both ways).
@tony-white I can confirm that the ROV still shows Mavlink Server.
I have re-downloaded the logs again today, and put in the dropbox here.
In addition to this, I screen recorded the process of booting up BlueOS, and what it displays.
EDIT Just realized the .BIN files are elsewhere, so have uploaded them also
Hi @3dMB -
An AI review of the logs definitely points to the compass calibration:
There are three causes, often happening together:
- Thruster current is distorting the compass, and nothing corrects for it. COMPASS_MOTCT=0 and all the COMPASS_MOT_* values are 0.
- MAG0’s field strength drops by about 6 mG per amp. MAG1 drops by about 2.2 mG per amp.
- At the 25–60 A peaks in log 27, that shift is 150–350 mG, which is the same order as Earth’s field at your site (roughly 480 mG, from memory).
- With 40 A or more, SM is above 0.46 76% of the time. Events at 1323, 2598, 2615, 5818, 5947 and 5958 s all happened at 22–56 A. During the 5947–5958 s events, the two compasses pointed 50–57° apart.
- The compass calibration is poor. Even at under 5 A, the field strength changes with heading, from about 390 to about 500 mG. It should be nearly constant. The two internal compasses also disagree by 12–20° at baseline. Because of this, turning the vehicle trips it:
- Most of the events from 5406 to 6476 s in log 27 happened during turns of 50–130° at 17–27°/s, with low current.
- That same stretch is where EKF3 switches back and forth between compass 0 and compass 1 every 10 seconds.
- There is likely a local magnetic disturbance. A few events in log 26 happened with low current and no yawing, but with field strength far off normal. At 2501 s, for example, the field was 206 mG and the compasses were 41° apart. That points to something nearby, such as a steel structure.
I ran your logs through the MAG fit tool (only possible because the log has estimated position, usually this tool is not useful for ArduSub), the resulting parameters may improve your compass calibration by about 50%?
From plot.ardupilot tool:
00000026.BIN.params-compassfit.txt (395 Bytes)
From MAGfit tool, with battery current compensation:
MAGFit (1).param.txt (578 Bytes)
Remove the .txt extension and load on the Autopilor Parameters page - one of those should improve things!
thanks @tony-white - I assume I should change the geolocation, though, as I am a long long way from Hawaii.
Hi @3dMB -
I don’t think my location is used by the Magfit tool!? You should be fine to use those parameters, but feel free to process a log file with the tool yourself…
The AI also suggests disabling the compass with the worse performance, turning off compass 0 in favor of only using compass 1.
ok - I may have misread that…I saw it said “(Hawaii-Aleutian Standard Time)” followed by a load of coordinates…
I’ll run these tomorrow and see what it comes back with.
ok, so, it looks as though one of my compasses is causing the issues. With MMC5883 & AK09915 enabled, the EKF warning slider on the compass is constantly moving up and down. When only the MMC5883 is enabled, the same happens. When only AK09915 is enabled, the EKF warning compass slider does not move. I assume the compasses are part of the Navigator board, so the only method to rectify is a new Navigator board?
Additional info to the previous. Here is a screen grab showing the EKF status with both compasses active, only MMC5885 compass & only AK09915 compass active:
Shortly after the above recording, the EKF decided to completely ‘off-piste’, with both compasses correctly showing the ROV facing a westerly direction, but the EKF showing North East. I then land the ROV and reboot the autopilot, and service seems to be resumed (to a fashion)
All of the above is having run the MAFit.param from @tony-white above, and a fresh on-site calibration prior to the dive.
I then re-ran the parameters @tony-white sent while recording the screen, as well as another compass calibration.
We also have another screen recording of the EKF issues:
Hi @3dMB
As noted, it did look like turning one of your compass inputs off was a good idea. Note that your rapid-fire toggles of what compass to use, without restarting the autopilot, I believe have no affect on what compass the autopilot is actually using… other than to make the EKF gauge read full scale of range it seems!
I don’t think you need a new Navigator - it’s typical to achieve only “yellow” calibrations as you illustrated - and
Are you still getting warning messages? After autopilot restarts everything would seem to be working nominally in some moments during your screen recording, were you still getting the EKF Bad warning with the problematic compass disabled?
Unrelated, I’m curious why you’ve lowered the bitrate of the 4K cam to 2 Mbit, and only 1080p resolution? Do you have tether bandwidth limitations? If you routed the power wires from the 4K Cam near the Navigator, it may be a good idea to adjust that to maximize their seperation, ensuring they are tightly twisted together.
Looking at other screen grabs, EKF warnings continued on the cockpit widget display with bad compass disabled post reboots. I do not recall if audible warnings were given or not. Talking audible warnings, the Cockpit 1.19 betas say “F bad”, the Cockpit 1.18.4 stable says “EKF Bad”.
.BIN logs uploaded to dropbox folder, HERE.
2 reasons:
-
My laptop monitor is 1920x1080
-
I save the bandwidth for other attached sensors, I regularly mount Canon R6 mk2 on, as well as the DWE Explore3D, which I have recently been testing on the FXTi tether.
I have played with higher bitrate, but visually there did not seem to be much change. The 4k resolution and higher bitrates are in the bag as an option for future use.
I have spent around 8 hours, over the weekend and this evening, on ChatGPT, having a thorough analysis of .BIN & .tlog files, I have also performed in-water tests today, which have also been inputted into the data analysis. ChatGPT has come up with the following analysis report.
Poppy_Blue_Robotics_Engineering_Report_v1.pdf (130.9 KB)
All logs used in this analysis can be found in this dropbox folder Dropbox