Cockpit 1.19.0-beta10 keeps saying “f bad” on both ROVs. Any ideas?
Is there a longer incomplete message just before that? MAVLink STATUSTEXT content has the option to be split between multiple messages, which Cockpit doesn’t handle yet (it just treats them as independent).
Is that stored in a log that maybe looked at retrospectively?
Hi @3dMB -
This is only a change in what cockpit displays- vehicle logs will reflect the default autopilot names. Changing that would require major firmware modification, especially if done on the fly!!
The MAVLINK STATUSTEXT, that @EliotBR refers to, @tony-white ?
@3dMB -
Those status text messages are recorded in .BIN logs, so if you see the custom name printing there it will indeed be in the logs. However in a log viewer the flight modes will still be named for the original firmware name when highlighting what mode the system was in at different portions of the log.
There are downloadable Data Lake logs that were introduced in v1.19-beta.5, but they don’t record every variable by default, so would be relevant to future recordings rather than past ones.
Beyond that your best bets are the MSG messages in the .bin DataFlash logs from the autopilot (as Tony mentioned), or the direct STATUSTEXT messages included in .tlog telemetry logs (which may or may not be recorded in BlueOS depending on your version and MAVLink routing service). Check the BlueOS Log Browser page for what’s available 
Using Cockpit 1.18.4, things have progressed from “F bad” to “EKF Bad”.
Hi @3dMB -
The message “EKF Bad” does not come from Cockpit, but ArduSub/ the vehicle itself! It indicates that something in the EKF, or filter that determines vehicle position, is having an issue. There is a mini EKF widget that can show you exactly which sensor / estimate is having issues, available under Edit Layout in Cockpit. Performing a compass calibration in an area free from interference, along with a gyro calibration with the vehicle perfectly motionless, may resolve the issue.
thanks @tony-white , Ill add said widget tomorrow and see what it comes back with.
Right…following on from this, I had my most problematic dive ever. Thankfully I had screen record running, so have everything recorded for analysis. I have sped up a ton of the recording to cut between the pertinent parts of the dive with issues.
Before the dive I checked the compass readouts on BlueOS, and all was good.
I had 2 ROVs in the water simultaneously. Both ROVs are on latest stable arduous and BlueOS (as shown in video). The ROV on the recording has new 4k cam, so appreciate that this required cockpit 1.18.4. The first issues encountered on this dive were with the other ROV, which once in the water kept disarming. Knowing that this particular ROV is normally rock solid, and have had it on cockpit 1.17.0 previously, I re-installed 1.17.0. This ROV had no issues for the remainder of the dive.
The recording that is linked below shows most of the ROV dive. For reference, you can see the time in cockpit in top right corner. Due to the installation of the 4k cam, and it requiring space in the ethernet switch, I have moved the ping360 from ethernet otherwise usb (yes I have changed this over in the ping360 head also). I also have a exploreHD plugged into the Pi of this ROV. For this dive, I turned killed the exploreHD stream to hopefully isolate that from being an issue (the explore HD only ever ran at 320pix @ 10fps anyway). The 4k camera is running 1080p25 @ lowest bitrate.
The first thing to stop working is the Ping360, when I change the gain. The Ping360 stopped working many times over the 2 days. I noticed that in the Pingviewer 2 BlueOS extension, it shows an error. I also found that if I remove the Ping360 from there, it would usually rectify the issue.
Then the first (of many) disarm occurs.
I have the EKF widget showing, which suggests the compass is the cause of EKF issues. I show the compass screen in BlueOS, which is all over the place. After many many disarms, I try to reboot the autopilot, then do full reboot of the ROV, then try to nurse the ROV back home. Once nursed back home, I re-calibrate the compass and reboot the autopilot, then put the ROV back in. The issues still remain. I noticed that during some of the issues, the Pi CPU is running at full chat on all 4 cores. Later on this decreases to running full chat on only 1 core.
At some point during the dive I check the failsafes. I used to have the set to depthhold on loss of control - but I see this is no longer an option. I also see that the sensor confusion options had changed to disarm. For future reference, do these changes occur with new BlueOS updates, or with Ardusub updates?
At some point on the dive, the EKF widget list goes from mostly red crosses (bar bottom 2) to mostly green crosses - yet no inputs have changed, been added or modified.
Unfortunately I had the 4k camera on, I would have loved to have been able to revert back to cockpit 1.17.0, as I did with the other ROV, which seemed to have fixed the issues it was having.
The video will show my view and my actions, I will grab the logs tomorrow and upload them.
Hi @3dMB -
Thanks for the screen recording! Some notes as I watch it -
- Have you calibrated the gyro recently? This can only be done with the vehicle on land, perfectly motionless, and ideally closed up and a bit warm.
- It’s strange you’re getting that EKF bad message even without the vehicle pulling much current!
- Failsafe behavior, and generally autopilot parameters are never changed when updating BlueOS or ArduSub - sometimes they may be created if the new firmware has parameters the old one didn’t, but the behavior you describe (your failsafe settings changing) doesn’t seem possible…
- Were both vehicles on the same network? Or independently connected to the control computers?
- Do you prefer using the 4k Cam manager widget to adjust focus and zoom, instead of the game controller buttons? You may prefer the buttons once you add generic indicator widgets to illustrate the focus/zoom current values…
- Are you running the dev version of ArduSub, or one with the “UDP Fix” enabled? As Eliot pointed out in another thread, for vehicles with positioning systems like yours this makes a dramatic difference in performance. Generally the way your system is behaving makes me think this is the primary problem to resolve… This problem typically gets worse the longer the system runs, as you found, and autopilot restarts don’t improve it!
It looks to be set that way on your video! Disarming due to sensor fusion uncertainty is definitely not the default - you must have changed it at some point before setting it back here?
- Finally, do you have an idea of your maximum tether bandwidth? With Cockpit closed and no traffic flowing…
Thanks for sharing - I think this is a support issue that is best handled via the traditional channels - the support form in the help center, used to coordinate a video call where we can get an expert like @rafael.lehmkuhl a direct look at the system. Something is definitely not right with your unit! The other ROVs issues with 1.18 but not 1.17 Cockpit is also quite strange and not something we’ve seen before…
Yes - Gyro was calibrated after installing 4k cam on Monday.
no - completely independent.
I have been using the buttons, I put the widget there because the 4k setup instructions said to do so - I also found that seeing the focus slider helps as visual reference of change.
Assuming you refer to this?
Previously when I tried this I encountered issues (on the other ROV) as highlighted here. The ROVs are on 4.7.1 (stable)
That is not something I have changed.
speedtest on this ROV shows 60-80mb/s up and down. The other ROV is gigabit fibre.
Hi @3dMB
I don’t think you actually were running a version that included the udp fix. If the 4.7 image with the fix was flashed, it would say “UDPFIX” after the version number on the autopilot firmware page! The fix is present in the current Dev firmware, or can be downloaded from the thread you reference - we left off there waiting to here how testing of it went!
@tony-white It states “successfully installed new firmware”, but still shows 4.7.0 
I have restored to default firmware, 4.5.7 - change in firmware shown. I then installed the UDPFIX file, and the firmware versions shows 4.7.0 - not the “UDPFIX” that you say it will
Full set of log files for both ROVs on 17th Sept.
*note that Rachael was running 1.18.2, not 1.18.4 with issues, before I changed to 1.17.0*
Additional videos:
-
Poppy network test https://youtu.be/mfHZbxEUv1I
-
Poppy 4k camera resolution settings and jumpy footage during camera tilt https://youtu.be/JuppcdjC214
Hi @3dMB -
My apologies - I believe that you should see UDP Fix text in the status messages in Cockpit when you first connect to the vehicle, or restart the autopilot.
I looked at your logs from Poppy, and was surprised to find many 0 byte .BIN logs. Can you confirm you’re using Mavlink Server? This is set at the top of the Mavlink Endpoints page, and should improve things if it is currently using something else…
I am using it however BlueRobotics set the ROV up, the only exception being that the topside FXTi box I have ethernet cable plugged into the FXTi board to wifi router, as opposed to the RJ45-USB-b adaptor.
CORRECTION - the other change from standard is Ardusub parameters for the Tracker 650 as found here.
@tony-white I have just looked and it was set on MavLInk Router.
Assume I just select the “MavLink Server” option, and nothing else is required?