Our ROV when armed tends to peg a CPU with the ardupilot_navig process when checking the system information it appears to be related to ardupilot-manager/firmware
I also noted log files for mavlink-camera-manager using a large amount of disk space, I have since added some log rotation to resolve.
As a rule should we be placing a log rotation policy for all log files in the /var/logs/blueos/services/
Is there a reason why the firmware manager is using high CPU and can this be turned down or logging level checked.
Would it be constantly trying to update or find new firmware? Seems strange that this process would be running while armed of does it have a specific function.
Recreated docker image to make sure it was a clean install but still experienced the issue.
Hi @UnderTheSea -
Welcome to the forums!
Can you share what version of BlueOS and ArduSub you’re experiencing issues with? Are you using a newer unit with Pi4 and Navigator?
Providing BlueOS system logs would enable us to review what may be happening with your system, these can be downloaded from the gear icon in the lower left.
Thanks for the reply, I will collect the required info for you as soon as ROV is powered up and I can get access to the logs, due to an issue with logs filling up the system drive and going offline, we were recommended to remove the current docker images reinstall. So, I believe they will be the most current.
I will attach some of the log I grabbed while tailing to see if I could find anything relevant to search the forums with.
Thanks for the recommendation should we be setting this to a value of 150 or gradually tuning it down until it’s at an acceptable rate or start low and increase.
Also confirming this setting is done in Attitude Controller Parameters in ground control. I presume the process would be to adjust and test the system for stability and performance under load and go from there.
Would you say this change is linear or will we find a point where the system stabilises?
Hi @UnderTheSea -
We’ve only tested ArduSub at 200hz, so lower that rate at your own risk. Since the Pi4 has 4 CPU cores, using one of them intensively for the autopilot on occasion isn’t usually a big deal. Can you share what is motivating you to free up more CPU cycles?
Keep in mind, the autopilot is running onboard BlueOS as its own docker container, unlike systems that run that code on an external autopilot like a pixhawk.
The initial issue was a system crash caused by the logs filling the drive and crashing the docker container.
The standard storage is 16GB, I was concerned about the amount logging that was accruing, if this size storage is a normal setup could the logging have been set to high or an error with the system generating a higher number of logs?
Once the log issues were addressed via log ration being implemented, we noticed some latency and packet loss when sub was armed, this may have been present before the crash and identified as a result of investigating the packet loss. When checking over the system processes when arming and disarming high CPU was detected for the process ardupilot_navig when armed. I understand this will go up due to it being armed but was concerned that it was potentially pegging the CPU.
Being new to BlueOS, I have been looking through the forums/manuals for possible issues and past problems to eliminate.
A clean installation of the container was completed and the config restored to eliminate any issues with the original install corruption as a side effect of the container crash.
I am wating to get access to collect the logs to share, thanks again for your assistance.