BlueOS major features and cockpit extensions usage issues (will continue to update)

Blue Robotics Software BlueOS
1

Recent use problems:
BlueOS chip temperature display abnormal;
cockpit stuck, disconnected vehicle, disconnected handle, ui design is incomplete, page no response.

1.BlueOS chip temperature display abnormal; :thermometer:
After an hour of continuous rov operation, we felt a significant cockpit delay, so we wanted to test the cabin temperature and cpu temperature. The result is that the water temperature is 31 Ā° C, the cabin temperature is 56 Ā° C, and the CPU temperature is actually displayed as 43 Ā° C.

1.cockpit :
We are experiencing some issues with the Cockpit ground control station and would appreciate any assistance.

Firstly, we have noticed significant lag while using Cockpit. During our robot debugging sessions, we alternate between using Cockpit and QGC, and there is a noticeable difference in data response times. Cockpit responds more slowly, which greatly affects the operational experience for robots requiring human control. From the moment we power up, Cockpitā€™s response is slightly slower than QGC. As time progresses, after about an hour of continuous motion control, the lag becomes more pronounced.

In terms of button control, for instance, when pressing the arm button, it takes approximately 3 seconds for the icon in Cockpit to switch from disarm to arm. At the same time, we attempted to modify the page layout by mapping data to the bottom bar to display current speed information. During this process, the page froze and a ā€œPage Unresponsive, wait or close?ā€ prompt appeared. After waiting for about 20 seconds, the customizable data list finally loaded.

Additionally, weā€™ve encountered two instances where the vehicle disconnected and one instance where the remote controller disconnected while using Cockpit. For the vehicle disconnections, refreshing or restarting Cockpit resolved the issue, but this clearly demonstrates a lack of stability in the platform, which is very frustrating!

For the remote controller disconnection, simply unplugging and replugging did not resolve the issue. After numerous attempts, the successful method was: unplug the remote controller, refresh BlueOS, open the joystick page, then plug in the remote controller when the ā€œno joystick connectedā€ message appeared. This reconnected the remote controller.

There is also an issue with the UI design; it does not provide a full-angle pitch display, which is a feature I need. This functionality is available in QGC and works very well there.

After using BlueOS for about an hour, it becomes increasingly laggy, with a significantly reduced data refresh rate. Switching to QGC, I found its data refresh rate to be much higher and its responsiveness much quicker, allowing me to stabilize the robotā€™s posture more effectively during control. :thought_balloon:

These issues significantly impact me. Any insights or solutions would be greatly appreciated. Thank you for your assistance! :smiling_face_with_three_hearts: :smiling_face_with_three_hearts: :smiling_face_with_three_hearts:

3 Likes
2

I have one additional piece of information: we pulled a four-way video stream while using BlueOS.Iā€™m not sure if thatā€™s what caused Catton

1 Like
3

Hi @xiaoliyu -
Iā€™ll leave some of the Cockpit issues to our relevant team members to respond, but I wanted to note / request -

  1. The temperatures youā€™re seeing are normal, and should not affect operation! Generally internal temperatures greater than 60-70C are cause for concern.
  2. Can you share what version of Cockpit, and BlueOS, youā€™re testing with? Weā€™ll be better able to assist you if you can download BlueOS system logs from the gear menu in the lower left of the BlueOS interface.
  3. Can you share what topside computer hardware youā€™re using? Cockpit does seem to be a bit more resource hungry than QGC, but if using a sufficiently powerful computer in our experience the video latency is actually better than QGC!
    Thanks!
2 Likes
4

Thank you for your responseļ¼ :smiling_face_with_three_hearts:
We are testing BlueOS version 1.2.6, with the cockpit version v1.0.0-beta.21. The attached file contains our logs, and the issue occurred on July 23, 2024.

Here are the hardware specifications of our topside computer:

  • Processor: 13th Gen Intel(R) Coreā„¢ i7-13620H 2.40 GHz
  • RAM: 32.0 GB
  • Hard Drive Capacity: 1T
  • System Type: 64-bit operating system, x64-based processor (Windows 11)
  • CPU Cores: 10
  • Logical Processors: 16

The model of the companion computer running BlueOS is: Raspberry Pi 4B (RAM: 4GB)

1 Like
5

Regarding the issue of page stuttering, we conducted the following tests:

  1. When the page stuttered, we checked the temperature of the Raspberry Pi, which was about 50ā„ƒ. The CPU usage was about 20%, and the memory usage was about 20%. The CPU frequency was dynamically adjusting, but there was no throttling.
  2. Using the controlled variable method, we tested whether temperature, video stream, or browser (IE or Chrome) were factors causing the cockpit stuttering. The conclusion was that none of these were the main cause of the stuttering.
  3. When the page stuttered, refreshing the cockpit front-end page made it smooth again, but after using it for a while, it would become stuttery again. This requires frequent page refreshes, which is very inconvenient.
1 Like
6

Hi Yu!

About the problem youā€™re having where cockpit becomes laggy with time, that was actually a memory leak bug that Iā€™ve fixed on Beta22.

Could you update Cockpit to the latest Beta and see if the problem is gone?

Thank you very much for all the feedback and patience!

3 Likes
7

After upgrading to the beta.27 version, the cockpit has become even more unresponsive.

Below is the cockpit log that has been running for some time after the update. Any assistance in diagnosing this issue would be appreciated.

ę—„åæ—.zip (375.8 KB)

Thank you for your patience and assistance!

1 Like
8

How many video streams do you have and which are their resolution/fps? If itā€™s more than one, can you try reducing it to one and see what happens? Just so we know if the problem is there.

Also, could you share your BlueOS logs? You can download them on BlueOS itself. Iā€™m specially interested in the mavlink-camera-manager and the mavlink2rest ones.

2 Likes
9

system_logs.zip (12.1 MB)

There are the mavlink-camera-manager and mavlink2rest In the system log. Are any other log contents neededļ¼Ÿ

1 Like
10

I tried not pulling any video streams in beta 21, but I still got stuck. I have not yet tested for the latest beta27 to see if there will be any delays when reducing the number of video streams. Iā€™ll try to do that. :thought_balloon:

1 Like
11

Ok. Let me know.

Also, which time zone are you in? If possible it would be interesting for us to do a call so I can see your problem live. Itā€™s still not clear to me whatā€™s happening.

Iā€™m on the brazilian timezone till saturday and will be on pacific time for 2 weeks them.

2 Likes
12

Very nice! Can you add me on Instagram?
My account is 12345678beyond and my nickname is Harry.

2 Likes
13

Just pointing here that we did a dedicated call with @xiaoliyuā€™s team and found out that there were two problems (outside of Cockpit), that were easily solved:

  1. The laptop was running on power save mode, which was sometimes causing rendering lags (although this was not the main problem)
  2. There was a TCP-client endpoint registered on mavlink-routerd that was not working properly. The endpoint was probably taking time to respond causing the router to hang. As soon as we removed this endpoint (which in the end was not needed) the lag problems were all gone. This was the main problem.

I recommend all users to check if itā€™s device is running on power save mode if having any rendering issues, as depending on the device it can throttle graphics a lot. This changes from device to device. On Cockpitā€™s side, we will continue to work on performance improvements so we can run fine even on those cases.

About the router problem, I can mention in advance here that we are working already on a more robust routing alternative, so we get rid of those issues.

2 Likes
14

Hi Rafael, would you like to share in detail how to remove the TCP-client endpoint?

Seems I have encountered the same issue which can probably solved by the same method (Cockpit 1.0.0 release! - #43 by AliceShanghai)

I was testing the latest BlueOS (1.3.1) and cockpit APP both on my Mac (Apple M1 Max, 64G Ram + 2TB) & a Rpi 4B desktop (4B RAM+32G SD Card, Ubuntu 22.04LTS). The lag of video stream is just much bigger than bearable. Thanks

15

You can follow this topic in the docs that explains how to do it.