Suddenly, BlueROV2 became unstable

BlueROV2 started acting strangely after a few days of use with the latest BlueOS and ArduSub.
Three spare units with the same settings are also in the same situation.

Sometimes they rotate or slant in Stabilize mode, sometimes the LED blinks when moving the arm, and sometimes the arm will not open.
When rotating in Stabilize mode, the QGC does not change the azimuth.
When slanting in Stabilize mode, the QGC shows a slant. When I change to manual mode, it returns to horizontal.
The azimuth may be out of alignment when restarting.

The following adjustments have been made, but the problem persists.

Sensor Calibration
Check thrusters
Balance adjustment
Replacing the QGC’s computer

What should I do?

Hi Itaru!
That’s very strange behavior for sure!
Did you find this to be true for all 4 of your ROVs?
Have you tried calibrating the unit in an area with low magnetic interference? Some areas of large vessels are not well suited for calibration - you want to be far away from big pieces of iron or steel. A recalibration in a different area might help things?
I assume all the thrusters are spinning ok and running in the right direction.
Have you performed the “level horizon” option in QGC with the ROV flat on a table?

Thank you.
I had not thought about magnetic interference.
I will try the configuration outside in a larger area.
There are 4 units that rotate. One of the arms is not right.
When the LED flickers when the arm opens and closes, the arm stops opening and closing after a while.
It is not cured by rebooting in blue OS.
It is cured by removing the battery and rebooting.

Hi Itartu -
I’m not sure I understand. You’re saying all 4 ROVs exhibit the rotating bug?
Of those 4, one has an issue with its gripper? Which LED is flickering when you actuate the gripper? That sounds like a bad power connection potentially?
If rebooting the vehicle fixes the gripper issue, that is very strange…

You’re saying all 4 ROVs exhibit the rotating bug?

Yes.

Of those 4, one has an issue with its gripper?

Yes.
LED = Lumen Subsea Light
It happens when I’m using it underwater, so I don’t know which one is blinking.

I also suspect BlueOS because I have never upgraded the BlueOS version before.
This has never happened before.

Does BlueOS do the attitude control? Or is it QGC?
I would like to determine if I should also suspect the network.

Is there any way to check if the Navigator Flight Controller is malfunctioning?

Ok!
If only one of the lumens in a set is flickering, it is likely a problem with that individual unit. If you reach out to the Blue Robotics support team we should be able to process a replacement - please provide the order # for the lumens.
If you’re running BlueOS 1.1.1, you are on the latest and best version to be using at this time (October 2023.) If not, upgrade to this version.
Attitude control is handled by Ardusub, which is running as a part of BlueOS. QGC is only used as a ground control station, that displays telemetry and sends commands. I personally enjoy the new cockpit extension over QGC from a usability standpoint!

We’ve seen this sort of roll bug crop up in some limited cases recently, our software and support teams are investigating. I don’t think the Roll issue is related to network or any problem with your hardware. I would assume this roll issue only occurs in stabilize or depth-hold mode?

If all of the lights are flickering when you actuate the gripper, double check what the servo function for the gripper channel is set to. If it accidently got set as the same channel output channel as the lights, the signals for one could influence the other.

Hope that helps!

I would assume this roll issue only occurs in stabilize or depth-hold mode?

Yes.
If downgrading the version of BlueOS or Ardusub might cure the problem, I would like to try it.
Can you tell me how to downgrade and which version is better?

As for the lights, I will change the channel of the gripper.

Hi @tony-white

I have tried everything and have not been able to solve the problem.

I tried BlueOS 1.1.1 and the following video is what I tried.
I am diving BlueROV2 in stabilize mode, but it rotates.
The same thing happened with BlueOS1.0.1.


I have tried changing to cockpit extension,
The camera switching is not in the joystick settings.
I have two USB cameras in my BlueROV2.

I also use the NodeRed extension to assign the camera zoom to the joystick button, though,
I can’t easily switch between the two because that stops working too.

This is not the original problem, but…
I was thinking about this when I used the cokpit extension,
Will the joystick settings be written to Ardusub?
The joystick settings in QGC are also changed.
Furthermore, even with the same controller (F310), the number of buttons is different (different recognition of L2 and R2), so everything is off.

In other words, I feel that this is a bug because BlueOS and QGC are both telling Ardusub what to do.
I’ve been using BlueROV2 since before BlueOS and I think it was more stable then.

Would changing the version of QGC make it better?
Is there a version you would recommend?

Hi Itaru -
Unfortunately, changing the BlueOS version will not affect this issue. It may have been an issue with Ardusub! I was able to recreate your issue with BlueOS 1.1.1 and Ardusub Stable, but when I updated to Ardusub BETA from the Autopilot Firmware menu in BlueOS, the issue was cured!

QGC or Cockpit are only used to display telemetry and relay control. Ardusub is the firmware that orchestrates stabilization of the ROV. BlueOS is the glue that binds it all together, making integrating new applications easier than ever before!

If you try Beta and it resolves or doesn’t resolve your issue, please let us know! Best of luck!

Thank you @tony-white
I’ll give it a try.
Could you please provide me with the link to the Ardusub firmware that you tried, just to be sure?

Hi Itaru -
You shouldn’t need to use a link - it is actually better if you don’t! Just make sure your ROV has internet access (via Pi WiFi or bridging your control computers connection to the Tether ethernet adapter), and update from the Autopilot Firmware interface in BlueOs 1.1.1.

I’ve tried it a few times, but it often takes a long time to find.

If it fails, I’ll try the following:
https://firmware.us.ardupilot.org/Sub/beta/navigator/

And If beta fails, can I try the following?
https://firmware.us.ardupilot.org/Sub/latest/navigator/

Hi @tony-white

I tested with Electronics Enclosure only on my desk.

When I changed the firmware to beta, it stabilized a bit, but after about 30~60 minutes from startup, it became unstable again.

The instability of BlueROV2 seems to be due to thermal runaway of the camera.
When the camera stops due to thermal runaway, the rapsberry pi automatically disarms the device.
When the camera cools down and starts working again, it repeats the process of auto-recognizing the device.
In other words, it seemed that the camera was being pulled in and out violently, increasing the processing of the raspberry pi, which was causing the processing to slow down.

After it became unstable, it stabilized when I unplugged the camera.

Also, even with the camera attached, from the terminal
“sudo iwconfig wlan0 txpower off”
It seemed to stabilize a bit.

I would like improvements in the handling of the camera.

Hi Itaru!
Glad to hear the update to beta at least partially corrected the issue. In your test, was the enclosure sealed for the 30-60 minutes? And then you had camera frame-rate issues, but not the roll issue?

The system is designed to operate underwater, and the cooling benefits this provides usually prevents the camera from having the issue you faced.
Turning WiFi off, either with the terminal command or via the icon in the upper right can improve video performance in cockpit - this is a known issue that we are resolving.
I’ve never see the system disarm because of a loss of camera stream - that may be something you configured?

If you test with the ROV in the water, please share how it goes!

I removed the USB camera and installed an IP camera and tested it on land.
I used a network hub from the Raspberry Pi and connected it to my computer.
After starting up BlueROV2(ArduSUB4.1.1BETA7) and QGC, the video blinked after about 5 minutes. (Heatbeat temp 40C)
At that time, Video Streams in BlueOS (1.1.1) gave an error.
Is BlueOS 1.0.1 more stable?

I am not sure what to use anymore.

No, the latest stable version is definitely the best to use. Can you please share screenshots of the error you are receiving? Screenshots of your camera configuration under Video Streams would also be helpful. Video is stopping after 5 minutes? How is the IP camera powered?
The issue you refer to there may be related to having WiFi connected or the interface powered on with hotspot. Have you turned off the WiFI connection from the icon in the upper right?
Have you tried using the Cockpit extension?

No, the latest stable version is definitely the best to use.

It seems like the people above resolved it by downgrading the version, but why?

Can you please share screenshots of the error you are receiving?

I can’t take a screenshot because it only appears for a moment.

Screenshots of your camera configuration under Video Streams would also be helpful.

image

When it blinks, the camera list disappears.

Video is stopping after 5 minutes?

The video blinks every few minutes.

How is the IP camera powered?

It’s on a separate power source, and the camera continues to function properly.

The issue you refer to there may be related to having WiFi connected or the interface powered on with hotspot.
Have you turned off the WiFI connection from the icon in the upper right?

I don’t remember how the Wi-Fi was at the time.

Have you tried using the Cockpit extension?

The video also blinks in the Cockpit.

You could benefit by following this other topic:

To my beginner’s eyes, they seem related.

Hi @Itaru -
You are likely suffering from a bad USB camera - this has been a frequent issue for us lately. Please reach out to request a replacement.

@Jan_Silvino - the pitch roll drift topic is unrelated to the camera issues experienced here.

Hi @tony-white

You are likely suffering from a bad USB camera - this has been a frequent issue for us lately. Please reach out to request a replacement.

The camera in the BlueROV2 Electronics Enclosure, which I purchased, didn’t start, so I had it replaced in August 2023. The symptoms persist with the replacement camera. Additionally, a BlueROV2 purchased in September also developed the same issue recently, so I purchased an additional camera to replace it.

Is it appropriate to email support@bluerobotics.com for the replacement process?