Pixhawk running Ardusub outputs max throttle PWM signal upon arming

Blue Robotics Software ArduSub
,
1

Hi all, I have been having a really hard time getting my Ardusub-based system to work properly and could use some help.

Context

I have a Simple ROV-3 frame with three Blue Robotics T200 thrusters, each with the Blue Robotics Basic ESC. This is a basic layout of the system:

Raspberry Pi (Raspberry Pi OS) <----[pymavlink scripts/MAVLINK]----> Pixhawk 6c (Ardusub) ----[I/O PWM]----> Basic ESC (x3) ----> T200 thruster (x3)

I also am using a Blue Robotics Depth sensor, the Bar03, and am using the onboard compass and IMU in the Pixhawk.

The Problem

The simplest description of the problem is that the PWM output of the Pixhawk is not what I expect it to be. Specifically, the PWM values for channel 3, the pitch thruster in the Simple ROV-3 frame, jump to 1900 without me sending any control commands. This happens after arming, and the time it takes for this to start happening is random. I have made sure that the parameters setting the trim PWM values for all three motor channels are set to 1500, which is neutral, but this still happens. If I subsequently send a command to change these values, either through manual_control or set_servo_pwm, channel 3 does not respond.

Also, this is happening on a brand new Pixhawk. I previously was using another Pixhawk 6c, but I accidentally shorted it when I was moving some things around my setup. With the old Pixhawk, I couldn’t even get channel 3 to output anything, although I did not get much time to troubleshoot this before it shorted.

After a lot of troubleshooting, I am starting to wonder if this is an issue with the firmware itself, as I have gone through many of the Pixhawk configuration parameters, including the FS_* ones to see if a failsafe is being triggered, and there has been an issue with channel 3 for both Pixhawks I tested out. I am definitely open to other explanations and troubleshooting suggestions.

Has anyone else here had a similar issue before? Any help is greatly appreciated! Thanks in advance.

Other relevant notes:

  • I am always operating in manual mode
  • The disarm command is able to set the thrusters back to neutral
  • I have been able to test the servos in the QGroundControl motor testing page with normal behavior.
  • I set the Pixhawk to a frame with 4 thrusters instead and this time saw both Ch3 and Ch4 misbehaving after arming.
  • I briefly thought I had fixed this issue be restarting my entire system, including the laptop running QGC. But the problem came right back and I realized that this was just an instance where it took a little longer for ch3 to get to 1900 than it had in previous attempts.
  • I cannot control the thrusters with my Logitech controller through QGC. I can arm and disarm if I use Mission Planner, but have not been able to control the thrusters.
  • rc_channel_override commands have not worked.
2

Hi @fishbot -

Check out your failsafe parameters - it sounds like you might be triggering surfacing mode somehow! Sharing a vehicle log from a problematic episode would allow us to troubleshoot with much more capacity! As well as any custom code you’re trying to use…

Are you controlling the vehicle with QGC or Cockpit? Or only your own code? If the latter, it may be an issue with not sending a heartbeat…

3

Hi Tony, thanks for your reply. I went through and disabled all of them except the heartbeat one, but the issue persists.

I have not been able to send any commands successfully from QGroundControl with our Logitech F310. I was able to send arm/disarm commands using Mission Planner, but not any thruster commands. This seemed like a separate issue to troubleshoot later, since I was able to send commands to thrusters 1 and 2 using python scripts, so I tabled that and stuck to using just my own code.

I have attached two log files (log 9 was taken last week and log1 was taken today). I have also attached my parameter list and a sample script I have been running.

log_1_UnknownDate.bin (326.0 KB)

log_9_UnknownDate.bin (1000.0 KB)

Pixhawk_config.params (28.3 KB)

testscript.txt (1.7 KB)

4

Quick followup: This param file shows RC3_trim set to 1100. This just happened because I completely wiped and reconfigured my Pixhawk earlier today. I set it to the correct value of 1500 and still witnessed the same problem I have been describing.

5

Another update: I just borrowed a fresh fully functional laptop and a completely new Pixhawk 6c and microSD card to try again, and I had the same thing happen.

This time, I do not have anything connected to my Pixhawk other than the laptop cable and an oscilloscope on the I/O PWM OUT pins. I downloaded the recommended stable release of QGC to the new laptop, flashed the stable ChibiOS release of Ardusub, calibrated the accelerometer, compass, and joystick (Logitech F310), and reconfigured just the following parameters:

  • SYSID_MYGCS = 1
  • SERIAL2_BAUD = 921600
  • FRAME_CONFIG = 4 (Simple ROV-3)
  • RC3_TRIM = 1500
  • MOT_1_DIRECTION = -1

I rebooted after every operation. The oscilloscope showed all channels at PWM=1500. Then I went back to the main QGC screen, made sure manual mode and joystick were enabled, and then tried to arm with the joystick buttons I had assigned. This did not work, so I clicked the “Arm” option from the dropdown menu using my laptop cursor. When I did this, I saw that the PWM values for just ch3 changed from 1500 to 1900 on the oscilloscope. Then I disarmed and ch3 returned to 1500.

So this can’t be an issue with my code. Either I am missing some key parameters to configure, or something is the matter with QGC, the Pixhawk, Ardusub, or some compatibility issue between these. I have attached log and parameter files from QGC.

log_1_UnknownDate.bin (840.1 KB)

basic_config.params (28.2 KB)

To rule out QGC, I tried Mission Planner which I see is the recommended GCS for Ardusub anyway. I downloaded it, opened the program, connected to the Pixhawk, and then immediately armed it without even setting up the joystick, since I had the controller physically disconnected anyway. The same thing happened, PWM for ch3 shot up to 1900 when armed and then back to 1500 when disarmed.

This brings the root of the problem back to Pixhawk parameters, the Pixhawk itself, Ardusub, or just compatibility of Ardusub with Pixhawk 6c.

6

Final update everyone: I fixed the PWM problem! But my code caused the original problem and I am not sure why. This led me down a kind of a crazy troubleshooting rabbit hole that definitely made things worse. Either way, I definitely jumped the gun a bit thinking it was a Pixhawk or Ardusub error.

Fundamentally, I had to change 2 parameters:

  • I had SYSID_MYGCS=1 but it needed to be set to SYSID_MYGCS=255
    • I had originally set it to 1 to communicate with my Raspberry Pi, but I should have changed it back to 255 to use it with my GCS. This fixed the problem of my joystick not working properly.
  • I had RC_TRIM=1500 but it needed to be RC_TRIM=1100
    • Setting RC_TRIM=1100, which is the default, fixed the channel 3 PWM problem when operating the Pixhawk through the GCS with a joystick. This parameter is mentioned in the older Ardusub website, which I neglected to reference and instead was just looking at the newer wiki.

I was able to get correct PWM values on all the channels when using my gamepad through Mission Planner. However, when I tried to run the pymavlink code that I uploaded in an earlier post, the ch3 PWM went back to 1900 during arming. I am not sure why this is happening, since all I am doing is using arducopter_arm(). Is there something I am misunderstanding about pymavlink? I do really need to get this script to work so that I can interface our system with our other research projects.

7

Hi @fishbot -

Without sharing your code that’s causing the issue, I’m not sure how we’re supposed to provide troubleshooting feedback! If the system works nominally with a standard GCS (although Blue Robotics has never recommended mission planner), then as you’ve deduced it’s likely something you’re doing.

Is there a reason you’re developing your own GCS? Have you tried Cockpit? It was built from the ground up to be incredibly customizable, I’d guess it could configured to meet your research needs…

8

testscript.txt (1.7 KB)

Hi @tony-white , sorry I should have specified that the python script I am using is the testscript.txt that I uploaded in a previous message. I reuploaded it here. Also, I am not trying to re-develop an entire GCS, I just want to be able to program in motion commands. Cockpit looks great but we want this system to eventually be autonomous and to interface with our other AUVs which currently run on python scripts. We’re just using joystick control for testing.

9

Hi @fishbot -

Both QGC and Cockpit support mission planning - does your vehicle not have a position source? Are you planning to use Auto mode in ArduSub or Rover?

Perhaps I’m misinterpreting what you’re after by “program in motion commands” - but quite a lot is possible!

10

My vehicle does not have a position source. We have other programs running vision algorithms, obstacle avoidance algorithms, etc. that we are conducting research with. We are just hoping to use this system that I am troubleshooting as a platform for this research. So we need to be able to incorporate the motion of our submersible into the code that we have already written, hence why I am using pymavlink. Are pymavlink scripts generally discouraged?

11

Hi @fishbot -

Pymavlink isn’t discouraged at all, but it can be a bit tricky to get working. AI seems to have caught the issue pretty easily, when asked to review your script vs. the docs specific to pymavlink and ardusub.

MANUAL_CONTROL “neutral” is wrong

testscript.txtLines 50-50

my_auv.mav.manual_control_send(my_auv.target_system, 0, 0, 0, 0, 0)  #send all neutral commands

manual_control_send(target_system, x, y, z, r, buttons) — so this sends x=0, y=0, z=0, r=0. The comment says “all neutral,” but per the docs the z (throttle) axis is not centered on 0:

“Because of some legacy workaround, z will work between [0-1000] where 0 is full reverse, 500 is no output and 1000 is full throttle. x, y and r will be between [-1000 and 1000].”

So z=0 is full reverse/down throttle, not neutral. For a truly neutral command you want:

my_auv.mav.manual_control_send(my_auv.target_system, 0, 0, 500, 0, 0)

Other things to check (not strictly doc violations)

  1. Blocking recv_match inside the command loop can throttle your send rate. The docs warn that movement commands (like HEARTBEAT) “must be sent at a constant rate… otherwise the autopilot will execute a failsafe.” Here:

testscript.txtLines 51-51

msg = my_auv.recv_match(type='SERVO_OUTPUT_RAW', blocking=True, timeout=1)

If SERVO_OUTPUT_RAW isn’t streaming, each iteration can block up to 1 s, creating gaps between manual_control_send calls. Usually that stream is on by default, but if it isn’t, you’d want to SET_MESSAGE_INTERVAL for it (see the “Request message interval” example) or use non-blocking reads so the command cadence stays constant.

12

Hi @tony-white, thanks for looking into this, but the problem seems to happen during arming. I can block out the entire chunk of my code commented “Send motor commands”, which would remove that manual control command, and I will still see the PWM value jump.

EDIT: Shortly after writing this, I did a little more troubleshooting and found that actually the manual_control function you mentioned was in fact the issue, and the reason it was happening even when commented out was because the value was stuck in the Pixhawk. Once I rebooted and ran the script again with that part commented out immediately, the problem went away!

Thank you very much for your time and help. At this point I’ll go ahead and mark this post resolved.