How do I use MAVLink Endpoints to get gamepad data?

Hey there!

For a custom project, I’m running BlueOS on a raspberry pi 4B, with no other hardware attached. Using the cockpit extension, I have a fantastic camera stream coming from the picamera, and I’m very impressed!

I want to write a python program on the PI to read the joystick and button data coming from cockpit. I also want to write python program on the pi that populates fields on the cockpit such as the depth, battery, compass etc.

Is this possible?

It seems like MAVLink endpoints with pymavlink are the way to go, but I’ve been having trouble getting them to work. The MAVLink inspector doesn’t show a MANUAL_CONTROL message, despite the docs saying here that “MAVLink messages are automatically sent to the vehicle at 25Hz…” How can I listen to those?

In other words, how can I listen to joystick data on the PI, and publish data from the PI to display in cockpit? (Preferably using python)

Thank you!

2 Likes

Hi Mark,

Iam not familiar with Cocpit, since I use ROS. To control your vehicle with a joystick, you will need to set (by Joystick)some PWM values using Mavlink. ROS, along with MAVROS packages, provides a great interface to move your vehicle using joystick.

I highly recommend using the latest achievement from @evan-palmer, which you can find here (ROS 2):

blue
documentation - joystick

For ROS:

bluerov_ros_playground

Regards,
Markus

1 Like

Hi Markus,

Thank you for your reply. Using ROS may be an alternative I look into, although I would love to take advantage of the features Cockpit offers.

@rafael.lehmkuhl Do you have any insight into how I might use pymavlink to read joystick data coming from Cockpit?

When I navigate to the Mavlink2rest webpage, I see there is a MANUAL_CONTROL message being received/sent in some way:

When I click the watcher link in a separate tab, and focus on the cockpit window while moving the joysticks, nothing changes. The values are all 0. The watcher for MANUAL_CONTROL doesn’t seem to update at all, despite reporting a frequency of 2.53Hz.

I created a MAVLink endpoint, a UDP Server at 0.0.0.0:9000 in hopes I could capture the MANUAL_CONTROL message via python, but when I connect to it, the only packets I receive are those sent by the SITL autopilot ArduSub firmware.

In my custom project, it doesn’t make sense to have autopilot and it’s associated firmware started/active. Turning it off of course stops the heartbeat and SITL telemetry coming over MAVLink, which is okay, because really the python programs I wish to write should provide those.

If you could help me understand the specifics of using pymavlink on the Raspberry PI to send/receive data from cockpit I would really appreciate that!

This is possible and a very interesting idea!
I’m myself am building a non-ardupilot vehicle and using cockpit for seeing the video stream. I was planning on using an Iframe widget to capture input (mostly keyboard for now), but getting the joystick input is a cool idea, too!.

You are looking at the right place. MANUAL_CONTROL will contain 4 of the joystick axis and the button presses as a bitmask.

Mavlink2Rest had a bug recently where the watcher was broken it seems fine on the latest master.

You should be able to capture GCS data (MANUAL_CONTROL) messages if you listen from vehicle ID 255 and component ID 240

IF you inject mavlink data into the stream (as vehicle 1, component 1) you should be able to populate the widgets, though I haven’t tried. I’m also not sure if a heartbeat is required.

1 Like

Hi Mary,

The MAVLink Protocol used by MAVROS and other Robot Operational Systems is very flexible and easy to use. There are several stardard MAVLink commands present on field MessageID. If the MessageID(Command) you need is not present on default list, you can create your own and then defines the payload data you want. Thos protocol was created to control AirPlanes and standard commands are present on them. Check if there are Joystick present on remote control equipments and use them. If not, Its possible you have to create your own MessageID(Command) and use one that is not present on standard MessageID(Command List) to avoid conflits.

1 Like

Hey Mark!

You’re on the right direction.

The MANUAL_CONTROL message is indeed being updated, and you should be able to consume from it.

Ignore the watcher page. It is not working for messages being sent from anybody but the vehicle. It’s a problem on mavlink2rest and @patrickelectric is working on it (issue to track).

About the problems with pymavlink, I think @EliotBR is a better person to help you, as it uses it frequently.

1 Like

Thank you all so much! This is super helpful, I really appreciate it.

1 Like

After some testing, I’d like to follow up with some findings and a related question.

I tried out the iFrame to get keyboard and joystick data, and it does work! However, I’d like to use the existing widgets in cockpit instead of recreating them. For this, I’ll have to go with @williangalvani 's idea and inject mavlink data to the GCS. (more on that at the end)

Though pymavlink doesn’t seem to work for getting MANUAL_CONTROL messages from the GCS, sending a get request via mavlink2rest for the MANUAL_CONTROL message returns the updated state of the gamepad. Awesome! This is what the get request looks like (sent from pi)
http://127.0.0.1/mavlink2rest/v1/mavlink/vehicles/255/components/240/messages/MANUAL_CONTROL

Stopping the Autopilot causes the heartbeat to be lost, and the get requests return the state of the gamepad when the autopilot was stopped.

Starting the Autopilot immediately brings things back to life and the get requests resume returning updated gamepad states.

From this I concluded that without the Autopilot started/active, the MANUAL_CONTROL messages will not be updated.

I assumed this was because of the loss of the heartbeat coming from the Autopilot, so I tried sending my own heartbeat in place of vehicle 1 component 1. I used the mavlink2rest guide, and was able to successfully send the heartbeat and see it show up at 1hz in the mavlink inspector.

However, the heartbeat indicator in blueOS remained lost, and the problem above persisted. In fact, injecting any kind of mavlink data (with a POST request) shows up on the mavlink inspector, but not cockpit. Additionally, if I had autopilot running, and I tried to send a post request to inject some mavlink data, it causes Autopilot to crash.

How can I send mavlink data (including a heartbeat) to BlueOS & Cockpit and make use of it’s widgets when Autopilot is disabled?

1 Like

Strange. From Cockpit’s side I just tested and we keep updating the joystick state and sending the MANUAL_CONTROL messages even if the vehicle disconnected.

@patrickelectric do you know if there’s any limitation for that on the mavlink2rest side?

Also @MarkFromCanada could you check if you didn’t got Cockpit out of focus during your tests? While you get the tab out of focus (by changing to another tab or minimizing the browser) the browser does not forwards the joystick updates to Cockpit, so you would indeed keep receiving the last state (if you have the configuration below enabled on configuration/joystick).

Hi Mark,

I believe that is not a problem of mavlink2rest. Can you provide your code and steps to replicate it ?

For sure,

@rafael.lehmkuhl I indeed had the tab in focus, and I think some more details will help

Here’s the exact steps to reproduce the joystick issue:

  1. I start the autopilot firmware (in the autopilot firmware section)
  2. I open Cockpit and focus on it
  3. I start the following python script on the pi:
import json, requests, time

IP = '127.0.0.1'

def getMessage(vehicle,component,message):
    response = requests.get(f"http://{IP}/mavlink2rest/v1/mavlink/vehicles/{vehicle}/components/{component}/messages/{message}").json()
    return response

while True:
    response = getMessage(255,240,"MANUAL_CONTROL")['message']
    print(response)
    time.sleep(1/25)
  1. I wiggle the joystick and see the values coming in as expected in the terminal output of the program

  1. I stop the autopilot firmware
  2. I refocus on Cockpit and wiggle the joystick, and the python output does indeed keep updating current values as expected (so yes, the MANUAL_CONTROL message does get sent when the vehicle is disconnected… but the next step is where things break down)
  3. I then reload Cockpit and make sure it’s in focus. This time, when I wiggle the joystick, the little joystick icon is lit which means the browser recognizes the joystick, but the python output is frozen (reporting the same outdated values at 25hz)
  4. I start autopilot again, then focus on the Cockpit. I wiggle the joystick, and program output is back to updating current values as expected.

There seem to be 4 states:

  1. If Autopilot is started before opening the cockpit tab, the MANUAL_CONTROL messages are sent regularly.
  2. If Autopilot is stopped while the tab is open, the messages are still sent regularly
  3. If the tab is closed and re-opened, or reloaded, the messages are not sent… despite the browser receiving the joystick inputs (this is the one that confuses me)
  4. If the cockpit tab is opened while the autopilot is stopped, and then autopilot is started, the messages begin sending immediately after starting autopilot.

For the heartbeat issue, here’s how to reproduce it:

  1. I stop the autopilot firmware
  2. I start this python script running on the pi. It sends a heartbeat message at 1hz.
import requests, time

IP = '127.0.0.1'

def sendHeartbeat():
    body = {
        'header': {
            'system_id': 1,
            'component_id': 1,
            'sequence': 0
        },
        'message': {
            'type': 'HEARTBEAT',
            'custom_mode': 0,
            'mavtype': {
                'type': 'MAV_TYPE_SUBMARINE'
            },
            'autopilot': {
                'type': 'MAV_AUTOPILOT_GENERIC'
            },
            'base_mode': {
                'bits': 128
            },
            'system_status': {
                'type': 'MAV_STATE_STANDBY'
            },
            'mavlink_version': 2
        }
    }

    response = requests.post(f'http://{IP}/mavlink2rest/v1/mavlink', json=body)
    return response

while True:
    print(sendHeartbeat())
    time.sleep(1)
  1. I see the heartbeat message is successfully received on the mavlink2rest page at 1hz for vehicle 1 component 1 (you’ll notice the other messages last updated 300+ seconds ago, because autopilot was updating those, and autopilot is stopped)

  1. I don’t see the message arriving in the mavlink inspector, and the heartbeat icon shows that it’s lost.

Also, if I try sending any other kind of message, like STATUSTEXT for example, nothing happens. It seems like the messages arn’t reaching BlueOS, despite being reportedly recieved in the figure under step 3.

I have a hunch that solving the heartbeat issue might solve the joystick issue, but I’m not sure.

Here’s some more information that may be useful

When the Autopilot is started, cockpit displays this:

When the Autopilot is stopped, it immediately changes to this:

Thank you for the detailed explanation! Now I got the problem!

Which happens is that we indeed wait for the vehicle connection to start the MANUAL_CONTROL messages, so if you restart Cockpit, it will wait for vehicle connection to begin sending those messages.

We do that because we need vehicle connection to know which parameter is set on the vehicle for each functionality (arm, disarm, etc), before actually sending those. This is because we have a logic inside Cockpit that abstracts the vehicle parameter buttons settings to functions, so we can facilitate setup for the user. The downside is that we need an initial vehicle connection to be able to send meaningful MANUAL_CONTROL messages.

What we can do in Cockpit’s side is to allow you to overwrite this and just send the raw data as it is, but I want to understand what would be the use case for that, to make sure we are in the right path.

Thank you! That makes perfect sense.

Here’s the big picture: I was tasked with using BlueOS to control a mini ROV, but with different hardware that is (to the best of my knowledge) incompatible with ArduSub. I was hoping to make a blueOS extension to house some python programs that would control the hardware, but also communicate via MAVLink to Cockpit. That way, I can use the widgets (like battery, compass, artificial horizon) to display telemetry, and use the joystick to drive it.

So I was really hoping it was possible to have my python programs “replace” the autopilot by sending a heartbeat, and other telemetry MAVLink packets that the autopilot would normally send… but so far all my testing with pymavlink and mavlink2rest hasn’t worked out. Is it possible?

It’s definitely possible! And your use case is very much valid.

About the heartbeat, I think patrick will be better able to help you.

About the MANUAL_CONTROL messages, I opened an issue to track that. I will try to take a look in the next days. If you have knowledge on Javascript, contributions are also welcomed :slight_smile:

1 Like

Thanks for your help Rafael! I’ll take a look and see if I can contribute.

And definitely @patrickelectric I would love some help with the heartbeat, and updating the widgets via MAVLink data sent from the PI, like what Willian was talking about here:

Thank You @rafael.lehmkuhl @williangalvani @patrickelectric for helping Mark with this custom implementation of BlueOS & Cockpit. Mark is working with me at The Quest Institute to get Cockpit working on our custom educational mini-ROVs we are working on for a summer program we are planning this summer. We are also wanting to gain some more familiarity with the software for possible implementation on some other projects associated with our partnership with MBARI and our 1000m BlueROV2.

2 Likes

@MarkFromCanada one question: how critical it is for you guys to keep using MAVLink MANUAL_CONTROL messages for the commands part? I’m asking because I’m seeing that we are more tied to the vehicle then I remembered.

Discussing with my colleagues, we agreed that we should untie them, so that Cockpit works with MAVLink without needing a Vehicle instantiated, but this will require some good amount of work, specially with this MANUAL_CONTROL part, which is highly tied to the vehicle parameters today.

I’m thinking on a way to support the usage of the MAVLink commands in general without a vehicle connected, which is easier, but in this case, this would not include the MANUAL_CONTROL initially, at least not the buttons part (which is the more complex part). Supporting the axes is easier, as they do not need parameter mapping.

For commands (apart from the axes) I would think that using the COMMAND_LONG message would be much better for you guys, as it’s much more flexible (you can send just a message or send it with parameters) and you can also answer with a confirmation or error to the GCS. You can also use the NAMED_VALUE_FLOAT messages, although they are usually used for sending sensors data, not commands.

Also Mark, for you to be aware: right now BlueOS does not support having the MAVLink router running without an autopilot running. I’ve opened an issue on the BlueOS repository to track that.

So you can have it working for now I suggest keeping the SITL instance of BlueOS running or instantiating a router alongside your script.

Update: Mark, can you clone the Cockpit repository and test this PR? I didn’t test it yet, but it should enable the forwarding of the MANUAL_CONTROL messages by Cockpit, even if a vehicle was never connected.

Cockpit will send those messages at target_id 1, which is the default (we don’t support changing it yet). Also, only axes will be sent for now (as for the buttons we need the parameters or a big rework).

Let me know if you can properly receive the axes updates and if the COMMAND_LONG way works for you. If so I will continue in this path.