I’m using the BlueOS BETA installed on a raspberry pi 3B+ so I’m expecting some issues but I hope there might be a known fix for this.
When connecting to the autopilot through the companion computer it often takes minutes to get a mavlink connection in qgroundcontrol if at all. I’m able to see the video feed while it shows ‘Disconnected’ but it rarely connects right away. However, if I wait with qgroundcontrol open for 2-5 minutes it usually connects.
Everything seems to work fine for a few minutes but the video begins to stutter and the mavlink update seems to slow down to 1-2Hz. Restarting qgroundcontrol generally fixes this but the issues then return after a few mins. Using a serial connection to the autopilot (Matek H743 V2) always works fine.
Thanks for your help!
UPDATE
I’ve done some more investigation today but still no luck. I updated to the BlueOS BETA version 11 and I now get the following error:
I’m sure it’s related to the problem but I’m not sure what to do about it.
It does seem to locate the autopilot board as ‘Pixhawk1’
If I reboot the companion computer 2-3 times while leaving the autopilot powered it will finally connect properly and all of the MAVlink data starts flowing:
Vehicle ID: 255 with Component ID: 190 is QGroundControl (you can look at the MAVLink Routing and MAV_COMPONENT docs if you want to better understand how the IDs correspond to different systems).
Given you’re seemingly always getting at least the QGroundControl connection, and other times also the vehicle connection, it seems your Companion computer is struggling to connect to the autopilot (or just taking a while to do so).
I’m curious - if you try turning on the vehicle without QGroundControl open on your computer, do you get Vehicle ID: 1 with Component ID: 1 showing up consistently? There will only be a few message types shown, because the other ones are normally requested by QGC (they don’t get sent by default). If you restart the autopilot via the web interface, how long does it take for the heartbeat to return in mavlink2rest (with a Pixhawk for me is < 10 seconds)?
Here’s what I’m seeing. UPDATE Further testing has shown the following not to be true all of the time–> The following start up process is repeatable for me now 100% of the time.
1. Power up system. Raspberry Pi and Autopilot are both powered up simultaneously. Nothing is reported in mavlink2rest:
2. Restart system by clicking on the reboot button on the companion web interface. Autopilot remains powered. The following information is displayed in mavlink2rest:
That seems odd, especially if it doesn’t end up coming online. As I understand it there’s supposed to be constant scanning for new mavlink devices (which is why QGC can be detected despite not being available on startup), but it’s possible the USB connections are only handled once on startup, in which case if your autopilot takes too long to start then it won’t be detected. I’ll confirm with the software team and raise an issue if relevant
Could you open the terminal service (on the tools menu), access the autopilot tmux (ctrl+b s) and send us the terminal logging? It can be a screenshot. With this it will be more clear what is happening (and your point about startup delay is problably right).
Sorry for the not-so-straightforward way of sending the logs. We are working on improving that
I think I’ve found the problem, but I have no idea what the solution is.
If the USB camera is not connected at start up, the connection to the autopilot and the mavlink2rest connection works perfectly every time.
If the USB camera is connected the mavlink connection only works about 10 - 20% of the time.
There is a clear relationship between the USB camera being connected and the ability to connect to the autopilot, but I’m not sure where to go from here.
Thanks again everyone for your help!
UPDATEThis is not the issue either. Even though I was getting consistent success booting everything successfully about 10 times in a row last night, this doesn’t seem to be the case this morning. It still seems completely random at this point.
If you press 2 while on that screen it should take you to the autopilot window, which is the one he’s after.
That’s interesting. Perhaps the autopilot detector is mistaking the camera for an autopilot, and trying to connect to it. The variability might depend on which device it detects first on the USB bus. Most likely that’s something we’ll need to fix in BlueOS, although it may be helpful to organise a video call/team viewer session if the autopilot terminal logging doesn’t yield enough clues to resolve the issue.
@rafael.lehmkuhl I’m curious if this is a pure USB mixup issue, or if it could be a side-effect of the camera manager presenting the cameras as MAVLink devices (admittedly they shouldn’t be presented as autopilots, but the autopilot detector may not know the difference).
We detect a USB device as an autopilot with UDEV rules. Basically, if a USB device has “3D robotics” or “ardupilot” in its manufacturer, or “PX4” on its product name, it will be detected as an autopilot.
About the screenshot of the autopilot service, as @EliotBR said, clicking “2” will open the autopilot terminal.
About the camera being detected as an autopilot, if it doesn’t appear in the previous screenshot, could you send a dmesg -w command, remove and reinsert the usb camera and send the result? The dmesg command should show the camera USB data (name, product name and manufacturer), which will help us see if that’s the case.
If you could do a call with us, I would be happy to help diagnosing this.
I have some more useful screenshots for you. Both of these are from an “unsuccessful” start up. in this case UDP video transmission is working fine but the mavlink is not.
No, as far as I can tell the restart button doesn’t seem to do anything. I’m not even sure it is rebooting the board. Do you know of any way to check if it’s actually restarting?
I have also noticed in QgroundControl in the MAVlink status in Application Settings shows a loss rate of 99%. When I connect to the autopilot via USB I see a loss rate of 0%. Is this an actual issue or just a difference in the way it’s reported?
It’s probably because the mavlink router on BlueOS is indeed not working.
I’m trying to reproduce the error here with a Pixhawk1. I will also create a dedicated image with some modifications to see if we can get more clear messages from the router.
After more testing I have found that if I remove the VBAT power to the autopilot and power it only from the USB on the raspberry pi, it will start successfully about 70 - 80% of the time. This is the most reliable configuration I have found so far. With the autopilot independently powered it would only connect about 25% of the time.
I’ve created a new image that will make what is happening to mavlink-router more clear for us.
Could you try it? To download it, you just need to go to the version-chooser on BlueOS (menu/tools/version-chooser), replace the bluerobotics/companion-core (under “Remote Versions”) my repository (rafaellehmkuhl/companion-core). You can then click on Pull and Apply right after the fix-mavlink-manager-logging-simple image. After about a minute it should be running, and I would ask you to send a new screenshot from the autopilot startup. It will have the mavlink-router startup info.
Also, you can disable this “GCS Server Link” endpoint (pencil icon, bulb icon). I’m not entirely sure, but it can mess up things, as we already have the “GCS Client Link” connection to QGroundControl.
