How can EKF state be updated using external computation (e.g., SLAM)?

Hi,

We are developing a program to estimate the position of an underwater drone using SLAM.

We are trying to apply the estimated position to the system, but we are having difficulty getting it to be accepted.

Using position estimated from SLAM, I performed the following procedure:

1. Connect to the vehicle.

2. Configure EKF parameters to use External Navigation:

(“EK3_SRC1_POSXY”, 6),

(“EK3_SRC1_POSZ”, 6),

(“EK3_SRC1_VELXY”, 6),

(“EK3_SRC1_VELZ”, 6),

(“EK3_SRC1_YAW”, 6)

3. Set the EKF origin.

4. Continuously send the current position using ODOMETRY or VISION_POSITION_ESTIMATE.

(Results of self-localization using SLAM)

5. Switch to GUIDED mode.

6. Arm the vehicle.

7. Continuously send commands using SET_POSITION_TARGET_LOCAL_NED.

(Target Point)

8. Monitor the state using LOCAL_POSITION_NED.

⇒Result:

The XYZ values sent via ODOMETRY are not reflected in the EKF.

※If step 4 is skipped and the internal EKF is used instead, the vehicle tries to move toward the position specified in step 7.

ArduSub:V4.8.0-dev

I have never used ODOMETRY, but I was able to get VISION_POSITION_ESTIMATE to work in SITL. I decided to use the barometer for Z, and use the SLAM for everything else.

How was the EKF while the SLAM system was sending data? The internal EKF is always running, and will fuse the internal IMU with the external SLAM data in this configuration. It might reject “bad” readings from your vision system. You can monitor the EKF_STATUS_REPORT MAVLink message during a dive.

Take a look at the dataflash (BIN) logs, they will tell you a lot. MAVExplorer.py is a good tool, and there are many others. The VISION_POSITION_ESTIMATE messages should be logged in the VISP table. You’ll find EKF outputs and health data in the XKF* tables.

Could it be as simple as rebooting the autopilot after changing the EK3_SRC parameters?

Sub makes some strong assumptions about being underwater… if the SLAM system says it is floating above the water, I am not sure what will happen.

Note that Cockpit has a dedicated EKF Status mini-widget just for this :slight_smile:

Thank you for your response.

I tested VISION_POSITION_ESTIMATE in SITL as you suggested.
However, only the XY position is being rejected, so I am currently testing with a simplified sample code (attached).

At this point, I have not yet been able to identify the reason why it is not being accepted.
I have also made sure that the XY values are not significantly different from the EKF state, but they are still not accepted.

It is possible that I may have a fundamental misunderstanding of the expected behavior or specifications.
If there is anything obviously incorrect in my approach or setup, I would greatly appreciate your guidance.

I have already rebooted the autopilot after changing the EK3_SRC parameters.

# %%
import math
import time

from pymavlink import mavutil

# connection
master = mavutil.mavlink_connection("udp:127.0.0.1:14550")
master.wait_heartbeat()
print("Connected")

# parameter setting
master.param_set_send("EK3_SRC1_POSXY", 6)
master.param_set_send("EK3_SRC1_POSZ", 1)
master.param_set_send("EK3_SRC1_VELXY", 0)
master.param_set_send("EK3_SRC1_VELZ", 0)
master.param_set_send("EK3_SRC1_YAW", 6)

time.sleep(1)

# REQUEST STREAM
master.mav.request_data_stream_send(
    master.target_system,
    master.target_component,
    mavutil.mavlink.MAV_DATA_STREAM_EXTRA1,
    5,
    1,
)

# INIT XY
base_x = 0.0
base_y = 0.0

print("Waiting EKF XY VALID...")

xy_valid = False
start = time.time()

# SEND VISION FOR EKF
while not xy_valid:
    timestamp = int(time.time() * 1e6)

    t = time.time()
    x = base_x + 0.05 * math.sin(t)
    y = base_y + 0.05 * math.cos(t)
    z = -0.5

    # yaw=0
    master.mav.vision_position_estimate_send(timestamp, x, y, z, 0.0, 0.0, 0.0)

    msg = master.recv_match(type="EKF_STATUS_REPORT", blocking=True, timeout=1.0)
    if msg:
        flags = msg.flags

        if flags & mavutil.mavlink.EKF_POS_HORIZ_REL:
            xy_valid = True
            print("XY position VALID achieved")

    if time.time() - start > 10:
        print("EKF failed to initialize XY")
        break

# Switch to Guided mode
mode_id = master.mode_mapping()["GUIDED"]
master.set_mode(mode_id)
print("ChangeMODE")

time.sleep(1)

# Arm
master.arducopter_arm()
master.motors_armed_wait()
print("Armed")

# initial position
init = master.recv_match(type="LOCAL_POSITION_NED", blocking=True)
base_x = init.x
base_y = init.y
print(f"Initial EKF: x={base_x}, y={base_y}")

target_x = base_x + 2.0
target_y = base_y
target_z = -0.5

# type_mask
type_mask = (
    mavutil.mavlink.POSITION_TARGET_TYPEMASK_VX_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_VY_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_VZ_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_AX_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_AY_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_AZ_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_YAW_IGNORE
    | mavutil.mavlink.POSITION_TARGET_TYPEMASK_YAW_RATE_IGNORE
)

try:
    while True:
        # Set position
        if xy_valid:
            master.mav.set_position_target_local_ned_send(
                0,
                master.target_system,
                master.target_component,
                mavutil.mavlink.MAV_FRAME_LOCAL_NED,
                type_mask,
                target_x,
                target_y,
                target_z,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
            )

        # Send vision position estimates
        timestamp = int(time.time() * 1e6)

        # TEST DATA
        t = time.time()
        x = base_x + 0.05 * math.sin(t)
        y = base_y + 0.05 * math.cos(t)
        z = -0.5  # NED
        roll = 0.0
        pitch = 0.0
        yaw = math.radians(30)

        master.mav.vision_position_estimate_send(timestamp, x, y, z, roll, pitch, yaw)

        # LOCAL_POSITION_NED
        msg = master.recv_match(type="LOCAL_POSITION_NED", blocking=True, timeout=0.1)

        if msg:
            print(f"[EKF] x={msg.x:.2f}, y={msg.y:.2f}, z={msg.z:.2f}")
        print(f"[SEND] x={x:.2f}, y={y:.2f}, z={z:.2f}")

        # EKF_STATUS_REPORT
        msg = master.recv_match(type="EKF_STATUS_REPORT", blocking=True, timeout=1.0)
        if msg is not None:
            flags = msg.flags
            print(f"[EKF_FLAGS] {flags}")
            if flags & mavutil.mavlink.EKF_POS_HORIZ_REL:
                print("  → XY position VALID")
            if flags & mavutil.mavlink.EKF_POS_VERT_ABS:
                print("  → Z VALID")
            if flags & mavutil.mavlink.EKF_ATTITUDE:
                print("  → Attitude OK")

        time.sleep(0.1)

except KeyboardInterrupt:
    print("Stopping...")

finally:
    master.close()
    print("Connection closed")

# %%

Thank you for your advice.
I was not aware of Cockpit, since I have never used it before.

Does QGroundControl provide a similar way to visualize the EKF status?
I will pay attention to it during my next hardware test.

First options that spring to mind are

  1. making sure VISO_TYPE is set to MAVLink (1)
  2. making sure the origin is set, and
  3. making sure the position estimate is a high enough frequency for the EKF to accept it (I believe ~10Hz is usually the minimum, though very possible I’m misremembering)
  4. it is maybe helpful to set the covariances, or at least explicitly specify that they’re unknown (by setting the first value as NaN)

In case it’s relevant, we’re most familiar with VISION_POSITION_DELTA messages being used as part of DVL drivers.

If there is one I’m not aware of it, and I can’t seem to find anything about it in the docs :man_shrugging:t3:

It’s been a while since I did anything with QGC.

+1 to all of @EliotBR 's suggestions. Two additional comments:

The VPE messages will not set the EKF origin, you’ll have to do that another way. See details here: Setting Home and/or EKF origin — Dev documentation

The EKF can take a while to “warm up”, 10 seconds might not be long enough. If you run SITL with the default parameters (“sim_vehicle.py …”) you will see the behavior when a simulated GPS is used. (In the simulation, the GPS works underwater, but you can ignore that.) See how long it takes to generate good xy positions in this canonical case.

Thank you very much for your kind and helpful response. We confirmed that it works in the simulator and conducted pool testing this week.

After fixing the issues you pointed out, I was able to confirm that VISION_POSITION_ESTIMATE is being properly applied and reflected in LOCAL_POSITION_NED. In particular, I verified that the position estimated by my SLAM implementation is being reflected in LOCAL_POSITION_NED in real time.

Before switching to GUIDED mode, I first enabled Depth Hold, and then started sending SET_POSITION_TARGET_LOCAL_NED messages at 10 Hz, specifying either position (X, Y, Z) or velocity.
After that, I switched to GUIDED mode, but it seemed that the target was not being applied. I also did not receive any POSITION_TARGET_LOCAL_NED messages. As a result, the vehicle entered a position-hold behavior. when I physically pulled the vehicle with a rope, it always returned automatically to the initial deployment position.

I also tried using LOCAL_OFFSET_NED and BODY_NED, but observed the same behavior. It seems that SET_POSITION_TARGET_LOCAL_NED is being ignored.

Do you have any idea what might be causing this?

I have tried the following two approaches, but I am unsure which one is correct or recommended:
Sending incremental position targets with small displacements (e.g., around 0.1 m from the current position)
Sending a target position further ahead, for example:
X = 5 m, Y = 1 m, Z = 3 m (i.e., (5, 1, 3))
Which approach should I use in this situation?

Thank you. I wasn’t aware of VISION_POSITION_DELTA. It does seem well-suited for sending relative motion.
By the way, how do you configure/set the target? Also, have you ever experienced cases where the target setting was rejected and the system fell back into a position-hold state?

Cockpit tries to discover a BlueOS vehicle on startup. Is it possible to use SITL by specifying the IP and port instead? I tested this with the latest version of Cockpit.

You are making good progress!

It seems that the targets are being rejected silently. If you haven’t already, you can trace the code to see how it rejects targets (this is 4.5.x, is that what you are using?): ardupilot/ArduSub/GCS_Mavlink.cpp at abe1721cf52535af6eb2340e5cabed430dac76b5 · ArduPilot/ardupilot · GitHub

Caveat: I have had good luck with the SET_POSITION_TARGET_GLOBAL_INT message with the RELATIVE_ALT frame (3). I have not tried SET_POSITION_TARGET_LOCAL_NED. It should work the same way.

There are some submodes to watch for. The simplest is WP (waypoint) submode, where you set the position target and GUIDED does the rest (via sub.mode_guided.guided_set_destination). You should not set targets at 10Hz. Instead, you should go to GUIDED mode, then send a single target some distance ahead. It should accelerate, move, then decelerate.

If you set both a target pos and a target vel you will get PosVel mode (via sub.mode_guided.guided_set_destination_posvel). Here you can (but don’t need to) send targets rapidly, and you can choose various distances. Note that this mode has a bug where the targets are not returned: Sub: POSITION_TARGET_GLOBAL_INT messages not sent in Guided_PosVel submode · Issue #33098 · ArduPilot/ardupilot · GitHub

I hope this helps!

Thank you very much for this valuable information. I plan to continue testing on the actual hardware while also running simulations.

I hope to be able to resolve this before the next water tank test in August.