Link 1 Down: Running ArduSub SITL with Custom Physics Sim Using JSON frame

[TMJ (Tim)]

Hi I’m trying to run the Ardusub SITL with a custom physics sim to allow modeling the bluerobotics with some custom sensors. My goal is to have an external python script sending vehicle commands via mavlink to the sitl while the ardusub SITL updates its sensors and PWMs with the simulation, just like one would with a companion computer talking to the onboard pixhawk:

python (companion comp) <-> mavlink <-> ardusub sitl (pixhawk) <-> custom sim physics (ocean env.)

As a test, if I launch the SITL with the vectored frame, everything works great. However when I launch with the “-f JSON” frame, I get the following output with the dreaded link 1 down message at the MAV> prompt.

Any help would much appreciated!

command: python ~/python_scripts/ardupilot/Tools/autotest/sim_vehicle.py -f JSON

SIM_VEHICLE: Start
SIM_VEHICLE: Killing tasks
SIM_VEHICLE: Starting up at SITL location
SIM_VEHICLE: WAF build
SIM_VEHICLE: Configure waf
SIM_VEHICLE: “/home/tim/python_scripts/ardupilot/modules/waf/waf-light” “configure” “–board” “sitl”
Setting top to : /home/tim/python_scripts/ardupilot
Setting out to : /home/tim/python_scripts/ardupilot/build
Autoconfiguration : enabled
Setting board to : sitl
Using toolchain : native
Checking for ‘g++’ (C++ compiler) : /usr/bin/g++
Checking for ‘gcc’ (C compiler) : /usr/bin/gcc
Checking for c flags ‘-MMD’ : yes
Checking for cxx flags ‘-MMD’ : yes
CXX Compiler : g++ 7.5.0
Checking for need to link with librt : not necessary
Checking for feenableexcept : yes
Checking for HAVE_CMATH_ISFINITE : yes
Checking for HAVE_CMATH_ISINF : yes
Checking for HAVE_CMATH_ISNAN : yes
Checking for NEED_CMATH_ISFINITE_STD_NAMESPACE : yes
Checking for NEED_CMATH_ISINF_STD_NAMESPACE : yes
Checking for NEED_CMATH_ISNAN_STD_NAMESPACE : yes
Checking for header endian.h : yes
Checking for header byteswap.h : yes
Checking for HAVE_MEMRCHR : yes
Configured VSCode Intellisense: : no
Checking for program ‘python’ : /home/tim/.virtualenvs/dl4cv/bin/python
Checking for python version >= 2.7.0 : 3.6.9
Checking for program ‘python’ : /home/tim/.virtualenvs/dl4cv/bin/python
Checking for python version >= 2.7.0 : 3.6.9
Source is git repository : yes
Update submodules : yes
Checking for program ‘git’ : /usr/bin/git
Checking for program ‘size’ : /usr/bin/size
Benchmarks : disabled
Unit tests : enabled
Scripting : enabled
Scripting runtime checks : enabled
Debug build : disabled
Coverage build : disabled
SITL 32-bit build : disabled
Checking for program ‘rsync’ : /usr/bin/rsync
‘configure’ finished successfully (4.251s)
SIM_VEHICLE: Building
SIM_VEHICLE: “/home/tim/python_scripts/ardupilot/modules/waf/waf-light” “build” “–target” “bin/ardusub”
Waf: Entering directory /home/tim/python_scripts/ardupilot/build/sitl' Embedding file locations.txt:Tools/autotest/locations.txt Embedding file models/Callisto.json:Tools/autotest/models/Callisto.json Waf: Leaving directory /home/tim/python_scripts/ardupilot/build/sitl’

BUILD SUMMARY
Build directory: /home/tim/python_scripts/ardupilot/build/sitl
Target Text Data BSS Total

bin/ardusub 2717456 136199 142104 2995759

Build commands will be stored in build/sitl/compile_commands.json
‘build’ finished successfully (6.175s)
SIM_VEHICLE: Using defaults from (/home/tim/python_scripts/ardupilot/Tools/autotest/default_params/sugar.params)
SIM_VEHICLE: Run ArduSub
SIM_VEHICLE: “/home/tim/python_scripts/ardupilot/Tools/autotest/run_in_terminal_window.sh” “ArduSub” “/home/tim/python_scripts/ardupilot/build/sitl/bin/ardusub” “-S” “-w” “–model” “JSON” “–speedup” “1” “–slave” “0” “–defaults” “/home/tim/python_scripts/ardupilot/Tools/autotest/default_params/sugar.params” “-I0”
SIM_VEHICLE: Run MavProxy
SIM_VEHICLE: “mavproxy.py” “–out” “127.0.0.1:14550” “–out” “127.0.0.1:14551” “–master” “tcp:127.0.0.1:5760” “–sitl” “127.0.0.1:5501”
RiTW: Starting ArduSub : /home/tim/python_scripts/ardupilot/build/sitl/bin/ardusub -S -w --model JSON --speedup 1 --slave 0 --defaults /home/tim/python_scripts/ardupilot/Tools/autotest/default_params/sugar.params -I0
WARNING: You should uninstall ModemManager as it conflicts with APM and Pixhawk
Connect tcp:127.0.0.1:5760 source_system=255
Log Directory:
Telemetry log: mav.tlog
Waiting for heartbeat from tcp:127.0.0.1:5760
MAV> link 1 down

[TMJ (Tim)]

I should mention that the SITL and sim appear to communicate correctly since the. sim receives PWM values and the SITL is receiving state updates via the JSON received message in the xterm window:

[EliotBR (Eliot)]

Hi @TMJ,

I unfortunately haven’t used SITL extensively, so this is beyond my experience. I’ve passed it on to the software team in the hopes they have some insight as to what might be going wrong and how to fix it

[williangalvani]

Hi @TMJ,

I suspect it is something in your sim code.How often are you sending the simulation data?

[williangalvani]

Hi @TMJ,

I suspect it is something in your sim code.How often are you sending the simulation data?

[TMJ (Tim)]

I’ve tried running the sim at 400 and 1200 frames per second.

[TMJ (Tim)]

But the heartbeat message, I thought, was supposed to be coming from the SITL and not my sim?

[williangalvani]

That’s right. My guess is that SITL is not running because it is waiting for something from the simulator.
Is there any chance you are closing the connection after sending the first packet or something like that?

We did work briefly on a simulator using the JSON interface, BlueSim. You can take a look at the json-talking code here (the code is not great, but it works ).

[TMJ (Tim)]

But the SITL does appear to be running because the sim receives pwm values and is able to send the simulated vehicle state (e.g. timestamp, acceleration, position, etc). You can see in the attached image that the ArduSub sitl receives the JSON file:

The problem is that MavProxy just sits and waits for the heartbeat from port 5670 but it never comes.

Waiting for heartbeat from tcp:127.0.0.1:5760
MAV> link 1 down

[williangalvani]

There are some debug functions in SIM_JSON.cpp. try enabling those, hopefully they can help you understand what is going on. gdb and just adding some printfs could help, too.

[TMJ (Tim)]

Ok figured it out.

The timestamp must be an increasing value; otherwise the SITL won’t send Mavproxy a heartbeat on socket tcp:127.0.0.1:5670.

I was sending a constant timestep by oversight.

Thanks for your help William.