Writing GPS Timestamps in Pixhawk Log File

Blue Robotics Software
1

Hi Everyone,
we successfully forwarded NMEA Data to the Pixhawk as described here: GPS Positioning · GitBook

The BlueROV is shown on the QGC map and the GPS data (longitude, latitude) are stored in the Pixhawk flash logs but the GPS timestamps (UTC time) are missing! .

Is there any way how to retrieve record these timestamps?
I checked all the parameter settings such as
LOG_BITMASK → (GPS Checked)
BRD_RTC_TYPES: → (GPS Checked)

Also, here Data Logging · GitBook it was mentioned that
“If GPS is available, the logs will be dated according to the GPS timestamp”
I checked that but nothing changed. The timestamps of the regular variables such as ACC1, RCIN, ATT,… are still in unix time.

Best Regards
Chris

2

Hi @cbusse1

That is a bug. I just checked and opened an issue.

If all you need is to have the timestamp in the log names (instead of 000001.bin, you will have something like 2020-01-23 12-56-11.bin), try setting BRD_RTC_SRC to 3.
This will allow ArduSub to use the timesync normally message sent by QGC to set it’s internal time.

1 Like
3

Hi Willian,

Thank you for your quick reply.

The link to nmea-receiver.py is very useful. If I want to change the python script do I just need to replace it on the companion computer? Is the script executed automatically on reboot of the raspberry or is it triggered when data is received on the UDP port?

I will also test setting BRD_RTC_SRC to 3. This might be a workaround for now. If I understood correctly, should the log names then contain the timestamp of file creation dated with surface pc time?

I forgot to mention that I upgraded from ArduSub 3.5.4 to 4.0.2 when testing the GPS streamer because I found that BRD_RTC_SRC parameter was not available in the old firmware.

Besides that, the behavior of log names seemed sort of random to me because previously when GPS was not involved they sometimes contained timestamps and sometimes not. With the new firmware, I also found that when in QGC I press “Erase all log files” on the flash, the Pixhawk reboots which can be clearly noted by the connection loss in QGC and the loud beeping of the ESCs (Note: I have chosen settings so, that logging should start immediately on Pixhawk boot even when the BlueROV is not armed and should stop when the BlueROV/Pixhawk is powered off.)
Seems to me like the Pixhawk does not like that and sometimes it creates a bunch of 16kb-sized log files before continue to write into a single log file. If I remember correctly there was no rebooting when erasing all log files on runtime with old firmware 3.5.4 (but I need to check that again).

4

I tried fixing (unsuccessfully yet) by replacing with a modified nmea-receiver.py on the companion computer.

However, now when replacing with the original nmea-receiver.py The service does not start automatically anymore when GPS data is sent to the companion computer. If I execute the script manually via the shell via companion webinterface it’s working normally.

I did "sudo chmod +x nmea-receiver.py" but it seems that something else needs to be done, so that the script starts automatically again?

5

Hi!

Try running it manually at first, so you catch any possible issues making it exit early.

try python /home/pi/companion/tools/nmea-receiver.py in the terminal and check if there are any error messages.

6

Hi,
yes, we tested it manually via terminal.
So there are no errors and it is actually running normally just as the original but the additional code to send the timestamps had no effect:

#!/usr/bin/python

import time

import pynmea2

import json

import socket

from os import system

# destination / output

ip="127.0.0.1"

portnum = 25100

sockitOut = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

sockitOut.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

sockitOut.setblocking(False)

# UDP source

sockitUdp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

sockitUdp.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

sockitUdp.setblocking(False)

sockitUdp.bind(('0.0.0.0', 27000))

# Alternative TCP source

sockitTcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sockitTcp.setblocking(False)

sockitTcp.bind(('0.0.0.0', 27001))

sockitTcp.listen(1)

parser = pynmea2.NMEAStreamReader()

data = {

    'time_usec' : 0,                        # (uint64_t) Timestamp (micros since boot or Unix epoch)

    'gps_id' : 0,                           # (uint8_t) ID of the GPS for multiple GPS inputs

    'ignore_flags' : 56,                    # (uint16_t) Flags indicating which fields to ignore (see GPS_INPUT_IGNORE_FLAGS enum). All other fields must be provided.

    'time_week_ms' : 0,                     # (uint32_t) GPS time (milliseconds from start of GPS week)

    'time_week' : 0,                        # (uint16_t) GPS week number

    'fix_type' : 3,                         # (uint8_t) 0-1: no fix, 2: 2D fix, 3: 3D fix. 4: 3D with DGPS. 5: 3D with RTK

    'lat' : 0,                              # (int32_t) Latitude (WGS84), in degrees * 1E7

    'lon' : 0,                              # (int32_t) Longitude (WGS84), in degrees * 1E7

    'alt' : 0,                              # (float) Altitude (AMSL, not WGS84), in m (positive for up)

    'hdop' : 0,                             # (float) GPS HDOP horizontal dilution of position in m

    'vdop' : 0,                             # (float) GPS VDOP vertical dilution of position in m

    'vn' : 0,                               # (float) GPS velocity in m/s in NORTH direction in earth-fixed NED frame

    've' : 0,                               # (float) GPS velocity in m/s in EAST direction in earth-fixed NED frame

    'vd' : 0,                               # (float) GPS velocity in m/s in DOWN direction in earth-fixed NED frame

    'speed_accuracy' : 0,                   # (float) GPS speed accuracy in m/s

    'horiz_accuracy' : 0,                   # (float) GPS horizontal accuracy in m

    'vert_accuracy' : 0,                    # (float) GPS vertical accuracy in m

    'satellites_visible' : 0                # (uint8_t) Number of satellites visible.

}

data_received = False

gps_type_set = False

last_output_t = 0;

import datetime

def gpsTime2WeekMs(doy, tod):

    gpsTime = datetime.datetime.combine(doy, tod)

    utc =  int(gpsTime.timestamp()*1000000)

    # gpsTime = datetime.datetime.strptime()

    datetimeformat = "%Y-%m-%d %H:%M:%S"

    epoch = datetime.datetime.strptime("1980-01-06 00:00:00", datetimeformat)

    timeDifference = gpsTime - epoch 

    gpsWeek = timeDifference.days // 7

    gpsMilliseconds = int((timeDifference.seconds + 86400* (timeDifference.days - 7*gpsWeek)) * 1000 + (timeDifference.microseconds / 1000))

    return utc, gpsWeek, gpsMilliseconds

# wait for connection

def waitConnection():

    while True:

        try:

            sockit, addr = sockitTcp.accept()

            print("TCP connected!")

            return sockit

        except:

            print("TCP not connected, waiting for data")

        try:

            sockitUdp.recvfrom(4096)

            sockit = sockitUdp

            print("UDP connected!")

            return sockitUdp

        except:

            print("UDP not connected, waiting for data")

        time.sleep(1) # 1 Hz update before connected

sockit = waitConnection()

# setup gps type parameter

system('screen -S mavproxy -p 0 -X stuff "param set GPS_TYPE 14^M"')

while sockit:

    time.sleep(0.05) # 20 Hz update once connected

    try:

        datagram = sockit.recv(4096) # This blocks for UDP, for TCP, it will return None when remote hangs up

        if not datagram: # TCP disconnect

            print("Remote has hung up")

            sockit.shutdown(socket.SHUT_RDWR)

            sockit.close()

            sockit = waitConnection()

        else:

            for byte in datagram:

                for msg in parser.next(byte):

                    if msg.sentence_type == 'GGA':

                        data['lat'] = msg.latitude * 1e7

                        data['lon'] = msg.longitude * 1e7

                        data['hdop'] = float(msg.horizontal_dil)

                        data['alt'] = float(msg.altitude)

                        data['satellites_visible'] = int(msg.num_sats)

                    elif msg.sentence_type == 'RMC':

                        

                        # utc, week, ms = gpsTime2WeekMs(msg.datestamp, msg.timestamp)

                        # data['time_usec'] = utc

                        # data['time_week'] = week

                        # data['time_week_ms'] = ms

                        data['time_usec'] = 1611668007

                    

                        data['lat'] = msg.latitude * 1e7

                        data['lon'] = msg.longitude * 1e7

                    elif msg.sentence_type == 'GLL':

                        data['lat'] = msg.latitude * 1e7

                        data['lon'] = msg.longitude * 1e7

                    elif msg.sentence_type == 'GNS':

                        data['lat'] = msg.latitude * 1e7

                        data['lon'] = msg.longitude * 1e7

                        data['satellites_visible'] = int(msg.num_sats)

                        data['hdop'] = float(msg.hdop)

            if time.time() > last_output_t + 0.1:

                last_output_t = time.time();

                buf = json.dumps(data)

                print("Sending: ", data)

                sockitOut.sendto(buf, (ip, portnum))

    except socket.error as e:

        if e.errno == 11:

            pass

        else:

            print("Error:", e)

    except Exception as e:

        print("Got error:", e)

My last question was about how to restore the original nmea-receiver.py without having to reflash the complete companion computer image.

7

Hi

This looks right. Why did you comment the block with time_week?
with it uncommented, it should be possible to see it in the gps messages of the MAVLink Inspector in QGroundControl.

Two possible ways:

This will store all your changes in the “companion” folder in git and revert to the latest commit (you could get the changes back with git stash pop:

cd companion
git stash

This will revert the file to the latest commit (but you lose your changes permanently)

cd companion
git checkout -- tools/nmea-receiver.py
8

Restoring the script with git worked, thank you!

The GPS Mavlink Messages are already visible but without correct timestamps.

We tested the commented block but it did not make a difference as well.
We were expecting that the timeUS variable would be overwritten but it’s not, even when writing a constant value to it.

9

Hi,

This is where this data is received by mavproxy:

And this is where MAVProxy’s MAVLink message gets consumed by ArduSub:

It looks like time_us is not being used at all, which is likely why it doesn’t change.

I can’t dig any deeper now, but hopefully this helps.

1 Like