Interpreting profile data in ping-python

I am attempting to log the output of the Ping1D get_profile() command for use in coastal surveying. When testing the Ping2 device in a fixed depth flat bottom pool, I found that the distance field of the get_profile() output was fairly accurate to the depth of the pool, but that the scan length and length of the profile array would fluctuate between scans, making it difficult to produce a pcolor plot of the backscatter data. Additionally, when dividing the scan length by the length of the profile array, I found that the bin widths were not a consistent value, fluctuating between 2.84mm and 3.35mm. Is there a way to fix the length of the profile array or the resolution of the data? When reviewing the ping protocol, it seemed we could at best indirectly control these values using the set_range() command.

Hi @MJS_FRF,

The ping1d protocol doesn’t provide direct control over the profile array length or its resolution, but if you disable auto mode then the array length should be consistent after you’ve set the range and gain (and the speed of sound, if you’re changing that) :slight_smile:

To clarify, I don’t actually know whether any of those variables affect the array length (my memory is that it’s always the same at the moment), but it’s auto mode that causes the range (and accordingly the resolution) to change without direct input.

Hi @EliotBR ,
I have disabled auto mode and set the gain and range parameters, however the profile arrays still vary in length. For a gain of 1, and a range of 0 to 2500mm, the profile array has lengths ranging from 690 to 780 bins, across 5 minutes of operation.

Can you share the code you’re using?

As I understand it our Ping Sonar firmware is hardcoded to send 200 bins in each profile array, regardless of the configured range, which is why we specify the range resolution as “0.5% of range” in the technical details on the product page.

We are using a python script which runs on system startup to log the data from the ping sonar. I have attached one of the CSVs.
20240627150822.csv (655.5 KB)

myPing = Ping1D()
myPing.connect_serial("/dev/ttyUSB0", 115200)

while myPing.initialize() is False:
    print("Failed to initialize Ping! Retrying")

mode_verify = myPing.set_mode_auto(0, verify=True)
while not mode_verify:
    mode_verify = myPing.set_mode_auto(0, verify=True)

print("Automatic mode disabled")

speed_of_sound_saltwater = 1485000
speed_of_sound_freshwater = 1475000
speed_verify = myPing.set_speed_of_sound(speed_of_sound_freshwater, verify=True)
while not speed_verify:
    speed_verify = myPing.set_speed_of_sound(speed_of_sound_freshwater, verify=True)

print("Speed of sound updated correctly")

scan_range_m = 2.5
range_verify = myPing.set_range(0, 2500, verify=True)
while not range_verify:
    range_verify = myPing.set_range(0, 2500, verify=True)
    
print("Scan range updated correctly")

gain_verify = myPing.set_gain_setting(1, verify=True)
while not gain_verify:
    gain_verify = myPing.set_gain_setting(1, verify=True)
    
print("Gain setting updated correctly")

profile_bins = ','.join([f"Bin{i}" for i in range(803)])
    
verbose_header = "Timestamp,Depth (m),Confidence (%),Transmit Duration (us),Ping Number,Scan Start (mm),Scan Length (mm),Gain Setting,Number of Buckets," + profile_bins
    
...
Output File Creation Here
...

start_time = time.time()
curr_time = start_time
file_start = start_time

prev_ping_number = -1
record_time = start_time
while True:
    record_time = time.time()
    data = myPing.get_profile()
    if data:
        if data["ping_number"] == prev_ping_number:
            data = None
        else:
            prev_ping_number = data["ping_number"]
    if data:
        datestr = str(datetime.datetime.utcnow().strftime('%H%M%S.%f')[:-3])
        profile = str(data['profile_data'])
        profile_bins = ''
        bin_count = 0
        for c in profile:
            profile_bins += str(ord(c))+','
            bin_count += 1
        char_count = bin_count 
       # 803 was the length of the largest profile array observed
       # pad shorter profile arrays to create a square dataset for writing to CSV
        while char_count < 803:
            profile_bins += str(-1)+','
            char_count += 1
        
        # write data to CSV
        datastr = datestr + ',' + str(data["distance"]/1000) + ',' + str(data["confidence"]) + ',' + str(data["transmit_duration"]) + ',' + str(data["ping_number"]) + ',' + str(data["scan_start"]) + ',' + str(data["scan_length"]) + ',' + str(data["gain_setting"]) + ',' + str(bin_count) + ',' + profile_bins[:-1]
        writer = csv.writer(file1)
        writer.writerow(datastr.split(','))

        ...
        Create new file if 60 seconds have ellapsed since previous file
        ....

    curr_time = time.time()

file1.close()

This is the source of the problem - the code here is converting bytes (i.e. already numbers) into a human-readable string, and then looping through the characters in that string, converting them to their ordinal values, and using those as the data values, instead of just directly using the data values in the initial bytes object.

As an analogy if I had the integer 123, and turned it into a string "123", and got the ordinal of each character, I’d have 49, 50, 51 (i.e. a single number just got turned into three numbers that represent its digits). It’s similar for bytes, but their string representation can include some non-numeric characters.

As a few suggestions,

  1. If you just want to see the data values within a Python context then you can convert the bytes object to a list, and print that:
    print(list(data['profile_data']))
    
  2. To add the data to a CSV string (when you’re manually constructing a CSV file) you’ll want to convert each number to a string, not the whole bytes container:
    bin_count = len(data['profile_data'])
    assert bin_count == 200, f"Unexpected data length encountered ({bin_count} != 200) - adjust CSV creation code"
    profile_bins = ','.join(str(response_strength) for response_strength in data['profile_data'])
    
  3. When using a higher level CSV writer, you can just add all the data directly to the row list (it should convert the values to strings automatically, as relevant):
    formatted_date = datetime.datetime.utcnow().strftime('%H%M%S.%f')[:-3]
    datarow = [
        formatted_date,
        data['distance'] / 1000,  # Convert mm to meters
        *(data[field] for field in (
            'confidence',
            'transmit_duration',
            'ping_number',
            'scan_start',
            'scan_length',
            'gain_setting',
        ),
        len(data['profile_data']),  # bin_count - should be 200
        *data['profile_data']  # Unpack the data as separate values
    ]
    writer.writerow(datarow)
    
  4. If you want to be able to visualise/replay the data afterwards in Ping Viewer, it’s possible to record the received ping-protocol messages directly into a binary file with the appropriate format
    • We have some relevant example code in the Ping Viewer repository, although it’s focused on working with existing log files - you could work from that, or I can write a “creating a Ping Viewer log from live data” example if that’s of interest
1 Like

Thank you @EliotBR. I had suspected this may be the case, but I had seen another topic on the forum use my original solution without being corrected. One thing I want to note for the 3rd solution is that I am receiving a keyError for the ‘profile_data_length’ field. I see it is listed as a field of the return value of the get_profile() function in the Ping1d docs, but printing the actual return dictionary returned by the function shows that no such field exists.

Unfortunately not every mistake gets noticed on the first pass.

I did happen to find this post while looking around, which seems to have used the same approach as your original one (so might be what you’re referring to), but the question being asked wasn’t focused on the presented code or perceived errors in the data, so I’m not too surprised that the code issue got missed at the time.

Fair enough that that’s confusing though - incorrect references always make things more complicated than they need to be, and ideally we’d have an example of processing and doing something with the data included with the library, since that’s a reasonably basic usage of it, but that’s not something we’ve made at this stage.

My bad - that looks to be missing in the library, so there might be an issue with the generator, or how it’s defined in the protocol. That said, it might also be intentional because in Python the profile data container has a queryable length (using the len builtin).

Either way, I’ve updated the example snippet to use len instead :slight_smile: