I am attempting to log the output of the Ping1D get_profile() command for use in coastal surveying. When testing the Ping2 device in a fixed depth flat bottom pool, I found that the distance field of the get_profile() output was fairly accurate to the depth of the pool, but that the scan length and length of the profile array would fluctuate between scans, making it difficult to produce a pcolor plot of the backscatter data. Additionally, when dividing the scan length by the length of the profile array, I found that the bin widths were not a consistent value, fluctuating between 2.84mm and 3.35mm. Is there a way to fix the length of the profile array or the resolution of the data? When reviewing the ping protocol, it seemed we could at best indirectly control these values using the set_range() command.
Hi @MJS_FRF,
The ping1d protocol doesnât provide direct control over the profile array length or its resolution, but if you disable auto mode then the array length should be consistent after youâve set the range and gain (and the speed of sound, if youâre changing that)
To clarify, I donât actually know whether any of those variables affect the array length (my memory is that itâs always the same at the moment), but itâs auto mode that causes the range (and accordingly the resolution) to change without direct input.
Hi @EliotBR ,
I have disabled auto mode and set the gain and range parameters, however the profile arrays still vary in length. For a gain of 1, and a range of 0 to 2500mm, the profile array has lengths ranging from 690 to 780 bins, across 5 minutes of operation.
Can you share the code youâre using?
As I understand it our Ping Sonar firmware is hardcoded to send 200 bins in each profile
array, regardless of the configured range, which is why we specify the range resolution as â0.5% of rangeâ in the technical details on the product page.
We are using a python script which runs on system startup to log the data from the ping sonar. I have attached one of the CSVs.
20240627150822.csv (655.5 KB)
myPing = Ping1D()
myPing.connect_serial("/dev/ttyUSB0", 115200)
while myPing.initialize() is False:
print("Failed to initialize Ping! Retrying")
mode_verify = myPing.set_mode_auto(0, verify=True)
while not mode_verify:
mode_verify = myPing.set_mode_auto(0, verify=True)
print("Automatic mode disabled")
speed_of_sound_saltwater = 1485000
speed_of_sound_freshwater = 1475000
speed_verify = myPing.set_speed_of_sound(speed_of_sound_freshwater, verify=True)
while not speed_verify:
speed_verify = myPing.set_speed_of_sound(speed_of_sound_freshwater, verify=True)
print("Speed of sound updated correctly")
scan_range_m = 2.5
range_verify = myPing.set_range(0, 2500, verify=True)
while not range_verify:
range_verify = myPing.set_range(0, 2500, verify=True)
print("Scan range updated correctly")
gain_verify = myPing.set_gain_setting(1, verify=True)
while not gain_verify:
gain_verify = myPing.set_gain_setting(1, verify=True)
print("Gain setting updated correctly")
profile_bins = ','.join([f"Bin{i}" for i in range(803)])
verbose_header = "Timestamp,Depth (m),Confidence (%),Transmit Duration (us),Ping Number,Scan Start (mm),Scan Length (mm),Gain Setting,Number of Buckets," + profile_bins
...
Output File Creation Here
...
start_time = time.time()
curr_time = start_time
file_start = start_time
prev_ping_number = -1
record_time = start_time
while True:
record_time = time.time()
data = myPing.get_profile()
if data:
if data["ping_number"] == prev_ping_number:
data = None
else:
prev_ping_number = data["ping_number"]
if data:
datestr = str(datetime.datetime.utcnow().strftime('%H%M%S.%f')[:-3])
profile = str(data['profile_data'])
profile_bins = ''
bin_count = 0
for c in profile:
profile_bins += str(ord(c))+','
bin_count += 1
char_count = bin_count
# 803 was the length of the largest profile array observed
# pad shorter profile arrays to create a square dataset for writing to CSV
while char_count < 803:
profile_bins += str(-1)+','
char_count += 1
# write data to CSV
datastr = datestr + ',' + str(data["distance"]/1000) + ',' + str(data["confidence"]) + ',' + str(data["transmit_duration"]) + ',' + str(data["ping_number"]) + ',' + str(data["scan_start"]) + ',' + str(data["scan_length"]) + ',' + str(data["gain_setting"]) + ',' + str(bin_count) + ',' + profile_bins[:-1]
writer = csv.writer(file1)
writer.writerow(datastr.split(','))
...
Create new file if 60 seconds have ellapsed since previous file
....
curr_time = time.time()
file1.close()
This is the source of the problem - the code here is converting bytes (i.e. already numbers) into a human-readable string, and then looping through the characters in that string, converting them to their ordinal values, and using those as the data values, instead of just directly using the data values in the initial bytes object.
As an analogy if I had the integer 123
, and turned it into a string "123"
, and got the ordinal of each character, Iâd have 49, 50, 51
(i.e. a single number just got turned into three numbers that represent its digits). Itâs similar for bytes, but their string representation can include some non-numeric characters.
As a few suggestions,
- If you just want to see the data values within a Python context then you can convert the bytes object to a list, and print that:
print(list(data['profile_data']))
- To add the data to a CSV string (when youâre manually constructing a CSV file) youâll want to convert each number to a string, not the whole bytes container:
bin_count = len(data['profile_data']) assert bin_count == 200, f"Unexpected data length encountered ({bin_count} != 200) - adjust CSV creation code" profile_bins = ','.join(str(response_strength) for response_strength in data['profile_data'])
- When using a higher level CSV writer, you can just add all the data directly to the row list (it should convert the values to strings automatically, as relevant):
formatted_date = datetime.datetime.utcnow().strftime('%H%M%S.%f')[:-3] datarow = [ formatted_date, data['distance'] / 1000, # Convert mm to meters *(data[field] for field in ( 'confidence', 'transmit_duration', 'ping_number', 'scan_start', 'scan_length', 'gain_setting', ), len(data['profile_data']), # bin_count - should be 200 *data['profile_data'] # Unpack the data as separate values ] writer.writerow(datarow)
- If you want to be able to visualise/replay the data afterwards in Ping Viewer, itâs possible to record the received ping-protocol messages directly into a binary file with the appropriate format
- We have some relevant example code in the Ping Viewer repository, although itâs focused on working with existing log files - you could work from that, or I can write a âcreating a Ping Viewer log from live dataâ example if thatâs of interest
Thank you @EliotBR. I had suspected this may be the case, but I had seen another topic on the forum use my original solution without being corrected. One thing I want to note for the 3rd solution is that I am receiving a keyError for the âprofile_data_lengthâ field. I see it is listed as a field of the return value of the get_profile() function in the Ping1d docs, but printing the actual return dictionary returned by the function shows that no such field exists.
Unfortunately not every mistake gets noticed on the first pass.
I did happen to find this post while looking around, which seems to have used the same approach as your original one (so might be what youâre referring to), but the question being asked wasnât focused on the presented code or perceived errors in the data, so Iâm not too surprised that the code issue got missed at the time.
Fair enough that thatâs confusing though - incorrect references always make things more complicated than they need to be, and ideally weâd have an example of processing and doing something with the data included with the library, since thatâs a reasonably basic usage of it, but thatâs not something weâve made at this stage.
My bad - that looks to be missing in the library, so there might be an issue with the generator, or how itâs defined in the protocol. That said, it might also be intentional because in Python the profile data container has a queryable length (using the len
builtin).
Either way, Iâve updated the example snippet to use len
instead