Ping360 and Altimeter freezing

Hi Everyone !

We had a test yesterday. And we tested the BR2 Heavy configuration with ping360 and altimeter.

we armed the ROV and operated our ROV After 10 min first ping360’s pingviewer screen froze, then the altimeter’s screen froze too.

When we took our ROV to the ground and we cut the power then we reconnected ping360’s and altimeter’s worked seamless.

We have a big test in 2 days. How can we solve that problem?

Thanks!

Hi @ugurdemirezen,

I’m not sure what’s happened here, although the potential causes I can think of are:

  • your Ping sensors got under-powered and shut down / stopped communicating
    • May need to upgrade your power circuitry, and/or reduce power usage
  • some kind of issue occurred with the serial to UDP software used for bridging the serial Ping device connections to the topside computer
    • I’m not aware of something that would be likely to do this, and it likely depends what software you were running on the onboard computer (Raspberry Pi)
  • some issue on your topside computer stopped the communication from working, or ruined the display updates

If the issue only happened once and isn’t reproducible, there likely isn’t much we can do to help. You can send through a GUI log and sensor log if you’ve got them, but we likely won’t be able to find and fix an issue like this on our end.

These are our GUI_LOG and sensor log files.

20220614-134142226.bin (14.0 MB)
20220614-134059708.txt (964.5 KB)

I tested them today. They worked fine for 40 min. But after that ping360’s connection was down.

And I tried to connect again but pingviewer couldn’t find the ping360.

I can send you a video of freezing via email.

Thanks

Hi @ugurdemirezen, sorry for the delay on getting back to this.

I tried replaying your Ping360 sensor log file and running some analysis on it - it seems to have occasional delays of ~300ms, but it’s not clear whether that’s caused by a general communications issue or something to do with Ping Viewer or the sensor. It’s also only ~6 minutes long, which doesn’t seem to match the description in your initial post about freezing after 10 minutes.

The provided GUI log is for a Ping Sonar recording, and from a brief look through it there don’t seem to be particular warnings or errors that would be of concern.

A video of the freezing could be helpful, especially if it’s actually freezing for longer than the ~200-300ms spikes shown on my plot. If you’d prefer not to post it directly on the forum you can send it via a private message.

I’ve also asked the software team whether they have some idea as to what might be going wrong, and will get back to you if they have particular suggestions.

Log Message Times Analysis Code

Makes use of decode_sensor_binary_log.py.
Requires pandas, numpy, matplotlib, and bluerobotics-ping libraries to be installed, with recent versions.

from decode_sensor_binary_log import PingViewerLogReader
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

log_path = '/path/to/file.bin' # TODO
log = PingViewerLogReader(log_path)
timestamps = pd.to_timedelta(timestamp.replace('\x00', '') for timestamp, _ in log)
differences = np.diff(timestamps)

plt.plot(timestamps[1:], differences)
plt.xlabel('timestamp [ns]')
plt.ylabel('time since last message [ns]')
plt.show()

Is there any process with that problem?

Do I have to do something? Like updating software?

Thanks!

Hi @ugurdemirezen,

I’ve asked about this internally, but haven’t yet received a response. A number of us have been travelling recently, so support questions have been harder to get to. I’ll raise it again and see if there are any ideas.

1 Like

Hi @ugurdemirezen,

As @EliotBR said, the logs are 6~7min long, are these logs from your freezing issue ?
Looking at it, there is not much information besides everything working as expected.

Some questions to help identify your problem:

  • Are you running Companion or BlueOS ?
  • If you access Companion or BlueOS and click on restart, does it fixes the issue ?
  • Does help if you reopen Ping-Viewer ?
  • Are you using Ping360 via the serial port ?
    • Can you check if there is any serial port connected ? You can check the system page on Companion, under detected devices. Or in the System Information page → Kernel of BlueOS, the last lines will show if you had a hardware disconnection.

Hi @patrickelectric ,

  • We are running Companion.
  • Yes, if I click on restart it fixes the issue sometimes.
  • It does not help reopen ping-viewer.
  • Yes we are using the serial port. I will test the ROV again and I will post the results for that question.

Hi Everyone,

We are using a ping360 and an altimeter in our ROV.

Sensors work fine for a while. But after a while, the connection starts to break and eventually the connection cannot be re-established.

This problem is solved by rebooting. But 10 minutes later the same thing happens again.

I have attached all the files.

We are using the last version of ardusub with pixhawk. Our pingviewer version is 9f1cbec i guess.

20221011-163233514.txt (10.5 MB)
20221011-163242900.bin (8.0 MB)
20221013-131950121.txt (4.3 MB)
20221013-120111391.txt (3.4 MB)
20221013-131956995.bin (270 Bytes)

Hi @ugurdemirezen,

Can you try the latest BlueOS 1.1.0-beta release, and confirm whether the connection issues are still present, and if so whether your Ping devices are still displayed on the Vehicle/Pings page in the web interface? That should help to narrow down whether it’s an issue in the vehicle (e.g. with the devices themselves, or BlueOS), or if it’s an issue with Ping Viewer.

If they are disappearing from the web interface, please upload the latest service log to the forum, as described here :slight_smile:

It would also be helpful to know how the devices are connected (e.g. are they both using USB to the onboard computer, or is the Ping360 using the Ethernet configuration?).

EDIT: I’ve moved your post to this previous one, because it’s seemingly on the same topic.

I experienced the same behaviour yesterday: the Ping360 produced scanning data for around 20 minutes, at which point ping-viewer stopped updating.

Environment:

BlueOS 1.0.1
ping-viewer 2.3.4 (commit 36168c2)
Model: 67
Firmware: 3.1
ID: 2
Device: /dev/ttyUSB0

The ping-viewer GUI log around the time of the failure began intermittently outputting:

[15:02:38:790] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.

It then reached a point where that was the only message being output (log truncated for brevity):

[15:02:38:790] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:38:804] ping.protocol.pingsensor[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:38:804] ping.protocol.ping360[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:39:890] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:39:893] ping.protocol.pingsensor[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:39:893] ping.protocol.ping360[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:39:908] ping.protocol.pingsensor[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:39:908] ping.protocol.ping360[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:40:911] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:40:924] ping.protocol.pingsensor[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:40:924] ping.protocol.ping360[Debug]: "Handling Message: DEVICE_DATA [2300]"
[15:02:41:760] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:42:663] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:43:651] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:45:619] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:46:121] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:46:739] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:47:490] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:47:860] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:48:628] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.
[15:02:49:865] ping.protocol.ping360[Warning]: Profile message timeout, new request will be done.

I can provide the .bin file as necessary (it’s ~44MB). Replaying the data with the debug option set showed that the RX error count began to increase until no more data was received.

Due to time constraints, I was unable to perform any debugging, and I could not find a way to restart the ping services. Upon restarting ping-viewer, no devices were listed.

I have updated to BlueOS 1.1.0-beta.10. The Ping360 is displayed in the Vehicle/Pings page. I will report back if the issue occurs again the next time I test.

Hi @oneironautics, welcome to the forum :slight_smile:

Thanks for the details - I’ve passed them onto the team.

Given the GUI log it seems like Ping Viewer was waiting for responses that weren’t arriving, so it’s unlikely to be an issue with Ping Viewer. It’s unclear whether there’s

  • a network issue stopping communication in one or both directions with BlueOS (although that should be noticeable with other communications also failing, like the telemetry and/or video stream)
  • an issue with the bridge in BlueOS failing to correctly transfer information between Ping Viewer and the UDP connection to the topside IP
  • an issue with the Ping360, where it for some reason stops responding (e.g. perhaps it crashes due to overheating, underpowering, or a firmware bug or something)