300m tether: link associates but throughput collapsed from ~30 Mbps to ~0.3 Mbps overnight

Summary

My BlueROV2 (Navigator / BlueOS) worked normally yesterday, achieving around 30 Mbps over a 300 m Fathom tether over numerous succesful dives to 50 meters. I brought the ROV back to the bench at the end of the days and was able to connect via fathomX as per usual and downlaod log files and video. I came back to the ROV today which has not been plugged in, wetted, or significantly moved since yesterday, but now I canno connect to the ROV at all. I have been unable to open Blue OS with the normal set up but did manage to get it open briefly after taking the tether spool and extender out of the pathway and saw upload / download speeds in the range of .3 to 1.5 Mbps with long latency.

I have isolated the issue to the 300 meter tether. I am able to interface normally with Blue OS by plugging twisted pair from ROV fxti board directly into fathomX topside interface or by running jumper cables from ROV fxti twisted pair to contacts on my tether extension cable into the fathomX topside unit through the normal binder interface.

Now for the 300 m tether itself. I measure ~40 ohms resistance over each of the 8 wires in twisted pairs (or ~85 ohms when I short a twisted pair on the ROV side and measure both contacts on the opened topside end of the tether. That number seems fine to me. I tested up to the 20 megaohm range for spillover between twisted pairs (one mulitmeter probe on a blue topside contact and testing all other non-blue contacts for any leakage) but did not pick up any measurement indicating current leaking between twisted pairs. I have inspected the insides of both the ROV and top side of the tether an nothing stands out as wrong. All four twisted pairs behave identically.

I also tested the tether in isolation from the topside binder connector by running jumper cables direct from twisted pair contact points with interior side of binder connector to the fathomX topside unit and still ony achieved intermittent connections with ubiquitously slow latency (500 to 3500 ms).

I’ve spent a full day isolating this and would appreciate input on whether this is a recoverable tether fault, a board fault that only manifests over distance, or something I’ve overlooked.

Details Below:

System

  • BlueROV2, R4 / Navigator flight controller, BlueOS at 192.168.2.2
  • 300 m Fathom tether, Fathom spool, spool extension cable
  • FXTI topside, USB to Windows laptop, static 192.168.2.1 / 255.255.255.0
  • Added Ethernet switch inside the ROV enclosure sharing the tether link between the Raspberry Pi and a custom 4K IP camera at 192.168.2.160
  • Realtek USB FE Family Controller appears cleanly in Device Manager, no driver issues
  • ROV disarmed and flight controller powered off during all tests below

Initial symptom

FXTI Power and Link LEDs both lit. ROV completes its power-up sequence. No connection in Cockpit, BlueOS, or the camera’s VMS software. Tried multiple USB ports and a second laptop with the same result. Windows showed 144 packets sent, 0 received. ping 192.168.2.2 returned no replies.

Components confirmed working

Raspberry Pi, SD card, BlueOS, internal switch, 4K camera. Laptop connected by short Ethernet cable directly into the ROV’s internal switch. Both 192.168.2.2 and 192.168.2.160 load normally.

FXTI internal path — USB-to-Ethernet converter and internal wiring. Ethernet cable run from the ROV’s internal switch directly into the FXTI topside unit with the twisted pairs disconnected. Both IPs load normally.

Both Fathom-X boards and both HomePlug modems. Approximately 2 ft of twisted pair run directly from the ROV-side Fathom-X board to the topside Fathom-X, everything else in normal configuration. Full connectivity, measured at roughly 30 Mbps.

Spool extension cable and the FXTI Binder bulkhead. Jumper wires from the Fathom-X board output onto the internal contacts at the far end of the extender, with the extender plugged into the FXTI bulkhead in normal configuration. Runs of consistent sub-50 ms ping replies.

Tether electrical measurements

Loop resistance. Pair shorted at the ROV end, measured at the topside Binder connector: approximately 85 Ω. That works out to about 42 Ω per conductor, consistent with 300 m of 26 AWG. Continuous, no break, no high-resistance joint.

Insulation. No measurable resistance between the two conductors of the pair in use, and none between those conductors and any other twisted pair contact. No sign of moisture bridging or insulation breakdown.

Tether Extension Cable. Approximately 1 Ω per channel measured standalone, approximately 1.5 Ω measured through the mated bulkhead into the FXTI internal leads.

Tests that did not resolve it

  • Tried all four twisted pairs in the tether, changed at both the ROV and FXTI ends. All four fail the same way.
  • Unspooled the entire tether to remove the slip ring from the circuit. No change.
  • Removed the spool and extender entirely, tether straight into the FXTI. Best result of the day, but still only about 0.3 Mbps up and down against the usual 30 and not even able to produce this level of connectivity consitently.
  • Flexed and rotated connections throughout the chain while pinging. No response.
  • Unspooled the whole tether to look for any obvious damage but saw none.
  • Removed ROV side bulkhead penetrator and disassembled it to look for any nicked cables or corrosion but saw nothing wrong.

Ping traces

Best case achieved, tether direct into FXTI without spool or extender:

Ping statistics for 192.168.2.2:
    Packets: Sent = 50, Received = 44, Lost = 6 (12% loss),
Approximate round trip times in milli-seconds:
    Minimum = 13ms, Maximum = 3455ms, Average = 717ms

The first several replies came back at 2000–3455 ms, followed by a run of timeouts, then settling to a fairly stable 400–600 ms with occasional excursions and the odd 13 ms or 89 ms outlier.

Later, in the same configuration, the link went to a blinking Link LED and 20 of 20 pings timed out.

Attempting to bypass the tether’s Binder termination

To rule out the tether’s own connector, I opened up the Binder end of the 300 m tether and ran jumper wires from the topside Fathom-X directly onto the tether’s internal contacts.

With jumpers held as steadily as I could manage, through the 300 m tether alone: inconsistent replies between 500 and 3000 ms.

Running through both the main tether and extender in normal order, with jumpers bridging the open tether ends: fairly consistent replies around 200 ms.

So bypassing the connector did not restore performance. Both figures are still far off the single-digit milliseconds this link should deliver.

Where this leaves me

By elimination: everything on both ends works at full speed over short wire, the extender and FXTI bulkhead are clean, the spool and slip ring are ruled out, the tether measures electrically sound at DC, and all four pairs behave identically. What remains is the 300 m cable itself, or something in the topside unit that only manifests once the channel is already attenuated by that distance.

Questions

  1. Has anyone seen a Fathom tether lose high-frequency performance while retaining normal DC continuity and insulation resistance? Is that a known failure mode, and is it diagnosable or repairable, or does it mean replacement?
  2. Is there a plausible mechanism by which a Fathom-X board or the FXTI performs perfectly over 2 ft and 30 Mbps but fails over 300 m, while the cable itself is sound?
  3. What do the Fathom-X Link LED states actually indicate? Specifically, does a solid Link LED confirm association with the peer board, and does the roughly 5-second flash on power-up have a defined meaning? This ambiguity cost me a lot of troubleshooting time.
  4. Is there anything else worth measuring before I commit to replacing a 300 m tether?

Happy to run further tests — I have the vehicle completely discombobulated on the bench and the tether unspooled.

Hi @koehler,

I’m short on time, but a few quick ideas:

  1. Can you try installing the Tether Diagnostics Extension, and seeing what kind of results you’re getting for each pair?
  2. Does your bench location have substantially different humidity to where you normally keep your tether?
    • @tony-white recently found that a tether with unsealed ends can absorb/lose humidity and have resulting changes in its communication behaviour, though that was more problematic for a different communication protocol we’re testing, and we’re still investigating the phenomenon

Hi @koehler

This caught my eye - a short twisted pair like that should be achieving much much higher speeds, 50-70Mbps?

In answer to your questions-

  1. I’ve not seen a tether damaged so that it passes continuity/insulation tests, but fails to connect. Generally those situations are a faulty Fathom-X LX200V20 failure - they are sensitive to salt water exposure, as most electronics are!
  2. @EliotBR ‘s reco of the tether diagnostic extension will shed light on the true behavior of the 2’ vs. 300m tether performance.
  3. The Link light indicates the two boards have successfully linked, and are transparent to ethernet traffic. Both lights come on on power up, the key is the link light must stay on and never go off. It’s not particularly ambiguous - if the link light goes off, no link present, otherwise if itis on and steady you should have a link.
  4. Definitely share screenshots of the tether diagnostic extension running with the 2’ and 300m tethers. You may want to use the BlueOS interface from the IP address it has on your WiFi network, so you can still see the extension interface even as tethers are disconnected and swapped.
    The heating solution Eliot mentioned was for g.hn based communications over slim tether, but if your tether has been damaged, like the outer jacket being cut, water in the tether can definitely negatively impact performance…
    Best of luck!

I have had similar with slightly damaged shell of the tether.
Water ingress gave prefect resistance measure, but communication drop to less then 1 Mbit.
Try to let the tether dry for some days, in my case that gave communcation slowly going back,
After two weeks drying full speed, but next dive, after 4 hours back to 1 Mbit.
So, try to let dry to isolate the problem.
Visually check damages might work.
You could also do pulse reflection check, in wet condition, with transmitter and oscilloscope to fin out where the leak is, then cut it.