Multiple cable penetrator leaks. Design flaw. (video)

My BlueROV2 has been plagued by leaks since I purchased it. We’ve had to abort several dives and use our backup ROV on multiple jobs due to severe leaks as soon as the BlueROV gets to depth.

The leaks have been extremely difficult to track down. They often don’t appear during vacuum tests or shallow test dives. Sometimes the ROV is fine for the first dive of the day, but the electronics enclosure would rapidly flood on the second dive after a battery swap.

We’ve finally traced the problem to the red cable penetrators. The potting epoxy is not bonding reliably with the anodized aluminum of the penetrator body. As a result, leaks can rapidly appear between the potting and metal if the cable is strained lightly.

We have had three penetrators fail in this fashion on our BlueROV2 Heavy in the last year. I’ve seen several photos of other user’s BlueROV2s with globs of epoxy added on the outside of the penetrators, presumably to address the same problem.

Bonding epoxy to aluminium has numerous documented challenges. These issues can be overcome by preparing the surface of the aluminum prior to bonding, but it doesn’t appear that Blue Robotics is doing any such preparations before potting.

When we make penetrators in house, we machine several grooves into the inside of the bore using a small metal lathe. We do this immediately prior to pouring the potting, and so far we haven’t had any failures.

It cost us dozens of hours and a couple of corroded speed controllers before we found this problem. I hope that Blue will make some improvements to your process to correct this issue.

In the meantime, do you have any recommendations for how users can repair leaking penetrators already in the field?

3 Likes

Hi Travis,

Thanks for sharing all these details and for identifying ways to make the penetrators better! This is something we’ve been struggling with for a while now and we’re putting a lot of effort into improving it. What you’re sharing is obviously concerning and this is really important to everyone on our forum so I’m going to make this long and go into some detail on what we’re doing.

We’ve been making penetrators (mostly) the same way for several years now and we’ve had leaks from time to time, but it’s definitely become more prevalent in the last year or so and as our overall volume has increased. When we started seeing more cases last year we made a number of changes in our production to try to improve the situation - we started storing epoxy cartridges differently, adjusted the potting method, changed cable cleaning and preparation steps, and tried surface preparation and anodizing options. We may have made some minor improvements but still kept seeing issues.

It’s really hard to confidently validate what works and what doesn’t work here! It’s a statistical problem generally - say the failure rate is 1% overall and we want to change something to improve that. Since the failure rate is low we need to make tens or hundreds of samples and test them all exhaustively to confidently identify whether the changes make things better or worse! We didn’t have the capacity to do that initially, so most of the changes we made were implementing epoxy best practices that we could be fairly confident with. That didn’t make a big enough impact.

Since the end of 2019, we’ve realized that we need to solve this problem conclusively. We’d like to solve it as quickly as possible, but we’re focused on solving it properly with a long-term solution so that our products can be used with great confidence. We’ve redirected nearly all of our engineering resources to doing that since January. We’ve tested a lot, learned a lot, and made some good progress towards fixing this. It takes a lot of time to make progress on this due to the testing requirements.

From January through March we tested hundreds of penetrator samples in relatively controlled experiments varying things like surface preparation and epoxy cure cycle. These were tested through pressure cycles in our chamber, different amounts of thermal cycling, and physical cable strain. Initially we were having an extremely hard time getting anything to fail at all, even out of large sample sets, but we eventually identified that the epoxy loses a lot of its mechanical strength and adhesion at elevated temperatures. That could happen during shipping, while sitting out in the sun, or even while operating in warm waters (like 80F). Our current data points to this being the primary failure point and we have seen a higher failure rate in warm environments compared to cooler environments.

We have identified a few changes to the process that seem to make a strong positive improvement on this situation. We were about to start a series of tests to validate that before we make those changes in our production, but that was cut short by the COVID-19 crisis and our engineering team is unable to complete that testing until the “Safer from Home” orders end in California.

In the meantime, from home, we’ve still been focused primarily on this issue and we are investigating further options and design changes to fix this issue. We’ve made some promising progress there but we won’t be able to test that much until we’re back at the office, sadly! We hope to have a solution soon after that!

In the meantime, please let us know if you’re having issues and we can make the best recommendation and help you stay in the water. Our support team will be happy to help you with that. We’ve been doing the best to take care of any customers affected by this issue.

  • For now I would recommend avoid high temperature exposure when possible and, in particular, avoid putting strain on the cables while they are at high temperatures from sitting in the sun.
  • I would also recommend removing the vent plugs when transporting to avoid a combination of pressure build-up and temperature increase that could push outwards on the potting and trigger issues.

I do want to take a moment to address your solution. I don’t know conclusively if it will work better or worse, but I can at least share some things we’ve learned that are relevant to it!

  1. There’s a good chance that the bare machined aluminum will have better adhesion to the epoxy. You might be able to improve that further with the use of a primer like Reltek A-3 or Forsch Primer.
  2. I’d be cautious with the grooves inside, they do provide mechanical restraint, but they also create stress concentrations that could trigger delamination of the epoxy. We have seen some correlation between sharp corners and delamination in our testing. (That applies to the sharp corner at the top as well).

We’ve had some recommendations from other customers that you could try to repair these in the meantime. We haven’t tried these ourselves so they are unvalidated at the moment:

  • You can backfill the other end of the penetrator to make sure it is full of potting compound
  • If the cable has come loose entirely, you can reinstall the penetrator with machining (like you’ve shown) and/or a different potting compound. We’ve heard good things about E-90FL, JB Weld Plastic Welder, 3M DP-420, and 3M DP-620 from various customers.

We realize the importance of solving this issue so that you and other can count on our products! I can assure you that we are doing everything we can to solve this problem and back up that solution with scientific testing and results. We hope to share the results of all of our testing, successful and unsuccessful, when we are done in hopes that others can learn from it.

Thank you again for sharing your concern and your solution.

-Rusty

6 Likes

Thanks for an explanation Rusty,
Have a few questions

  • what depth are these leaks starting to appear (assuming mid 70F)
  • I would have thought those o rings we fit in penetrators would prevent leaks?
    Rgs
    Grant
    Oz

Hi Grant,

I got back to your email as well. To answer these questions:

  1. If the penetrator is strained or otherwise damaged while under warm conditions then it could cause a leak at shallow depths or during a vacuum test.

  2. The O-rings seal the penetrator body to the end-cap, but there is also a seal between the epoxy potting compound and the cable and penetrator. That second seal is what is affected here.

-Rusty

Thanks Rusty, yes read email reply :ok_hand:

Appreciated the quick response

Regards
Grant
Oz

Very interesting information. I was unaware that you were putting so much effort into solving this problem. I certainly understand that it can be difficult to replicate real world conditions and failure modes in the lab. I’m glad to know you’re working on it. I can say for sure that in our operating conditions these fittings are failing at a much higher than normal rate.

The heat issue is very concerning. We operate our ROV from a boat in Florida, and there is no practical way to keep it cool while transporting it offshore in the summer sun. We store and maintain all of our equipment in a warehouse that is kept cool with fans. No A/C. Additionally, the design of the BlueROV makes it nearly impossible to avoid putting some amount of strain on the penetrators.

That said, we have had zero leaks with cable penetrators from other manufacturers in these conditions. So this is a solvable problem.

Have you considered using a material other than aluminum for these things? I know that there is a lot of pressure to keep the overall cost down, and stainless steel commercial cable penetrators are $100+ each. But aluminum is a horrible material for chemical bonds…

What about some type of plastic like PVC or polypropylene? Maybe brass if plastic isn’t strong enough… Have you experimented with those at all?

1 Like

Thanks for the post. Ive been struggling for a while with getting a vaccum on my unit… This explains the problem. we are working in 27C water and at 100m depths. Constantly getting failures.

Is there a good reason why you stopped potting the inside of the penetrator?

@StrikeLines - Thanks for understanding and for the suggestions! To be clear, if the epoxy is heated, it will return to its full strength and properties when cooled, but it’s susceptible to damage while it is warm.

We have considered a lot of design options including alternative materials like stainless steel or plastic. We did some testing but some of that got cut short by COVID. We’re pursuing some promising options that don’t have potting at all so we wouldn’t have to worry about those issues. It’s still early to say much on that.

@JoeB - I’m sorry you’ve had issues with failures in these conditions. Please feel free to reach out to our support team to get help making this right.

@Mikxie - Yes, in our original testing of the current potting compound we determined that there wasn’t a performance difference between potting both ends of the penetrator or just the top end. We’re looking into this again, however, as a few people have mentioned that they have had good luck by potting the inside.

1 Like

Hi @StrikeLines,

Can you provide examples of the other cable penetrators you have used in the past? We are not are not familiar with many marine cable pass-throughs that are not full connectors.

-Adam

DOW BetaSeal 1527EP is used to fasten aluminum parts together on my aluminum chassis Lotus Elise. The aluminum appears to be anodized as well, but could be some other process. The parts are sometimes mechnically adhered using rivets, but what I’ve read is that those rivets are only useful for holding the parts until the product has cured.

1 Like

Adam,

You know, that’s a good point. I made an assumption there. I just looked over all our sonar equipment. In every case, the subsea electronics enclosures are accessed via rubber molded Subconn-style circular connectors.

This was the case for Klein, Edgetech, Benthos and Geoacoustics hardware. None of them used a epoxied pass-through cable penetrator.

Sonar tow cables are routinely terminated with epoxy molds to splice the cable to the connectors, and that’s what I was thinking about when I made that comment. But of course, that’s a totally different animal.

Is it possible to use 316L stainless steel material to process this kind of bolt? We used 316L stainless steel to process a little about one hundred such hollow bolts. So far we have not found any leakage problems. Our sealing method is to use a black AB epoxy glue for sealing. This epoxy glue looks like glass or crystal after solidification, the surface is very smooth and the hardness is very high, the hardness is about 80 SHORE D. After this epoxy glue is completely cured, we will use a relatively high viscosity and very soft silicone rubber to coat the ends of the screw. Until now we have not had any problems

Hi @adam,

As other BR2 users we also experiment more and more potting issues, I have 2 pottings starting to delaminate after a -100m dive we did last week. We already sealed the inside of these penetrators, do you think it can prevent the leak and stop the delamination process if we apply a layer of G-flex resin with a brush on the potting in order to fix it back to the penetrator.

Thanks in advance

Steven

. I got aGood advise regarding epoxy… You need to get All air (vacuum) out of the resin or it can leak, so maby this can be an issue?.. Another idea: use scotchcast at the top of the epoxy, so it will close itself with the preassure?.. But im soon going over to Seacon (or similar) for moore Pro system, but will change the bottle config too, downtime will be at a minimum for changing components.

Please let me know if you’re able to find a Subconn style connector that can handle full battery current. I haven’t come across one yet.

I think it is about 100 amps at 12 volts.

We have encountered more and more thruster epoxy leaks recently. Two on our last dive, and one on each of the two fields tests before. These are tricky to detect and especially problematic when you have 18 holes end plate that requires complete ROV disassembly.

Hi @Stevenlb,

Throughout this continuing process, we’ve learned that we cant say for sure what will improve penetrator performance, until after we have tested an appreciable number across numerous environmental and operational conditions to get a statistically significant idea of performance.

I honestly have no idea if adding G-flex as you describe would help, but I do know that easy fixes often prove temporary, or don’t work quite as intended. In any case, it shouldn’t hurt.

-Adam

Hi all,

When using potting compound are you warming the two components up? We have found by warming the two potting compounds in hot water we get a better adhesion than using the same product at room temperature.

Please be careful when warming!

Kind regards

Deep.supplies

1 Like

Let’s say we have leaks, what’s the method of repair? I’m sure we aren’t replacing the core at $120 a pop, so are we cutting the penetrator off and soldering extensions?