An Ethernet channel is the complete end-to-end connection of a cable run from a router/switch to your ROV. Suffice it to say that from powered-device to powered-device the limit is 328 feet.
If you don’t intend to dive deeper than that, running 8 wires instead of two may work for you. Keep in mind you will use up some of that footage going sideways as well as down. If any water gets inside the Cat5 ethernet cable, it will have to be replaced. If any one of the 8 wires breaks, again, the whole cable will have to be replaced.
Bulk ethernet cable is meant to be installed from one location to another then never move again. For example, ran from the reception area computer to the servers in a back room…fished through walls and across ceiling joists, but once done it never gets twisted around or moved. So it is always composed of 8 solid copper wires (if you avoid the cheapo copper-coated aluminum junk), rather than the stranded (more flexible) cables meant to connect two close computers together temporarily. So it isn’t meant to be rolled up, unrolled off the spool every time you dive the ROV, submerged in water, etc. It will work for a few dives then you may have problems.
With serial communication over two wires, you can use off the shelf stranded speaker cable, and the RS485 protocol is good for about 4000 feet.
Try putting the FathomX into another WTC?