I’m not sure about material testing/approval for potable water, so I’m probably not helpful on that front, particularly if you’ve done some dives with the same ROV in non-potable water (in which case the whole ROV may no longer be allowed without explicit testing and approval after a thorough clean, regardless of the materials it’s made of).
Buoyancy works by volume displacement, and air is highly compressible so bags of air are a poor choice. Maximum buoyancy comes from minimising material weight while maximising volume, so you’d want a lightweight pressure vessel that can handle the pressures you’re going to. Some empty sealed bluerobotics acrylic tubes could do the job, although you’d need to do a mass comparison with the foam you’d be replacing to determine the viability (or technically an average density comparison of sealed tube vs foam).
The shape may be problematic, so if you’re interested in alternatives you could source (or make) some sealed boxes (at least with a flat base), but you’d need to pressure rate them to whatever depths you expect to go to. That could end up as a winning solution though, because they might not need to be as thick as the bluerobotics tubes if your water tanks aren’t as deep, so you could end up shaving off some mass that way.