Did this end up getting resolved?
I’ve just tried with both ArduSub 4.1.0 (latest stable) and 4.0.3 (previous stable), and leaving the AUXOUT5 as the default RELAY_PIN works fine for me with the relay_1 joystick functions.
I then tried to get AUX 1 working as a relay, which worked once I set BRD_PWM_COUNT to No PWMs (since they start counting from AUX 1) 