These ‘forward, lateral, throttle’ commands that you are referring to are just pilot input. They are do not result in proportional change in motor speed. The motor output will depend on the operating mode that you are using. In manual mode, the input command is just passed directly to the speed controllers (there is no closed loop control of rotor speed here, or in any case). In the other modes, the motor output is closed loop to maintain attitude and speed of the vehicle. This closed loop control (again it’s not controlling rotor speed, but relative thrust to maintain attitude) will compensate effectively for the nonlinearities you pointed out.
ArduSub already has position and waypoint controllers (not deemed stable, but demonstrated to work). I recommend you look into this rather than rolling your own with automated pilot input.
You don’t have to reverse engineer anything, the code is open source and available on github. I recommend you check that out to gain an intimate understanding of how it works.
The motor output is saturated, no normalization is performed.
It again, depends on the operating mode that you are in. In stabilize or depth hold mode, the angle target is locked 250ms after the rate command drops to zero. Then it is stabilizing angle.