The “Custom” checkbox in BlueOS is for manually specifying a value for a parameter, instead of choosing one of the available options, and the “Force” option allows specifying a value that is not within the range of values the parameter is described as supporting. That’s not relevant for your situation.
It’s a little confusing, but in this case there are a couple of available options for the AHRS_ORIENTATION parameter (depending on firmware version) that allow you to indicate that you want to use a non-standard orientation, and then you specify the actual values for that orientation using other corresponding parameters: