Hi @HoaiThanh -
Can you expand on what you mean by “controller LQR?” What are your goals for the vehicle and your custom software?
The ArduPilot firmware that typically runs on BlueOS uses the Navigator as a source of input/output/motion sensors. This firmware has nothing to do with Matlab…
I’d recommend researching ArduSub, and seeing what functionality you’d like to modify to accomplish your goals. Depending on the application, a Lua script, which is executed by the autopilot, may also be a good solution!