If you don't agree with the medical opinion
You can get an alternative opinion if you wish. Alternatively, you can question the opinion of your IRMP to understand their opinion better.
The medical opinion will guide your decision.
You must be able to support the reasons for your decision, remembering that non-medical factors (such as availability of other employment) are not material factors for these decisions.