The router has MWAN3 enabled which will change the interface you are using to access internet.

So I think when you disable modem the route table is changed my mwan3.