If you only want to use ssh. I think you can disable push route, or you can try to delete all route rules about vpn.