I'm not sure where to start with troubleshooting this, so please bear with me.
I have a GLiNet Flint2 at home on my home network. I have a Beryl-AX Travel modem for use when I'm away.
Between the two, I have followed the plethora of guides to set up a Wireguard VPN between the two so that I can access my home network, services in my home country, be able to work etc...
Nearly everything works perfectly fine with services I have running at home on my home server (docker containers which are open to the internet using NGINX Proxy Manager and Cloudflare DNS). The odd thing that just refuses to work no matter what setting I change on the two routers is accessing my NAS on it's public address (using my own domain and Cloudflare DNS).
If I turn off the Wireguard, everything is working perfectly fine, if I turn it on - it refuses to accept the certificates for any connections to my NAS.
Any thoughts on where I can start to troubleshoot this issue?
You just need to change the VPN Proxy mode to "Based on the target domain or IP" and exclude your NAS FQDN there. This should work, but you should test it to be sure.
I don't know your exact network topology, so it's difficult to troubleshoot.
If you get the Flint2's login screen, the VPN is still routing the traffic wrong.
My Wireguard client settings on my Beryl AX were set to use the remote DNS server addresses (my two Adguard Servers), I put a DNS rewrite on them so that my NAS's FQDN points to the local IP and bingo, it works.