I gave up on wireguard and switch to OpenVPN. Even using the same port as Wireguard, it worked right away.