You could try setting it to the same port the server listens on 54321 as this was working on some configurations I had.
Could you post your redacted settings from the wireguard client page in the router interface.