MAC works only inside a network. As soon as some network package crosses the border of a router, the MAC will be deleted from the package. So you won't be able to reference a MAC address of a device that's connected by another WireGuard server.
I have Client and Server both on the same Gl inet router. So turning off maybe not going to work or solution as this will turn off the server and Device D will not be able to communicate at all back.