Quidn
5
Personally I never concern this much, but if someone has the ability and will to breach your devices in home through the USB port, probably it doesn't matter whether that USB port is powered or not. Because that can be so easily bypassed with a externally-powered USB hub or cable.
If you're really that concerned, you should physically destroy the USB and TTL pins on the SoC. But even after doing so, your keys stored in the unencrypted flash would still vulnerable.
In my opinion, it'd be better to consider some additional security like granting access only when your device is connected by you. Additional security inside the network would also be needed.
Me? I've configured both ethernet ports as different WAN. I don't care about the USB port, and there's no additional security measures. Do I feel safe? Well, even if someone were breach my network, there would be almost nothing to do without very detailed information and exact credential.