this happens when you just switch it to on?

Correct?

make sure its only checked against turning on, for turning off such things are normal because the killswitch won't work on a manual user interaction.

as a work around you can enable Block Non-VPN Traffic.

or when you want more advanced fine control:

via luci -> network -> firewall, and then remove the forwarding zone wan from the lan zone, if vpn policies fail you can type ip rule in the cli you will find the mark for wan, then just make a traffic rule allowing every traffic with this mark to passthrough wan like:

src: any zone
dest zone: wan
match mark: (a hexadecimal long string from ip rule)
target/action: accept