Kill Switch not taking effect on VPN disconnect

I disconnected my VPN and I have the kill switch turned on and I see that it just reverts back to my normal ISP provider, is this just because I am disconnecting manually or is something wrong?

What model and firmware?