which router are you using now?
I just tried my GL-MV1000 and does the same as you did. Using vpn or not does not affect the speed.