Well it really was the interference between the USB3 Controller from my SSD and the 2.4 GHZ Wifi.
I took a longer cable and placed the disk on an other spot and Wifi speed is almost back to normal.

Now I understand why most router manufacturer still are using USB2 ports for their devices.