GL-X1200 5G WiFi hostapd Stops Authenticating

evangilbert · 2020-01-17T17:58:12.379Z

I’m encountering a critical bug with the GL-X1200 router and I’m curious if anyone else has been experiencing the same thing. I can very quickly cripple the default 5G WiFi network by simply connecting between 5-10 devices to the network. Once broken, any devices already connected to the 5G no longer reach the internet (or resolve the router’s ip) and any new devices attempting to connect are rejected for failed authentication (with correct security info being sent).

Here’s my repro steps:

Take brand new GL-X1200 out of the box.
Plug in an ethernet cable to the WAN port (the private network we are plugging into is running 10.x.x.x so no chance of conflicts).
Turn on the router
Begin connecting devices to the default GL-X1200-xxx-5G WiFi (iPads, iPhones, Android phones, and laptops are what I was connecting)
This has varied for me but somewhere between 5-10 devices suddenly things break and connected devices no longer can access the internet (or resolve the router’s ip), and no new devices can connect.

Digging a little deeper it appears to be an issue with hostapd. Looking at hostapd logs it looks like handshaking is continually failing as I start seeing this in the logs:

Thu Jan 16 01:20:21 2020 daemon.debug hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: sending 1/4 msg of 4-Way Handshake
Thu Jan 16 01:20:22 2020 daemon.debug hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: EAPOL-Key timeout
Thu Jan 16 01:20:22 2020 daemon.debug hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: PTKSTART: Retry limit 4 reached

The issue can be resolved by running the following command on the 5G wlan interface:

hostapd_cli -i wlan0 disable
hostapd_cli -i wlan0 enable

However, while this fixes things it’s only a matter of time before the issues reappears. Keeping around 5-10 devices connected the issue usually reappears within about 10-20 minutes. And that’s without connecting/disconnecting any devices.

Interesting to note is that the 2.4G WiFi does not have this issue. That network remains fully functional no matter how many devices I try connecting. When the 5G network is broken I can connect just fine to 2.4G and access the internet as expected.

These are brand new GL-X1200 devices that are running the 3.027 firmware.

Seems to me like this should effect anyone running these devices. Curious if anyone else has encountered this. Happy to provide more details. Help!

evangilbert · 2020-01-20T18:07:18.655Z

Has no one out there experienced anything like this with the GL-X1200? Anyone out there with a GL-X1200 that can verify they haven’t encountered anything like this?

Reproducing this has been difficult. With a similar number of devices, sometimes I can get it to happen within 10 minutes and other times it’s taken much longer. But leaving the device on with devices connected, the longest we’ve managed to have the router on and working is a day and a half.

Really could use any helpful thoughts or ideas anyone has to offer

alzhao · 2020-01-21T10:36:54.235Z

Guys went to holidays for 2 weeks.

I have to bookmark your post and come back later. I do not have an idea of why.

wellnw · 2020-01-31T05:30:31.911Z

@evangilbert?
Sorry to reply to you so late. Since there is no equipment on hand during vavation,you can modify it as follows
1.wifi-device adds the following fields
option noscan ‘0’

2.wifi-iface increase
option disassoc_low_ack ‘0’

After adding, use the following instructions
wifi restart

png632×695 97.9 KB

In addition, I will return to the company on February 5th,Beijing time,and I will deal with your problem as soon as possible

evangilbert · 2020-01-31T21:43:14.879Z

Thanks for the suggestion!

We tried out the X1200 testing firmware: openwrt-x1200-3.100-1218.tar which appeared to improve reliability of the router. We looked through a diff and had noticed the exact changes you commented on above! The testing firmware initially appeared to fix our issues, however I heard the device finally entered this bad state again today. That means it stood up for about a day and a half (an improvement on the 20-30 minutes window we previously saw with similar activity).

I will try making your above changed on the 3.027 firmware next week and report findings,
Thanks again!

Hoff · 2020-02-01T09:54:47.434Z

Thanks for the feedback.
We are trying to solve the problem you mentioned as soon as possible, but because of the delay of the Spring Festival, we need you to wait for a while.

wellnw · 2020-02-08T09:25:32.546Z

evangilbert:

I’m encountering a critical bug with the GL-X1200 router and I’m curious if anyone else has been experiencing the same thing. I can very quickly cripple the default 5G WiFi network by simply connecting between 5-10 devices to the network. Once broken, any devices already connected to the 5G no longer reach the internet (or resolve the router’s ip) and any new devices attempting to connect are rejected for failed authentication (with correct security info being sent).

Here’s my repro steps:

Take brand new GL-X1200 out of the box.

Plug in an ethernet cable to the WAN port (the private network we are plugging into is running 10.x.x.x so no chance of conflicts).

Turn on the router

Begin connecting devices to the default GL-X1200-xxx-5G WiFi (iPads, iPhones, Android phones, and laptops are what I was connecting)

This has varied for me but somewhere between 5-10 devices suddenly things break and connected devices no longer can access the internet (or resolve the router’s ip), and no new devices can connect.

Digging a little deeper it appears to be an issue with hostapd. Looking at hostapd logs it looks like handshaking is continually failing as I start seeing this in the logs:
Thu Jan 16 01:20:21 2020 daemon.debug hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: sending 1/4 msg of 4-Way Handshake
Thu Jan 16 01:20:22 2020 daemon.debug hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: EAPOL-Key timeout
Thu Jan 16 01:20:22 2020 daemon.debug hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: PTKSTART: Retry limit 4 reached
The issue can be resolved by running the following command on the 5G wlan interface:
hostapd_cli -i wlan0 disable
hostapd_cli -i wlan0 enable
However, while this fixes things it’s only a matter of time before the issues reappears. Keeping around 5-10 devices connected the issue usually reappears within about 10-20 minutes. And that’s without connecting/disconnecting any devices.

Interesting to note is that the 2.4G WiFi does not have this issue. That network remains fully functional no matter how many devices I try connecting. When the 5G network is broken I can connect just fine to 2.4G and access the internet as expected.

These are brand new GL-X1200 devices that are running the 3.027 firmware.

Seems to me like this should effect anyone running these devices. Curious if anyone else has encountered this. Happy to provide more details. Help!

@evangilbert
I have set up the environment and hung up the equipment, waiting for the problem to appear. If you have a faster way to reproduce the problem, please provide it to me. Also, you can follow the steps below, if you can reproduce the problem.

Please ssh login router and run the following command
uci set wireless.radio1.log_level=‘0’
uci set wireless.radio0.log_level=‘0’
uci commit wireless
wifi
After connect failed, give me logread and dmesg info. Thank you.

wellnw · 2020-02-10T11:42:24.960Z

@evangilbert
I used 4 android phones and 1 iphone test here, and reproduced the problem, and found the abnormal information as follows:
wlan0: STA 50: 5b: c2: cf: 3a: eb IEEE 802.11: deauthenticated due to local deauth request
After the modification, there were no exceptions after testing for 6 hours. I will continue to test and synchronize the modification to you:
Modify wireless configuration
/etc/config/wireless ，add the option to 5G wifi-iface
option wpa_group_rekey ‘0’
option disassoc_low_ack ‘0’

image983×915 199 KB

If you have no problem testing, I will update the firmware to you, thanks again.

evangilbert · 2020-02-10T18:53:34.689Z

@wellnw
Thanks for investigating. I’ll go update our X1200 and report back tomorrow.

evangilbert · 2020-02-10T21:01:17.524Z

@wellnw

After adding

option wpa_group_rekey '0'
option disassoc_low_ack '0'

to our /etc/config/wireless file for the 5g wifi-iface (and restarting router) I was still able to recreate the issue within ~15 minutes on our target deploy. The router broke when connecting around 8-10 devices this time. I am connecting with a mix of dynamic and static IP assigned devices, comprised of iPads, macbooks, android phones, and a single zebra printer. I will try your wifi-iface changes on an out of box device next and report.

I’d be happy to provide a backup of our target deploy or a full syslog via email. When the router breaks, along with seeing the

Mon Feb 10 20:20:50 2020 daemon.info hostapd: wlan0: STA ##:##:##:##:##:## IEEE 802.11: deauthenticated due to local deauth request

messages, I also see continual dnsmasq errors:

Mon Feb 10 20:21:45 2020 daemon.err dnsmasq[3350]: failed to send packet: Resource temporarily unavailable

Some top level details on our target deploy:
Within /etc/config/network (router ip is 172.30.192.1 with netmask of 255.255.240.0):

config interface 'lan'
	option type 'bridge'
	option ifname 'eth0.1'
	option proto 'static'
	option ip6assign '60'
	option hostname 'GL-X1200-1f2'
	option macaddr '##:##:##:##:##:##'
	option default_macaddr '##:##:##:##:##:##'
	option ipaddr '172.30.192.1'
	option netmask '255.255.240.0'

We define a large number of DHCP static IPs. Goal is for dynamically distributed IPs to be in 172.30.192.x, and then static IPs will assign within 172.30.193.x, 172.30.194.x, and 172.30.195.x depending on type of device.
In /etc/config/dhcp:

config dhcp 'lan'
	option interface 'lan'
	option leasetime '12h'
	option force '1'
	option dhcpv6 'server'
	option ra 'server'
	option ra_management '1'
	option start '2'
	option limit '253'
...
config host
	option dns '1'
	option mac '##:##:##:##:##:##'
	option ip '172.30.19[3, 4, or 5].#'
	option leasetime '365d'
	option name 'NAME'

Aside from configuring our 2.4g and 5g wifi’s with new unique SSIDs and passwords we aren’t touching anything else on the router. We aren’t changing any DNS settings.

evangilbert · 2020-02-10T21:43:45.736Z

I’m also still able to reproduce on a factory reset to the 3.027 firmware. I sshed onto the clean router and used uci set to mark the wireless config changes. Restarted the router, verified the wpa_group_rekey and disassoc_low_ack settings took effect. Then again I connected 8-12 devices (~3 Android phones, an iPhone, a macbook, ~6 iPads, and a zebra printer). Within about 5-10 minutes I was back in the broken state.

logread is still showing the “deauthenticated due to local deauth request” hsotapd messages with a lot of dnsmasq “failed to send packet: Resource temporarily unavailable” messages mixed in. 2.4G still seems unaffected.

wellnw · 2020-02-11T01:03:19.636Z

@evangilbert
Trouble sending me a detailed abnormal log by email. I sent you an email yesterday. You can reply directly to me. I will continue to analyze this issue. Thank you for your feedback.

wellnw · 2020-02-12T08:25:49.695Z

@evangilbert
Hi, I updated the wifi driver and configuration. The firmware has been sent to you by email. I look forward to your feedback.
I have now connected 11 terminals and four are playing videos. At present everything is normal, as shown below

image1026×739 43.7 KB