Spitz AX (GL-X3000) hangs several times per week, only reboot helps

Hello,

I started to have issues with my GL-X3000 router connected to GSM network about 5 months ago.

Symptoms: device "hangs", only power indicator is shown. All Wi-Fi is off. It happens about two times per week. Pretty annoying. Only powering device off and on helps.

I installed external MMC card and redirected syslog there.

This night it happened again. I woke up at 8 a.m. and saw router in this state.
Grepped the log for "Thu Jul 4":
jul-4.zip (5.3 KB)

Thu Jul 4 08:21:01 - start of the last reboot (successfull)

Suspicious log parts are (logged when device "hanged"):

Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.745039] 7981@C01L3,wifi_sys_conn_act() 1115: wdev idx = 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.751070] 7981@C08L3,hw_ctrl_flow_v2_connt_act() 215: wdev_idx=2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.873428] 7981@C15L3,WPABuildPairMsg1() 5280: <=== send Msg1 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.880092] 7981@C15L3,PeerPairMsg2Action() 6202: ===>Receive msg 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.886873] 7981@C15L3,WPABuildPairMsg3() 5557: <=== send Msg3 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.893556] 7981@C15L3,PeerPairMsg4Action() 6632: ===>Receive msg 4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140] 7981@C15L2,PeerPairMsg4Action() 6994: AP SETKEYS DONE(rax0) - AKMMap=WPA2PSK, PairwiseCipher=AES, GroupCipher=AES, wcid=6 from 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140]
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPREQUEST(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPACK(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:1ce2:89fa:8942:676b%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:d4a3:a76b:6c1d:6028%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:25 2024 cron.err crond[4346]: time disparity of 215 minutes detected
Thu Jul  4 02:26:45 2024 cron.err crond[4014]: time disparity of 216 minutes detected
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.697560] 7981@C08L3,ap_peer_auth_req_at_idle_action() 717: AUTH - MBSS(2), Rcv AUTH seq#1, Alg=0, Status=0 from [wcid=1023]2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.debug kernel: [  124.710500] entrytb_aid_aquire(): found non-occupied aid:9, allocated from:4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.717566] 7981@C13L2,MacTableInsertEntry() 1537: New Sta:2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.726586] 7981@C08L3,ap_cmm_peer_assoc_req_action() 1647:  Recv Assoc from STA - 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.735931] 7981@C08L3,ap_cmm_peer_assoc_req_action() 2170: ASSOC Send ASSOC response (Status=0)...
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.745039] 7981@C01L3,wifi_sys_conn_act() 1115: wdev idx = 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.751070] 7981@C08L3,hw_ctrl_flow_v2_connt_act() 215: wdev_idx=2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.873428] 7981@C15L3,WPABuildPairMsg1() 5280: <=== send Msg1 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.880092] 7981@C15L3,PeerPairMsg2Action() 6202: ===>Receive msg 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.886873] 7981@C15L3,WPABuildPairMsg3() 5557: <=== send Msg3 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.893556] 7981@C15L3,PeerPairMsg4Action() 6632: ===>Receive msg 4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140] 7981@C15L2,PeerPairMsg4Action() 6994: AP SETKEYS DONE(rax0) - AKMMap=WPA2PSK, PairwiseCipher=AES, GroupCipher=AES, wcid=6 from 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140]
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPREQUEST(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPACK(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:1ce2:89fa:8942:676b%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:d4a3:a76b:6c1d:6028%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:25 2024 cron.err crond[4346]: time disparity of 215 minutes detected
Thu Jul  4 02:26:45 2024 cron.err crond[4014]: time disparity of 216 minutes detected
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.697560] 7981@C08L3,ap_peer_auth_req_at_idle_action() 717: AUTH - MBSS(2), Rcv AUTH seq#1, Alg=0, Status=0 from [wcid=1023]2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.debug kernel: [  124.710500] entrytb_aid_aquire(): found non-occupied aid:9, allocated from:4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.717566] 7981@C13L2,MacTableInsertEntry() 1537: New Sta:2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.726586] 7981@C08L3,ap_cmm_peer_assoc_req_action() 1647:  Recv Assoc from STA - 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.735931] 7981@C08L3,ap_cmm_peer_assoc_req_action() 2170: ASSOC Send ASSOC response (Status=0)...
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.745039] 7981@C01L3,wifi_sys_conn_act() 1115: wdev idx = 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.751070] 7981@C08L3,hw_ctrl_flow_v2_connt_act() 215: wdev_idx=2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.873428] 7981@C15L3,WPABuildPairMsg1() 5280: <=== send Msg1 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.880092] 7981@C15L3,PeerPairMsg2Action() 6202: ===>Receive msg 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.886873] 7981@C15L3,WPABuildPairMsg3() 5557: <=== send Msg3 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.893556] 7981@C15L3,PeerPairMsg4Action() 6632: ===>Receive msg 4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140] 7981@C15L2,PeerPairMsg4Action() 6994: AP SETKEYS DONE(rax0) - AKMMap=WPA2PSK, PairwiseCipher=AES, GroupCipher=AES, wcid=6 from 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140]
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPREQUEST(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPACK(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:1ce2:89fa:8942:676b%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:d4a3:a76b:6c1d:6028%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:25 2024 cron.err crond[4346]: time disparity of 215 minutes detected
Thu Jul  4 02:26:45 2024 cron.err crond[4014]: time disparity of 216 minutes detected
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.697560] 7981@C08L3,ap_peer_auth_req_at_idle_action() 717: AUTH - MBSS(2), Rcv AUTH seq#1, Alg=0, Status=0 from [wcid=1023]2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.debug kernel: [  124.710500] entrytb_aid_aquire(): found non-occupied aid:9, allocated from:4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.717566] 7981@C13L2,MacTableInsertEntry() 1537: New Sta:2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.726586] 7981@C08L3,ap_cmm_peer_assoc_req_action() 1647:  Recv Assoc from STA - 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.735931] 7981@C08L3,ap_cmm_peer_assoc_req_action() 2170: ASSOC Send ASSOC response (Status=0)...
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.745039] 7981@C01L3,wifi_sys_conn_act() 1115: wdev idx = 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.751070] 7981@C08L3,hw_ctrl_flow_v2_connt_act() 215: wdev_idx=2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.873428] 7981@C15L3,WPABuildPairMsg1() 5280: <=== send Msg1 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.880092] 7981@C15L3,PeerPairMsg2Action() 6202: ===>Receive msg 2
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.886873] 7981@C15L3,WPABuildPairMsg3() 5557: <=== send Msg3 of 4-way
Thu Jul  4 02:26:53 2024 kern.notice kernel: [  124.893556] 7981@C15L3,PeerPairMsg4Action() 6632: ===>Receive msg 4
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140] 7981@C15L2,PeerPairMsg4Action() 6994: AP SETKEYS DONE(rax0) - AKMMap=WPA2PSK, PairwiseCipher=AES, GroupCipher=AES, wcid=6 from 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 kern.warn kernel: [  124.905140]
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPREQUEST(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.info dnsmasq-dhcp[5014]: DHCPACK(br-lan) 192.168.1.246 2a:02:17:60:72:f1
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:1ce2:89fa:8942:676b%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 02:26:53 2024 daemon.err odhcpd[3892]: Failed to send to 2001:999:250:96f0:d4a3:a76b:6c1d:6028%modem_0001@rmnet_mhi0 (Bad file descriptor)
Thu Jul  4 08:20:59 2024 cron.err crond[4446]: time disparity of 569 minutes detected

The issue shall not be related to the hot temperature - it was relaively cold night in my place.

Details:

  • Router: Spitz AX (GL-X3000)
  • Firmware version: 4.0 / 0409release3 / 2024-06-15 13:30:17(UTC+08:00)
  • Router is connected to internet via GSM (one SIM is used), mobile provider: Elisa (Finland). IPv6 is in use
  • DNS: received from Elisa
  • DHCP is used.

Will be happy to get any help!

I also encountered the same issue with this release. Can you try the firmware 4.4.8? Else you can disable IPV6 from LuCi -> Network-> Devices and keep monitoring.

@SpitzAX3000 @vitaly_repin
hello
When you encounter this problem, can you still access the web page to obtain the log?

Yes I can access the web page, but I didn’t bother to investigate it at the time because I was on a vacation.

In order to investigate it and get related logs I need to reinstall 4.4.9 firmware. My issue is little different from the reported one above: my modem was stuck with no internet and no WiFi. Yet, the admin panel was accessible. When I logged in I could not manually not automatically reconnect to the internet, till it was rebooted.

1 Like

Thank you for your reply
Mr. SpitzAX3000, is your current module firmware R03A03?
I would also like to ask that after having encountered it once in 4.4.9, you went back to 4.4.8 and then never encountered it again?

Correct.

Correct.

Thanks.

Not via Wi-Fi. The wireless network is down.

I have not tried LAN socket however. Good point, will check next time.

But this was the reason why I offloaded logs to the external MMC.

And I copied the log file from there after powering device off and on.

II added this line to crontab to save logs to MMC:

* * * * * logread -F /tmp/mountd/disk1_part1/log

hello
This log doesn't have much useful information
I think maybe too much time has passed since the problem occurred
The resulting useful logs have been washed out

If it happens again, can you give the router another way to access the Internet
The device is then shared with goodcloud
I'll investigate remotely
Technical Support via GoodCloud.tar (192 KB)

Hello,

Yesterday I downgraded to firmware 0408release4.

This morning I had two issues (approx. 1 hour between them):

  • Router rebooted
  • Router powered off (exactly in the way I described in my first post) and when I powered it off and on, it could not boot - even "power" indicator was not shown. I had to disconnect it from the charger for 10 mins and then connect again. It was able to boot. It was pretty hot. And it can not be related to the temperature conditions. Room temperature is 25 degrees Celsius.

Attaching the logs (including two dmesg logs) in case they contain any interesting data.

I am upgrading now to the recent firmware and will wait for a new issue to check is it possible to connect to the router via Ethernet.

logs-firmware-0408release4.zip (574.7 KB)

Hello,

It happened again with firmware 4.0 0409release3.

Unfortunately in this state, no router network interface is available. LAN ethernet connection also stops functioning.

Could you instruct me what kind of logs do you need to investigate?

I have huge 120 GB MMC card inserted into the router. It means that I can generate whatever logs/periodic dumps from cron.

What commands would you run in the router if you manage to connect to it remotely (which is impossible as no network interfaces are active). I can simply add them to crontab and run periodically. Then when the router hangs, I'll extract the MMC card and copy the logs from it.

Is your power adapter original?
Does the modem use a GSM network?
If I remember correctly, the x3000 does not support GSM networks
Router hot? The fan's not on?

Yes, the power adapter is original.

The network is LTE.

This is what I see in the admin panel after reboot:

  • LTE FDD

  • Band 3

  • ul_bandwidth 20M

  • dl_bandwidth 20M

  • RSRP -93 Excellent

  • RSSI -58 Excellent

  • RSRQ -14 Fair

  • SINR 10 Good

  • ID 18E003

  • NR5G-NSA

  • Band 78

  • dl_bandwidth 100M

  • RSRP -95 Excellent

  • RSRQ -11 Excellent

  • SINR24 Excellent

BTW, I also noticed that sometimes after powering the device off and on, the SIM card is switched (I have only one SIM card therefore I have to manually switch back to the active SIM).

I do not hear the fun.

But I feel it's hot. (And it's not related to the indoor temperature - we have quiet cold summer in Finland now)

image
Did you turn on this switch?
If not, does it's all the time stay in the sim2 card slot after the startup?

/etc/init.d/gl_fan restart

Set the starting temperature of the fan to 40 degrees

This morning the situation repeated.

This time I extracted MMC card after device was rebooted into the hanging state.

Attaching the logs. It looks like there is an issue with cellular modem:

Fri Jul 12 08:36:05 2024 user.notice mwan3track[23259]: Interface modem_0001 (rmnet_mhi0) is offline
Fri Jul 12 08:36:07 2024 daemon.notice netifd: modem_0001 (29491): sh: 1: unknown operand
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491): SIM not ready
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:10 2024 daemon.notice netifd: modem_0001 (29491): ERROR
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): Failed to parse message data
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): ERROR
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): OK
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): Device only supports raw-ip mode but is missing this required driver attribute: /sys/class/net/rmnet_mhi0/qmi/raw_ip
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491):
Fri Jul 12 08:36:11 2024 daemon.notice netifd: modem_0001 (29491): ERROR

attaching the logs.

fel.zip (23.7 KB)

I also changed the fan settings as instructed.

Monitoring the affect. But now router reboots several times a day. Looks like it's a hardware failure of some kind. It becomes unusable. I will contact support with warranty request.

Looks like it switched to a different slot
Had it already been rebooted once before when the problem occurred?

Can I take a look at it remotely?

Yes, it rebooted once when the problem occured.

I can provide you with remote access , sure.

But not when it is in this hanging state as no network connectivity is available in that state.

BTW, it hanged again while I was typing this message :frowning:

I made a video which demonstrates the powering on process: Spitz issie | vitaly.repin | Flickr

I set up GoodCloud.

How can I send you the credentials to access the router?

I have sent you a private message, please check

A totally other idea: The distance between the LTE-antenna and the power-supply is very short. Although the power-supplies of GLinet seam to be built very good, they can't have an infinite immunity against radiation.
Is it possible that the power-supply is disturbed by the RF-radiation of the LTE-modem?

Can confirm, I got into same situation, all LED off except the power icon, wifi off, soft reboot via system UI didn't help, must unplug the power cable to cycle. Fan didn't turn on, so I added a 12" USB fan under the modem and now cool(49-50F) and stable again. Probably due to over heat issues.