GL-SF1200 dnsmasq stops responding after few hours - few days of uptime

ikseek · February 25, 2023, 8:32pm

I have a pair of GL-SF1200 as a main 5GHz + 2.4GHz router and GL-E750 Mudi as a 5GHz WDS extender.
Once every day or two DNS server on GL-SF1200 stops responding. When I enter it’s SSH, I see that dnsmasq process is stuck in a “D” sate, so it’s impossible to kill it and restart, I have to reboot the router.

The GL-SF1200’s firmware is 3.215 (2022-09-21 11:49:06)

Here is some info from dnsmasq /proc filesystem:

root@GL-SF1200:/proc/1751# cat status
Name:	dnsmasq
Umask:	0022
State:	D (disk sleep)
Tgid:	1751
Ngid:	0
Pid:	1751
PPid:	1
TracerPid:	0
Uid:	453	453	453	453
Gid:	453	453	453	453
FDSize:	256
Groups:	 
VmPeak:	    2936 kB
VmSize:	    2936 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	    1708 kB
VmRSS:	    1708 kB
RssAnon:	     140 kB
RssFile:	    1568 kB
RssShmem:	       0 kB
VmData:	     156 kB
VmStk:	     132 kB
VmExe:	     320 kB
VmLib:	    2284 kB
VmPTE:	      16 kB
VmPMD:	       0 kB
VmSwap:	       0 kB
Threads:	1
SigQ:	2/932
SigPnd:	00000000000000000000000000000100
ShdPnd:	00000000000000000000000000004100
SigBlk:	00000000000000000000000000000000
SigIgn:	00000000000000000000000000001000
SigCgt:	0000000000000000000000000003e003
CapInh:	0000000000000000
CapPrm:	0000000000003400
CapEff:	0000000000003400
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Speculation_Store_Bypass:	unknown
Cpus_allowed:	f
Cpus_allowed_list:	0-3
voluntary_ctxt_switches:	6479
nonvoluntary_ctxt_switches:	261

root@GL-SF1200:/proc/1751# cat syscall 
0 0x4 0x8915 0x7fbcc970 0x1 0x0 0x0 0x7fbcc850 0x77ef15a8

root@GL-SF1200:/proc/1751# cat stack 
[<805354b8>] devinet_ioctl+0x158/0xb10
[<80446620>] sock_ioctl+0x300/0x35c
[<802317a0>] do_vfs_ioctl+0x764/0x800
[<8023188c>] SyS_ioctl+0x50/0x94
[<80117578>] syscall_common+0x34/0x58

luochongjun · February 27, 2023, 7:22am

The state D means uninterruptible sleep. Does it stay that way?
Maybe you can write a monitoring script to fix this.

ikseek · February 27, 2023, 5:13pm

Nope, it never exits that state and stays in that syscall until rebooted. The process can not be SIGKILLed.
I switched off Mudi to check if it’s WDS that triggers the bug.

ikseek · February 28, 2023, 7:06am

Nope, without WDS client dnsmasq is blocked again. Attaching dmesg output with errors in case someone is interested. I probably will try to reflash the router…
dmesg.txt.zip (5.3 KB)

dengxinfa · March 1, 2023, 1:57am

Have you modified any configuration of sf1200 after the factory configuration?

ikseek · March 1, 2023, 2:33am

What do you mean by configuration? I set it up using it’s web GUI to enable both 2.4g and 5g networks and other network options. I configured VPN client in it but didn’t use it for half a year. It worked quite fine for 4 months and then I added E750 as a WDS client and this started to happen.
I didn’t modify it’s firmware, didn’t install any additional software. I enabled luci interface attempting to add a cron task that reboots it at night but it hangs at day time after reboots anyway.

It just hanged again, I checked if there other processes in ‘D’ state besides dnsmasq and turned out there are.

It’s weird to see brctl showmacs br-guest being stuck in the same syscall because guest networks are disabled on the router UI! I tried to run it in the shell and it blocked my shell forever too. I’ll reflash it tonight.

 1743 dnsmasq   2936 D    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 3592 root      5116 D    /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf
 4557 root      5240 D    /usr/sbin/hostapd -s -P /var/run/wifi-phy1.pid -B /var/run/hostapd-phy1.conf
27152 root      1580 D    ip -4 route list table main
27212 root       896 D    brctl showmacs br-guest


root@GL-SF1200:/# cat /proc/1743/stack
[<805354b8>] devinet_ioctl+0x158/0xb10
[<80446620>] sock_ioctl+0x300/0x35c
[<802317a0>] do_vfs_ioctl+0x764/0x800
[<8023188c>] SyS_ioctl+0x50/0x94
[<80117578>] syscall_common+0x34/0x58

[<873e1854>] cfg80211_ch_switch_notify+0x3d0/0x4470 [cfg80211]
[<801eaad0>] kmalloc_order+0x20/0x54
[<80452038>] __kmalloc_reserve.isra.8+0x40/0xa4

root@GL-SF1200:/# cat /proc/4557/stack
[<804ae388>] genl_lock_dumpit+0x30/0x6c
[<804aacb8>] netlink_dump+0x124/0x2d0
[<804ab034>] netlink_recvmsg+0x1d0/0x364
[<80446bb8>] ___sys_recvmsg+0xc0/0x178
[<80447c7c>] __sys_recvmsg+0x50/0x80
[<80117578>] syscall_common+0x34/0x58

root@GL-SF1200:/# cat /proc/27152/stack
[<804aabb8>] netlink_dump+0x24/0x2d0
[<804ab4b0>] __netlink_dump_start+0x130/0x28c
[<80482498>] rtnetlink_rcv_msg+0x1d0/0x394
[<804adfec>] netlink_rcv_skb+0xd4/0x178
[<804ad49c>] netlink_unicast+0x1b4/0x2e8
[<804adb50>] netlink_sendmsg+0x43c/0x498
[<8044763c>] SyS_sendto+0xd4/0x128
[<80117578>] syscall_common+0x34/0x58

root@GL-SF1200:/# cat /proc/27212/stack
[<8048cf34>] dev_ioctl+0x504/0x630
[<802317a0>] do_vfs_ioctl+0x764/0x800
[<8023188c>] SyS_ioctl+0x50/0x94
[<80117578>] syscall_common+0x34/0x58

ikseek · March 8, 2023, 3:12am

Resetting the router didn’t help.
I didn’t set any options except LAN IP range, static DHCP assignments and wifi security/passwords.

Here is the dmesg log

[83286.315218] siwifi_calculate_legrate invalid legrate: 4
[83286.337347] siwifi_calculate_legrate invalid legrate: 4
[83286.360751] siwifi_calculate_legrate invalid legrate: 4
[83286.383752] siwifi_calculate_legrate invalid legrate: 4
[83292.774956] [0]Total force ps drop len : 1
[83343.402041] [1]Total force ps drop len : 7
[83351.716692] [0]Total force ps drop len : 1
[83382.424378] [1]Total force ps drop len : 7
[83392.791207] [0]Total force ps drop len : 1
[83406.109724] [1]Total force ps drop len : 1
[83429.566646] [1]Total force ps drop len : 7
[83459.852887] [0]Total force ps drop len : 1
[83472.274519] [1]Total force ps drop len : 7
[83492.808157] [0]Total force ps drop len : 1
[83512.492623] [1]Total force ps drop len : 7
[83525.913001] siwifi_calculate_legrate invalid legrate: 4
[83525.925009] siwifi_calculate_legrate invalid legrate: 4
[83525.949882] siwifi_calculate_legrate invalid legrate: 4
[83525.990258] siwifi_calculate_legrate invalid legrate: 4
[83526.029687] siwifi_calculate_legrate invalid legrate: 4
[83526.052435] siwifi_calculate_legrate invalid legrate: 4
[83526.075567] siwifi_calculate_legrate invalid legrate: 4
[83526.126618] siwifi_calculate_legrate invalid legrate: 4
[83526.149518] siwifi_calculate_legrate invalid legrate: 4
[83526.172312] siwifi_calculate_legrate invalid legrate: 4
[83526.196146] siwifi_calculate_legrate invalid legrate: 4
[83532.817995] [0]Total force ps drop len : 1
[83535.302121] siwifi_calculate_legrate invalid legrate: 4
[83535.384561] siwifi_calculate_legrate invalid legrate: 4
[83535.407858] siwifi_calculate_legrate invalid legrate: 4
[83535.445887] siwifi_calculate_legrate invalid legrate: 4
[83535.485066] siwifi_calculate_legrate invalid legrate: 4
[83535.507659] siwifi_calculate_legrate invalid legrate: 4
[83535.530693] siwifi_calculate_legrate invalid legrate: 4
[83535.545977] siwifi_calculate_legrate invalid legrate: 4
[83535.567669] siwifi_calculate_legrate invalid legrate: 4
[83535.590785] siwifi_calculate_legrate invalid legrate: 4
[83535.614194] siwifi_calculate_legrate invalid legrate: 4
[83535.636071] siwifi_calculate_legrate invalid legrate: 4
[83535.659048] siwifi_calculate_legrate invalid legrate: 4
[83535.682074] siwifi_calculate_legrate invalid legrate: 4
[83535.704043] siwifi_calculate_legrate invalid legrate: 4
[83535.726649] siwifi_calculate_legrate invalid legrate: 4
[83535.792298] siwifi_calculate_legrate invalid legrate: 4
[83535.965295] siwifi_calculate_legrate invalid legrate: 4
[83535.988557] siwifi_calculate_legrate invalid legrate: 4
[83536.100847] siwifi_calculate_legrate invalid legrate: 4
[83536.124987] siwifi_calculate_legrate invalid legrate: 4
[83536.147517] siwifi_calculate_legrate invalid legrate: 4
[83536.170515] siwifi_calculate_legrate invalid legrate: 4
[83536.193305] siwifi_calculate_legrate invalid legrate: 4
[83536.217924] siwifi_calculate_legrate invalid legrate: 4
[83536.241284] siwifi_calculate_legrate invalid legrate: 4
[83554.368701] wait cmd timeout, tout : 6000(ms), defer_push : 0!
[83554.374661] tkn[1409258]  flags:0008  result: -4  cmd: 119-unknown                  - reqcfm(   0-MM_RESET_REQ)
[83554.385095] tkn[1409258]  flags:0008  result: -4  cmd: 119-unknown                  - reqcfm(   0-MM_RESET_REQ)
[83554.395635] tkn[1409259]  flags:001f  result:  0  cmd:5132-ME_TRAFFIC_IND_REQ       - reqcfm(5133-ME_TRAFFIC_IND_CFM)
[83554.406747] tkn[1409260]  flags:001e  result: -4  cmd:  63-MM_BCN_CHANGE_REQ        - reqcfm(  64-MM_BCN_CHANGE_CFM)
[83554.417656] tkn[1409261]  flags:001f  result:  0  cmd:5132-ME_TRAFFIC_IND_REQ       - reqcfm(5133-ME_TRAFFIC_IND_CFM)
[83554.428456] [lb]gic mask : 0x5, pending : 0x8
[83554.432933] ipc_env->reg_base : b1000000
[83554.437098] a2e status : 0x202, raw staus : 0x202, mask : 0x1f03, trigger(e2a) : 0x0
[83554.445245] [by API]e2a status : 0x0, raw staus : 0x0, mask : 0xfff, trigger(a2e) : 0x202
[83554.453792] siwifi_hw->dbgdump_elem.err_cur_idx 0,0,848ab500,54,1
[83554.460362] set max idx to LMAC failed!
[83555.408978] [1]Total force ps drop len : 1
[83557.447723] [1]Total force ps drop len : 7
[83558.208082] cmd queue[0] crashed
[83558.211422] siwifi_hw->dbgdump_elem.err_cur_idx 1,1,85f96e80,-20,1
[83569.088042] wait cmd timeout, tout : 18000(ms), defer_push : 1!
[83569.094104] tkn[1409260]  flags:001e  result: -4  cmd:  63-MM_BCN_CHANGE_REQ        - reqcfm(  64-MM_BCN_CHANGE_CFM)
[83569.104854] tkn[1409259]  flags:001f  result:  0  cmd:5132-ME_TRAFFIC_IND_REQ       - reqcfm(5133-ME_TRAFFIC_IND_CFM)
[83569.115570] tkn[1409260]  flags:001e  result: -4  cmd:  63-MM_BCN_CHANGE_REQ        - reqcfm(  64-MM_BCN_CHANGE_CFM)
[83569.126293] tkn[1409261]  flags:001f  result:  0  cmd:5132-ME_TRAFFIC_IND_REQ       - reqcfm(5133-ME_TRAFFIC_IND_CFM)
[83569.136992] [lb]gic mask : 0x5, pending : 0x8
[83569.141530] ipc_env->reg_base : b1000000
[83569.145530] a2e status : 0x202, raw staus : 0x202, mask : 0x1f03, trigger(e2a) : 0x0
[83569.153371] [by API]e2a status : 0x0, raw staus : 0x0, mask : 0xfff, trigger(a2e) : 0x202
[83569.161771] siwifi_hw->dbgdump_elem.err_cur_idx 2,2,86db4100,20,1
[83569.169148] cmd queue[0] crashed
[83569.172561] siwifi_hw->dbgdump_elem.err_cur_idx 3,3,85f96980,-20,1
[83569.180819] cmd queue[0] crashed
[83569.184295] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83569.191930] cmd queue[0] crashed
[83569.198185] cmd queue[0] crashed
[83569.201518] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83569.219352] cmd queue[0] crashed
[83569.229368] cmd queue[0] crashed
[83569.233665] cmd queue[0] crashed
[83569.237083] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83569.253740] cmd queue[0] crashed
[83569.257196] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83569.282544] cmd queue[0] crashed
[83569.286450] cmd queue[0] crashed
[83569.289958] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83569.304014] cmd queue[0] crashed
[83569.307330] siwifi_hw->dbgdump_elem.err_cur_idx 4,4,85f96280,0,10
[83569.313717] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83570.354058] cmd queue[0] crashed
[83570.357598] cmd queue[0] crashed
[83570.361007] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83570.372011] cmd queue[0] crashed
[83570.375291] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.415985] cmd queue[0] crashed
[83571.419878] cmd queue[0] crashed
[83571.423139] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.434016] cmd queue[0] crashed
[83571.437301] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.453977] cmd queue[0] crashed
[83571.457684] cmd queue[0] crashed
[83571.461069] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.472077] cmd queue[0] crashed
[83571.475482] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.491523] cmd queue[0] crashed
[83571.494805] siwifi_hw->dbgdump_elem.err_cur_idx 5,5,8480a400,0,10
[83571.501517] cmd queue[0] crashed
[83571.504790] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.515547] cmd queue[0] crashed
[83571.518934] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83571.538471] cmd queue[0] crashed
[83572.560522] cmd queue[0] crashed
[83572.564073] cmd queue[0] crashed
[83572.567313] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.579838] cmd queue[0] crashed
[83572.583120] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.599520] cmd queue[0] crashed
[83572.603079] cmd queue[0] crashed
[83572.606320] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.617213] cmd queue[0] crashed
[83572.620714] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.636549] cmd queue[0] crashed
[83572.639948] siwifi_hw->dbgdump_elem.err_cur_idx 6,6,85740c00,0,10
[83572.646529] cmd queue[0] crashed
[83572.650002] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.661319] cmd queue[0] crashed
[83572.664600] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.681432] cmd queue[0] crashed
[83572.685125] cmd queue[0] crashed
[83572.688679] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.699902] cmd queue[0] crashed
[83572.703183] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.719382] cmd queue[0] crashed
[83572.723296] cmd queue[0] crashed
[83572.726562] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.737638] cmd queue[0] crashed
[83572.741057] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.757324] cmd queue[0] crashed
[83572.761218] cmd queue[0] crashed
[83572.764486] siwifi_hw->dbgdump_elem.err_cur_idx 7,7,85ea3d00,0,10
[83572.770860] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.782420] cmd queue[0] crashed
[83572.785718] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.801949] cmd queue[0] crashed
[83572.805494] cmd queue[0] crashed
[83572.809087] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.820097] cmd queue[0] crashed
[83572.823385] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.839476] cmd queue[0] crashed
[83572.843168] cmd queue[0] crashed
[83572.846423] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.857741] cmd queue[0] crashed
[83572.861177] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.877480] cmd queue[0] crashed
[83572.881439] cmd queue[0] crashed
[83572.884919] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.896080] cmd queue[0] crashed
[83572.899503] siwifi_hw->dbgdump_elem.err_cur_idx 8,8,85740400,0,10
[83572.905634] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.965149] cmd queue[0] crashed
[83572.969023] cmd queue[0] crashed
[83572.972300] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83572.983531] cmd queue[0] crashed
[83572.987034] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83573.567983] cmd queue[0] crashed
[83573.571359] lmac_dump_registers band=0
[83573.575119] umac task_dump_registers-baseaddr=0x81ffd000 backtrace=0x81ffcc00 gpr=0x81ffce00 cpu=0x81ffcf00
[83573.585022] 
[83573.585022] $ 0:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585022] $ 4:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585022] $ 8:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585022] $12:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585022] $16:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585022] $20:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585022] $24:	ffffffff	ffffffff	
[83573.585022] $28:	ffffffff	ffffffff	ffffffff	ffffffff
[83573.585185] epc: ffffffff
[83573.628438] ra: ffffffff
[83573.630977] Status: ffffffff
[83573.633903] Cause: ffffffff
[83573.636728] BadVA: ffffffff
[83573.639670] Prid: ffffffff
[83573.642443] Call Trace:
[83573.644897] [<0xffffffff>] <-
[83573.647934] [<0xffffffff>] <-
[83573.651013] [<0xffffffff>] <-
[83573.654023] [<0xffffffff>] <-
[83573.657028] [<0xffffffff>] <-
[83573.660173] [<0xffffffff>] <-
[83573.663215] [<0xffffffff>] <-
[83573.666186] [<0xffffffff>] <-
[83573.669254] [<0xffffffff>] <-
[83573.672294] [<0xffffffff>] <-
[83573.675262] [<0xffffffff>] <-
[83573.678308] [<0xffffffff>] <-
[83573.681278] [<0xffffffff>] <-
[83573.684241] [<0xffffffff>] <-
[83573.687205] [<0xffffffff>] <-
[83573.690246] [<0xffffffff>] <-
[83573.693217] [<0xffffffff>] <-
[83573.696186] [<0xffffffff>] <-
[83573.699415] [<0xffffffff>] <-
[83573.702398] [<0xffffffff>] <-
[83573.705367] [<0xffffffff>] <-
[83573.708461] [<0xffffffff>] <-
[83573.711538] [<0xffffffff>] <-
[83573.714582] [<0xffffffff>] <-
[83573.717582] [<0xffffffff>] <-
[83573.720675] [<0xffffffff>] <-
[83573.723712] [<0xffffffff>] <-
[83573.726681] [<0xffffffff>] <-
[83573.729744] [<0xffffffff>] <-
[83573.732783] [<0xffffffff>] <-
[83573.735752] [<0xffffffff>] <-
[83573.738796] [<0xffffffff>] <-
[83573.741767] [<0xffffffff>] <-
[83573.744731] [<0xffffffff>] <-
[83573.747774] [<0xffffffff>] <-
[83573.750965] [<0xffffffff>] <-
[83573.753945] [<0xffffffff>] <-
[83573.756941] [<0xffffffff>] <-
[83573.759987] [<0xffffffff>] <-
[83573.762959] [<0xffffffff>] <-
[83573.765923] [<0xffffffff>] <-
[83573.768955] [<0xffffffff>] <-
[83573.772011] [<0xffffffff>] <-
[83573.774980] [<0xffffffff>] <-
[83573.778032] [<0xffffffff>] <-
[83573.781070] [<0xffffffff>] <-
[83573.784088] [<0xffffffff>] <-
[83573.787056] [<0xffffffff>] <-
[83573.790193] [<0xffffffff>] <-
[83573.793171] [<0xffffffff>] <-
[83573.796136] [<0xffffffff>] <-
[83573.799218] [<0xffffffff>] <-
[83573.802206] [<0xffffffff>] <-
[83573.805169] [<0xffffffff>] <-
[83573.808261] [<0xffffffff>] <-
[83573.811364] [<0xffffffff>] <-
[83573.814348] [<0xffffffff>] <-
[83573.817314] [<0xffffffff>] <-
[83573.820377] [<0xffffffff>] <-
[83573.823358] [<0xffffffff>] <-
[83573.826324] [<0xffffffff>] <-
[83573.829373] [<0xffffffff>] <-
[83573.832355] [<0xffffffff>] <-
[83573.835322] [<0xffffffff>] <-
[83573.838347] [<0xffffffff>] <-
[83573.841324] [<0xffffffff>] <-
[83573.844290] [<0xffffffff>] <-
[83573.847256] [<0xffffffff>] <-
[83573.850342] [<0xffffffff>] <-
[83573.853326] [<0xffffffff>] <-
[83573.856290] [<0xffffffff>] <-
[83573.859335] [<0xffffffff>] <-
[83573.862314] [<0xffffffff>] <-
[83573.865278] [<0xffffffff>] <-
[83573.868308] [<0xffffffff>] <-
[83573.871296] [<0xffffffff>] <-
[83573.874265] [<0xffffffff>] <-
[83573.877231] [<0xffffffff>] <-
[83573.880315] [<0xffffffff>] <-
[83573.883306] [<0xffffffff>] <-
[83573.886271] [<0xffffffff>] <-
[83573.889319] [<0xffffffff>] <-
[83573.892300] [<0xffffffff>] <-
[83573.895264] [<0xffffffff>] <-
[83573.898293] [<0xffffffff>] <-
[83573.901284] [<0xffffffff>] <-
[83573.904254] [<0xffffffff>] <-
[83573.907218] [<0xffffffff>] <-
[83573.910344] [<0xffffffff>] <-
[83573.913336] [<0xffffffff>] <-
[83573.916300] [<0xffffffff>] <-
[83573.919347] [<0xffffffff>] <-
[83573.922326] [<0xffffffff>] <-
[83573.925291] [<0xffffffff>] <-
[83573.928323] [<0xffffffff>] <-
[83573.931302] [<0xffffffff>] <-
[83573.934268] [<0xffffffff>] <-
[83573.937232] [<0xffffffff>] <-
[83573.940274] [<0xffffffff>] <-
[83573.943252] [<0xffffffff>] <-
[83573.946218] [<0xffffffff>] <-
[83573.949256] [<0xffffffff>] <-
[83573.952244] [<0xffffffff>] <-
[83573.955209] [<0xffffffff>] <-
[83573.958235] [<0xffffffff>] <-
[83573.961211] [<0xffffffff>] <-
[83573.964176] [<0xffffffff>] <-
[83573.967140] [<0xffffffff>] <-
[83573.970201] [<0xffffffff>] <-
[83573.973184] [<0xffffffff>] <-
[83573.976148] [<0xffffffff>] <-
[83573.979192] [<0xffffffff>] <-
[83573.982171] [<0xffffffff>] <-
[83573.985135] [<0xffffffff>] <-
[83573.988163] [<0xffffffff>] <-
[83573.991147] [<0xffffffff>] <-
[83573.994111] [<0xffffffff>] <-
[83573.997075] [<0xffffffff>] <-
[83574.000114] [<0xffffffff>] <-
[83574.003104] [<0xffffffff>] <-
[83574.006075] [<0xffffffff>] <-
[83574.009178] [<0xffffffff>] <-
[83574.012264] [<0xffffffff>] <-
[83574.015337] [<0xffffffff>] <-
[83574.018518] [<0xffffffff>] <-
[83574.021660] [<0xffffffff>] <-
[83574.024777] [<0xffffffff>] <-
[83574.027969] [<0xffffffff>] <-
[83574.031086] [<0xffffffff>] <-
[83574.034136] [<0xffffffff>] <-
[83574.035127] cmd queue[0] crashed
[83574.037209] [<0xffffffff>] <-
[83574.042882] cmd queue[0] crashed
[83574.043563] [<0xffffffff>] <-
[83574.043569] [<0xffffffff>] <-
[83574.043573] [<0xffffffff>] <-
[83574.043577] [<0xffffffff>] <-
[83574.043581] [<0xffffffff>] <-
[83574.043585] [<0xffffffff>] <-
[83574.043589] [<0xffffffff>] <-
[83574.043592] [<0xffffffff>] <-
[83574.043596] [<0xffffffff>] <-
[83574.043600] [<0xffffffff>] <-
[83574.043604] [<0xffffffff>] <-
[83574.043608] [<0xffffffff>] <-
[83574.043612] [<0xffffffff>] <-
[83574.043616] [<0xffffffff>] <-
[83574.043640] [<0xffffffff>] <-
[83574.046897] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.054330] cmd queue[0] crashed
[83574.056125] [<0xffffffff>] <-
[83574.059218] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.062186] [<0xffffffff>] <-
[83574.075113] cmd queue[0] crashed
[83574.077077] [<0xffffffff>] <-
[83574.077082] [<0xffffffff>] <-
[83574.077086] [<0xffffffff>] <-
[83574.077089] [<0xffffffff>] <-
[83574.077094] [<0xffffffff>] <-
[83574.077098] [<0xffffffff>] <-
[83574.077101] [<0xffffffff>] <-
[83574.077105] [<0xffffffff>] <-
[83574.077109] [<0xffffffff>] <-
[83574.077113] [<0xffffffff>] <-
[83574.077139] [<0xffffffff>] <-
[83574.081421] cmd queue[0] crashed
[83574.083194] [<0xffffffff>] <-
[83574.083199] [<0xffffffff>] <-
[83574.083203] [<0xffffffff>] <-
[83574.083207] [<0xffffffff>] <-
[83574.083211] [<0xffffffff>] <-
[83574.083215] [<0xffffffff>] <-
[83574.083219] [<0xffffffff>] <-
[83574.083222] [<0xffffffff>] <-
[83574.083226] [<0xffffffff>] <-
[83574.083231] [<0xffffffff>] <-
[83574.083252] [<0xffffffff>] <-
[83574.086286] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.089273] [<0xffffffff>] <-
[83574.089293] [<0xffffffff>] <-
[83574.096891] cmd queue[0] crashed
[83574.098374] [<0xffffffff>] <-
[83574.101625] siwifi_hw->dbgdump_elem.err_cur_idx 9,9,85740f00,0,10
[83574.104564] [<0xffffffff>] <-
[83574.110842] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.113607] [<0xffffffff>] <-
[83574.113615] [<0xffffffff>] <-
[83574.113619] [<0xffffffff>] <-
[83574.113623] [<0xffffffff>] <-
[83574.113626] [<0xffffffff>] <-
[83574.113630] [<0xffffffff>] <-
[83574.113634] [<0xffffffff>] <-
[83574.113638] [<0xffffffff>] <-
[83574.113642] [<0xffffffff>] <-
[83574.113646] [<0xffffffff>] <-
[83574.113650] [<0xffffffff>] <-
[83574.113654] [<0xffffffff>] <-
[83574.113657] [<0xffffffff>] <-
[83574.113661] [<0xffffffff>] <-
[83574.113665] [<0xffffffff>] <-
[83574.113669] [<0xffffffff>] <-
[83574.113673] [<0xffffffff>] <-
[83574.113677] [<0xffffffff>] <-
[83574.113681] [<0xffffffff>] <-
[83574.113685] [<0xffffffff>] <-
[83574.113689] [<0xffffffff>] <-
[83574.113693] [<0xffffffff>] <-
[83574.113696] [<0xffffffff>] <-
[83574.113700] [<0xffffffff>] <-
[83574.113704] [<0xffffffff>] <-
[83574.113708] [<0xffffffff>] <-
[83574.113712] [<0xffffffff>] <-
[83574.113716] [<0xffffffff>] <-
[83574.113720] [<0xffffffff>] <-
[83574.113723] [<0xffffffff>] <-
[83574.113727] [<0xffffffff>] <-
[83574.113731] [<0xffffffff>] <-
[83574.113735] [<0xffffffff>] <-
[83574.113739] [<0xffffffff>] <-
[83574.113743] [<0xffffffff>] <-
[83574.113747] [<0xffffffff>] <-
[83574.113751] [<0xffffffff>] <-
[83574.113754] [<0xffffffff>] <-
[83574.113759] [<0xffffffff>] <-
[83574.113763] [<0xffffffff>] <-
[83574.113766] [<0xffffffff>] <-
[83574.113770] [<0xffffffff>] <-
[83574.113774] [<0xffffffff>] <-
[83574.113778] [<0xffffffff>] <-
[83574.113782] [<0xffffffff>] <-
[83574.113813] [<0xffffffff>] <-
[83574.207085] cmd queue[0] crashed
[83574.211017] [<0xffffffff>] <-
[83574.215017] cmd queue[0] crashed
[83574.220084] [<0xffffffff>] <-
[83574.220090] [<0xffffffff>] <-
[83574.220116] [<0xffffffff>] <-
[83574.223285] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.226046] [<0xffffffff>] <-
[83574.226064] [<0xffffffff>] <-
[83574.233850] cmd queue[0] crashed
[83574.235056] [<0xffffffff>] <-
[83574.238201] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.241078] [<0xffffffff>] <-
[83574.254005] cmd queue[0] crashed
[83574.256116] [<0xffffffff>] <-
[83574.259718] cmd queue[0] crashed
[83574.262157] [<0xffffffff>] <-
[83574.265128] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.268155] [<0xffffffff>] <-
[83574.268170] [<0xffffffff>] <-
[83574.277096] cmd queue[0] crashed
[83574.277221] [<0xffffffff>] <-
[83574.280353] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.283370] [<0xffffffff>] <-
[83574.298735] cmd queue[0] crashed
[83574.299077] cmd queue[0] crashed
[83574.301735] [<0xffffffff>] <-
[83574.304697] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.307653] [<0xffffffff>] <-
[83574.315801] cmd queue[0] crashed
[83574.316644] [<0xffffffff>] <-
[83574.319964] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.322711] [<0xffffffff>] <-
[83574.335783] cmd queue[0] crashed
[83574.337691] [<0xffffffff>] <-
[83574.340732] siwifi_hw->dbgdump_elem.err_cur_idx 10,10,85ea3800,0,10
[83574.341315] cmd queue[0] crashed
[83574.343741] [<0xffffffff>] <-
[83574.346711] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.349744] [<0xffffffff>] <-
[83574.349763] [<0xffffffff>] <-
[83574.357336] cmd queue[0] crashed
[83574.358762] [<0xffffffff>] <-
[83574.362015] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.364950] [<0xffffffff>] <-
[83574.378728] cmd queue[0] crashed
[83574.383273] [<0xffffffff>] <-
[83574.386930] cmd queue[0] crashed
[83574.389181] [<0xffffffff>] <-
[83574.392429] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.395370] [<0xffffffff>] <-
[83574.406070] cmd queue[0] crashed
[83574.407633] [<0xffffffff>] <-
[83574.410707] siwifi_send_dbg_get_vendor_mp_info_req send msg fail
[83574.413911] [<0xffffffff>] <-
[83574.413916] [<0xffffffff>] <-
[83574.413919] [<0xffffffff>] <-
[83574.413923] [<0xffffffff>] <-
[83574.413927] [<0xffffffff>] <-
[83574.413931] [<0xffffffff>] <-
[83574.413935] [<0xffffffff>] <-
[83574.413939] [<0xffffffff>] <-
[83574.413943] [<0x3c1a8200>] <-
[83574.413947] [<0x275ad000>] <-
[83574.413950] [<0x3c1b2000>] <-
[83574.413954] [<0x35bd025>] <-
[83574.413958] [<0x241b01fc>] <-
[83574.413962] [<0x35bd023>] <-
[83574.413966] [<0xaf40fffc>] <-
[83574.413970] [<0x10000001>] <-
[83574.413990] heart_beat  restart over. count 1
[83574.610986] lb-fmac 11000000.wifi-lb wlan1: restart Del sta 1 (84:f7:03:2d:e8:28)
[83574.618647] remove STA 84:f7:03:2d:e8:28 from hash table success
[83574.624684] lb-fmac 11000000.wifi-lb wlan1: restart Del sta 3 (64:16:66:22:39:4f)
[83574.632315] remove STA 64:16:66:22:39:4f from hash table success
[83574.638442] lb-fmac 11000000.wifi-lb wlan1: restart Del sta 9 (84:f7:03:2d:db:dc)
[83574.645971] remove STA 84:f7:03:2d:db:dc from hash table success
[83574.652062] lb-fmac 11000000.wifi-lb wlan1: restart Del sta 29 (48:7e:48:41:54:4c)
[83574.659914] remove STA 48:7e:48:41:54:4c from hash table success
[83574.665955] lb-fmac 11000000.wifi-lb wlan1: restart Del sta 31 (c4:23:60:cb:69:9d)
[83574.673777] remove STA c4:23:60:cb:69:9d from hash table success
[83574.680207] lmac_glue_stop(0)
[83574.683246] stop_task, 0
[83574.685816] stop aresetn 2 por_resetn 1 
[83574.690126] tkn[1409259]  flags:001f  result:  0  cmd:5132-ME_TRAFFIC_IND_REQ       - reqcfm(5133-ME_TRAFFIC_IND_CFM)
[83574.700954] tkn[1409261]  flags:001f  result:  0  cmd:5132-ME_TRAFFIC_IND_REQ       - reqcfm(5133-ME_TRAFFIC_IND_CFM)
[83574.713865] platform shutdown : 0
[83574.721278] tx headroom : 224, tx amsdu headroom : 50, rx headroom : 182, rx pd offset 224
[83574.732931] lmac_glue_start(0)
[83574.736027] start_task, 0
[83574.738882] task entry_addr=0x1f00000
[83574.742554] start aresetn 0 por_resetn 0 
[83574.746563] wait lmac init(0)>>>>>>>>>>>>>>>>>>>>>>>
[83602.291219] [1]Total force ps drop len : 1
[83703.479343] [1]Total force ps drop len : 1
[83763.263995] [1]Total force ps drop len : 1
[83798.426895] [1]Total force ps drop len : 7
[83851.959310] [1]Total force ps drop len : 1
[83895.277388] [1]Total force ps drop len : 7
[84004.125709] [1]Total force ps drop len : 1
[84045.273629] [1]Total force ps drop len : 1
[84145.852213] [1]Total force ps drop len : 7
[84150.352905] [1]Total force ps drop len : 1
[84189.435055] [1]Total force ps drop len : 7
[84237.315687] [1]Total force ps drop len : 7
[84282.834246] [1]Total force ps drop len : 7
[84299.242527] [1]Total force ps drop len : 1
[84320.719954] [1]Total force ps drop len : 7
[84363.403597] [1]Total force ps drop len : 7
[84436.345670] [1]Total force ps drop len : 7
[84448.132112] [1]Total force ps drop len : 1
[84488.788042] [1]Total force ps drop len : 7
[84537.747003] [1]Total force ps drop len : 7
[84597.636106] [1]Total force ps drop len : 1
[84628.565168] [1]Total force ps drop len : 7
[84676.500910] [1]Total force ps drop len : 7
[84729.636085] [1]Total force ps drop len : 7
[84746.116027] [1]Total force ps drop len : 1
[84770.158644] [1]Total force ps drop len : 7
[84807.848803] [1]Total force ps drop len : 7
[84855.893414] [1]Total force ps drop len : 7
[84895.620137] [1]Total force ps drop len : 1
[84912.213069] [1]Total force ps drop len : 7
[84971.925392] [1]Total force ps drop len : 7
[85045.124175] [1]Total force ps drop len : 1
[85116.814867] [1]Total force ps drop len : 7
[85194.218514] [1]Total force ps drop len : 1
[85206.797200] [1]Total force ps drop len : 7
[85214.159714] [1]Total force ps drop len : 1
[85307.475929] [1]Total force ps drop len : 7
[85342.698523] [1]Total force ps drop len : 1
[85359.437581] [1]Total force ps drop len : 1
[85391.202974] [1]Total force ps drop len : 7
[85402.076161] [1]Total force ps drop len : 1
[85452.647374] [1]Total force ps drop len : 7
[85492.202545] [1]Total force ps drop len : 1
[85501.933390] [1]Total force ps drop len : 1
[85517.256010] [1]Total force ps drop len : 7
[85542.515278] [1]Total force ps drop len : 1
[85563.276040] [1]Total force ps drop len : 7
[85602.296705] [1]Total force ps drop len : 7
[85625.150282] [1]Total force ps drop len : 1
[85641.707021] [1]Total force ps drop len : 1
[85666.382868] [1]Total force ps drop len : 1
[85690.423305] [1]Total force ps drop len : 7
[85742.280822] [1]Total force ps drop len : 7
[85790.596178] [1]Total force ps drop len : 1
[85800.973568] [1]Total force ps drop len : 1
[85824.239399] [1]Total force ps drop len : 7
[85867.174423] [1]Total force ps drop len : 1
[85915.936616] [1]Total force ps drop len : 7
[85938.871463] [1]Total force ps drop len : 1
[85955.977704] [1]Total force ps drop len : 7
[85989.849623] [1]Total force ps drop len : 1
[86007.440261] [1]Total force ps drop len : 7
[86044.755811] [1]Total force ps drop len : 1
[86061.013198] [1]Total force ps drop len : 7
[86079.659457] [1]Total force ps drop len : 1
[86088.170607] [1]Total force ps drop len : 1
[86103.205002] [1]Total force ps drop len : 7
[86116.345891] [1]Total force ps drop len : 1
[86142.155050] [1]Total force ps drop len : 7
[86236.036171] [1]Total force ps drop len : 1
[86281.095019] [1]Total force ps drop len : 3
[86374.572757] [1]Total force ps drop len : 7
[86379.258471] [1]Total force ps drop len : 3
[86384.106641] [1]Total force ps drop len : 1
[86429.357712] [1]Total force ps drop len : 3
[86441.664379] [1]Total force ps drop len : 7
[86467.008322] [1]Total force ps drop len : 3
[86532.996048] [1]Total force ps drop len : 1
[86682.500137] [1]Total force ps drop len : 1
[86831.184919] [1]Total force ps drop len : 1
[86885.030408] [1]Total force ps drop len : 1
[86919.232027] [1]Total force ps drop len : 1
[86980.688895] [1]Total force ps drop len : 1
[87128.554532] [1]Total force ps drop len : 1
[87276.624903] [1]Total force ps drop len : 1
[87458.060814] [1]Total force ps drop len : 1
[87574.199311] [1]Total force ps drop len : 1
[87723.703305] [1]Total force ps drop len : 1
[87873.207879] [1]Total force ps drop len : 1
[88024.145135] [1]Total force ps drop len : 1
[88169.962691] [1]Total force ps drop len : 1
[88321.515536] [1]Total force ps drop len : 1
[88468.970505] [1]Total force ps drop len : 1
[88617.450668] [1]Total force ps drop len : 1
[88766.954500] [1]Total force ps drop len : 1
[88914.557960] [1]Total force ps drop len : 1
[88953.575779] [1]Total force ps drop len : 1
[89064.324184] [1]Total force ps drop len : 1
[89213.828095] [1]Total force ps drop len : 1
[89363.332105] [1]Total force ps drop len : 1
[89512.836114] [1]Total force ps drop len : 1
[89661.520939] [1]Total force ps drop len : 1
[89810.615347] [1]Total force ps drop len : 1
[89960.120442] [1]Total force ps drop len : 1
[90109.623724] [1]Total force ps drop len : 1
[90220.455970] [1]Total force ps drop len : 3
[90257.693807] [1]Total force ps drop len : 1
[90405.968922] [1]Total force ps drop len : 1
[90554.449005] [1]Total force ps drop len : 1
[90559.599306] [1]Total force ps drop len : 3
[90612.800070] [1]Total force ps drop len : 1
[90700.688325] [1]Total force ps drop len : 3
[90702.928939] [1]Total force ps drop len : 1
[90736.462311] [1]Total force ps drop len : 3
[90850.795379] [1]Total force ps drop len : 1
[90949.491319] [1]Total force ps drop len : 1
[90998.660129] [1]Total force ps drop len : 1
[91161.254405] [1]Total force ps drop len : 1
[91296.439303] [1]Total force ps drop len : 1
[91445.124584] [1]Total force ps drop len : 1
[91594.628106] [1]Total force ps drop len : 1
[91744.132527] [1]Total force ps drop len : 1
[91891.998223] [1]Total force ps drop len : 1
[92046.604810] [1]Total force ps drop len : 1
[92190.596393] [1]Total force ps drop len : 1
[92287.244998] [1]Total force ps drop len : 1
[92338.462290] [1]Total force ps drop len : 1
[92490.013930] [1]Total force ps drop len : 1
[92635.569130] [1]Total force ps drop len : 1
[92785.540489] [1]Total force ps drop len : 1
[92875.430474] [1]Total force ps drop len : 1
[92934.839219] [1]Total force ps drop len : 1
[93083.934338] [1]Total force ps drop len : 1
[93263.526399] [1]Total force ps drop len : 1
[93380.279313] [1]Total force ps drop len : 1
[93529.783305] [1]Total force ps drop len : 1
[93679.287443] [1]Total force ps drop len : 1
[93828.586506] [1]Total force ps drop len : 1
[93877.516911] [1]Total force ps drop len : 1
[93914.176260] [1]Total force ps drop len : 1
[93977.168945] [1]Total force ps drop len : 1
[94000.664200] [1]Total force ps drop len : 3
[94110.375232] rx pkt(1) disassoc from [90:81:58:a0:9d:e6] reasoncode: 8
[94128.618528] [1]Total force ps drop len : 1
[94275.460683] [1]Total force ps drop len : 1
[94328.281627] [1]Total force ps drop len : 1
[94424.964153] [1]Total force ps drop len : 1