GL-MT3000 / Beryl AX Connection Issues / Intermittent Dropping

conker · December 5, 2024, 5:56pm

Hi,
I don't even know how to really describe these weird issues

Basically originally(3-4 weeks ago) it just lost the wifi connection and I couldn't connect at all for a while, until it started working again after a few minutes.

Now it seems the wifi was still connected to the laptop, but in repeater mode, it was blinking as it lost the (2.4G) to the upstream wifi router.
But I was also not even able to connect to the web interface, nor to the ssh.
I had to connect a lan cable to get ssh access, and usually when I bothered around that time it started working again.
Tho everything is rather wonky at this point.

I just got 2 dmesg logs exported via ssh.
Tho it is really annoying that this dmesg version doesnt support -T parameter, so I can't correlate the times easily.
Like HOW ANCIENT is this dmesg version ... ?
(And from the last time I think I have the dmesg excerpts somewhere...)

Some other details:

It happened both

3-4 weeks ago when I first noticed it, I was tethering (connected via usb hub + had a usb hdd connected and mounted).
Now, when no usb at all was connected.

It's using the latest fw. Admin Panel says it's currently the v4.6.9
The device has an uptime of around 26 days.

So my questions are

are there any other things/logs to export to properly understand what the issue is ?
is it worth bothering or should I just try the latest openwrt instead, that I wanted to test anyway.

buntspext · December 5, 2024, 8:10pm

Are you using a powered or unpowered hub? Your problem might be related with USB power demands, because when no usb devices are connected it all works.

conker · December 6, 2024, 11:39am

Well initially I was using the (unpowered) hub that had the phone plugged in for tethering, that might have been a bit of a stretch but it worked.
And the ext 3.5" HDD was ofc separately powered.

I did some tests and adding another usb thumb drive did reset the whole hub, but even then it worked again with all 3 devices plugged in.
So I think the phone alone should be also fine (even via the hub).

But that is not the cause of the actual issue,
as I said "* Now, when no usb at all was connected." , so that is how I rule out that it was due to usb power demands/issues.
I still got this type of disconnect/drop even with USB not used at all.
So my conclusion is, it has to be something else (maybe even related to the wifi driver). I will have to find the previous dmesg excerpts and compare them to the ones from yesterday.
There were a lot of weird wifi fifo buffer whatever entries/errors.

conker · December 7, 2024, 7:19pm

Ok, just quick update, as it happened AGAIN.

I did copy the 3rd dmesg log from the router, I just don't want to update it for privacy reasons.
Also I'm still rather annoyed that the dmesg -T option is missing, that makes pinpointing the exact time unnecessarily difficult.
Anyway, it these last 2 logs it happened right before I called dmesg, and here this is what happens / some weird log lines that only occur during the issue:


[2423115.037395] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 242580, Old_FRxFifoFullCount = 235381, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 255476, Old_FRxFifoFullCount = 254492, RxMpduCount 316988787981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 257982, Old_FRxFifoFullCount = 242580, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 257398, Old_FRxFifoFullCount = 255476, RxMpduCount 316988787981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 274795, Old_FRxFifoFullCount = 257982, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 258371, Old_FRxFifoFullCount = 257398, RxMpduCount 316988787981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 290207, Old_FRxFifoFullCount = 274795, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 258509, Old_FRxFifoFullCount = 258371, RxMpduCount 31698878
[2423175.286343] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 305553, Old_FRxFifoFullCount = 290207, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 259092, Old_FRxFifoFullCount = 258509, RxMpduCount 316988787981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 320645, Old_FRxFifoFullCount = 305553, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 259644, Old_FRxFifoFullCount = 259092, RxMpduCount 316988787981@C13L1,tx_free_v3_notify_handler() 3530: ContTxFailCntTotal = 1, ContTxFailCnt300ms = 1
[2423201.459852] 7981@C13L1,tx_free_v3_notify_handler() 3533: token used by current wcid = 8130, free_token_cnt = 2303
[2423205.454829] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 335560, Old_FRxFifoFullCount = 320645, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 350365, Old_FRxFifoFullCount = 335560, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 364859, Old_FRxFifoFullCount = 350365, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 379406, Old_FRxFifoFullCount = 364859, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 394354, Old_FRxFifoFullCount = 379406, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 260271, Old_FRxFifoFullCount = 259723, RxMpduCount 316988787981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 409260, Old_FRxFifoFullCount = 394354, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 423851, Old_FRxFifoFullCount = 409260, RxMpduCount 52044232
[2423304.518100] WiFi@C15L1,RTMPDeletePMKIDCache() 1311: IF(2), del PMKID CacheIdx=0
[2423309.764792] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 438428, Old_FRxFifoFullCount = 423851, RxMpduCount 520442327981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 453056, Old_FRxFifoFullCount = 438428, RxMpduCount 52044232
[2423326.184750] mtk_soc_eth 15100000.ethernet eth1: Link is Up - 100Mbps/Full - flow control rx/tx
[2423326.216867] br-lan: port 1(eth1) entered blocking state
[2423326.222272] br-lan: port 1(eth1) entered forwarding state

Sometimes only 1-2 lines, sometimes more like here, and after that the eth1 port is enabled again, which I also noticed, that during the issue I wasn't even able to login even with lan cable connection.

For me it looks like some rather serious driver issue/bug.

conker · December 12, 2024, 12:11pm

I tried to do some more research but google only found this other post

It has the same mt7981_hw_auto_debug() 12144: RxFifoFullCount ... error at the end, but I don't remember if my log had the AndesDequeueAndKickOutCmdMsgs() 683: kick out msg fail, too.
I will check later and update this post.

conker · December 14, 2024, 7:09am

Finally the UI showed me the 4.7.0 notification and I updated yesterday.
I'll continue to monitor the situation.

conker · January 13, 2025, 5:08pm

I'll give a short update:
It seems this particular wifi issue might have been solved with the 4.7.0 fw.
Iirc I didn't encounter this particular issue anymore for the last month or so (after upgrading to 4.7.0).

conker · February 9, 2025, 6:32pm

Sadly I'm starting to hate this stupid thing. Initially I really liked the router, esp. the concept of it (such a small pocket device) but now I'm really starting to regret buying it.

Just now ALL connections dropped AGAIN.

And ironically a few days ago, as a test I disconnected ALL USB devices from it. So nothing could interfere there.

FW is still the same 4.7.0

uptime 7 days, 11:11, load average: 0.00, 0.02, 0.00

In dmesg I see again:

[644191.892094] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 13658, Old_FRxFifoFullCount = 6280, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 6761, Old_FRxFifoFullCount = 4547, RxMpduCount 302739167981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 31020, Old_FRxFifoFullCount = 13658, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 7343, Old_FRxFifoFullCount = 6761, RxMpduCount 302739167981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 57614, Old_FRxFifoFullCount = 31020, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 7763, Old_FRxFifoFullCount = 7343, RxMpduCount 302739167981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 80009, Old_FRxFifoFullCount = 57614, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 8072, Old_FRxFifoFullCount = 7763, RxMpduCount 30273916
[644252.660728] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 113372, Old_FRxFifoFullCount = 80009, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 8650, Old_FRxFifoFullCount = 8072, RxMpduCount 302739167981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 129194, Old_FRxFifoFullCount = 113372, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 8955, Old_FRxFifoFullCount = 8650, RxMpduCount 302739167981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 144286, Old_FRxFifoFullCount = 129194, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 9961, Old_FRxFifoFullCount = 8955, RxMpduCount 302739167981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 159876, Old_FRxFifoFullCount = 144286, RxMpduCount 620695697981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 11132, Old_FRxFifoFullCount = 9961, RxMpduCount 30273916

Can somebody from the gl-inet team please tell me, what this is and how to fix that.
Or are there any other/more logs that you need ?

conker · March 1, 2025, 5:11pm

I'm getting a bit tired of these issues.

I was testing with iperf and connected via lan cable, too.
Then when I unplugged the cable, the wifi started acting weird again.

Could someone from support please tell me, if this a SW issue,
or a HW issue with my unit.

[779475.960669] 
[779673.791686] mtk_soc_eth 15100000.ethernet eth0: Link is Down
[779673.797688] br-lan: port 2(eth0) entered disabled state
[779673.803334] del path: eth0(DN)->apclix0(UP), active path:3
[779677.811668] add new: eth0(UP)->apclix0(UP), active path:4
[779729.525712] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 41150, Old_FRxFifoFullCount = 33920, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 9799, Old_FRxFifoFullCount = 6704, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 56191, Old_FRxFifoFullCount = 41150, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 15149, Old_FRxFifoFullCount = 9799, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 70728, Old_FRxFifoFullCount = 56191, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 17642, Old_FRxFifoFullCount = 15149, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 85489, Old_FRxFifoFullCount = 70728, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 22008, Old_FRxFifoFullCount = 17642, RxMpduCount 59955618
[779790.058526] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 100426, Old_FRxFifoFullCount = 85489, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 32959, Old_FRxFifoFullCount = 22008, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 114849, Old_FRxFifoFullCount = 100426, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 44637, Old_FRxFifoFullCount = 32959, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 129238, Old_FRxFifoFullCount = 114849, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 49864, Old_FRxFifoFullCount = 44637, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 143610, Old_FRxFifoFullCount = 129238, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 51766, Old_FRxFifoFullCount = 49864, RxMpduCount 59955618
[779850.675253] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 158116, Old_FRxFifoFullCount = 143610, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 54110, Old_FRxFifoFullCount = 51766, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 172749, Old_FRxFifoFullCount = 158116, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 56840, Old_FRxFifoFullCount = 54110, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 187263, Old_FRxFifoFullCount = 172749, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 59249, Old_FRxFifoFullCount = 56840, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 201965, Old_FRxFifoFullCount = 187263, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 66899, Old_FRxFifoFullCount = 59249, RxMpduCount 59955618
[779911.172001] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 216429, Old_FRxFifoFullCount = 201965, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 80497, Old_FRxFifoFullCount = 66899, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 230971, Old_FRxFifoFullCount = 216429, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 88462, Old_FRxFifoFullCount = 80497, RxMpduCount 59955618
[779934.190597] 7981@C01L2,wifi_sys_disconn_act() 1002:  wdev_idx=2
[779934.239725] 7981@C08L3,hw_ctrl_flow_v2_disconnt_act() 172: wdev_idx=2
[779934.246883] 7981@C13L2,MacTableDeleteEntry() 1938: Del Sta:####
[779941.476376] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 245712, Old_FRxFifoFullCount = 230971, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 94124, Old_FRxFifoFullCount = 88462, RxMpduCount 59955618WiFi@C15L1,RTMPDeletePMKIDCache() 1311: IF(0), del PMKID CacheIdx=1
[779956.600654] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 260926, Old_FRxFifoFullCount = 245712, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 97077, Old_FRxFifoFullCount = 94124, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 275891, Old_FRxFifoFullCount = 260926, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 100444, Old_FRxFifoFullCount = 97077, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 290534, Old_FRxFifoFullCount = 275891, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 105709, Old_FRxFifoFullCount = 100444, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 305199, Old_FRxFifoFullCount = 290534, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 119044, Old_FRxFifoFullCount = 105709, RxMpduCount 59955618
[780017.257356] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 319989, Old_FRxFifoFullCount = 305199, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 131486, Old_FRxFifoFullCount = 119044, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 335005, Old_FRxFifoFullCount = 319989, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 134296, Old_FRxFifoFullCount = 131486, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 350434, Old_FRxFifoFullCount = 335005, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 136590, Old_FRxFifoFullCount = 134296, RxMpduCount 59955618WiFi@C15L1,RTMPDeletePMKIDCache() 1311: IF(2), del PMKID CacheIdx=0
[780062.845851] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 365029, Old_FRxFifoFullCount = 350434, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 138643, Old_FRxFifoFullCount = 136590, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 380315, Old_FRxFifoFullCount = 365029, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 141279, Old_FRxFifoFullCount = 138643, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 394932, Old_FRxFifoFullCount = 380315, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 143677, Old_FRxFifoFullCount = 141279, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 409378, Old_FRxFifoFullCount = 394932, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 145385, Old_FRxFifoFullCount = 143677, RxMpduCount 59955618
[780123.574558] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 423829, Old_FRxFifoFullCount = 409378, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 147229, Old_FRxFifoFullCount = 145385, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 438657, Old_FRxFifoFullCount = 423829, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 148823, Old_FRxFifoFullCount = 147229, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 453350, Old_FRxFifoFullCount = 438657, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 150893, Old_FRxFifoFullCount = 148823, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 467965, Old_FRxFifoFullCount = 453350, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 153152, Old_FRxFifoFullCount = 150893, RxMpduCount 59955618
[780184.255291] 7981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 482466, Old_FRxFifoFullCount = 467965, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 155176, Old_FRxFifoFullCount = 153152, RxMpduCount 599556187981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 496930, Old_FRxFifoFullCount = 482466, RxMpduCount 100469237981@C00L1,mt7981_hw_auto_debug() 12144: RxFifoFullCount = 158756, Old_FRxFifoFullCount = 155176, RxMpduCount 59955618
[780200.495324] 7981@C01L2,wifi_sys_disconn_act() 1002:  wdev_idx=2
[780200.537925] 7981@C13L1,tx_free_v3_notify_handler() 3530: ContTxFailCntTotal = 1, ContTxFailCnt300ms = 1
[780200.537933] 7981@C13L1,tx_free_v3_notify_handler() 3533: token used by current wcid = 1105, free_token_cnt = 2301
[780200.564642] 7981@C08L3,hw_ctrl_flow_v2_disconnt_act() 172: wdev_idx=2

Cathy · March 7, 2025, 6:29am

Hi @conker
Sorry for the inconvenience caused. This issue seems to be related to interference. Please run the following command on the ssh terminal to see if it helps.
iwpriv rax0 set edccaenable=0