Frequent freezes with a Beryl

Hi,

I’ve been using my beryl for about a week now, but over the last few days I’ve been having complete connection freezes every couple of hours.

The router will be showing its usual white glow, but for all intents and purposes seems to be unresponsive. Here’s some of the stuff I was trying to debug with during the last one:

tlater ~ $ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
^C
tlater ~ $ ping 192.168.8.1
PING 192.168.8.1 (192.168.8.1) 56(84) bytes of data.
From 192.168.8.193 icmp_seq=1 Destination Host Unreachable
From 192.168.8.193 icmp_seq=2 Destination Host Unreachable
From 192.168.8.193 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.8.1 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3055ms
pipe 3
tlater ~ $ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 18:c0:4d:39:64:87 brd ff:ff:ff:ff:ff:ff
    altname enp5s0
    inet 192.168.8.193/24 brd 192.168.8.255 scope global dynamic noprefixroute eno1
       valid_lft 26006sec preferred_lft 20606sec
    inet6 fe80::1ac0:4dff:fe39:6487/64 scope link
       valid_lft forever preferred_lft forever
3: wlp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 34:cf:f6:b8:3a:76 brd ff:ff:ff:ff:ff:ff
tlater ~ $ ping 192.168.8.193
PING 192.168.8.193 (192.168.8.193) 56(84) bytes of data.
64 bytes from 192.168.8.193: icmp_seq=1 ttl=64 time=0.027 ms
^C
--- 192.168.8.193 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.027/0.027/0.027/0.000 ms
tlater ~ $ ping 192.168.8.1
PING 192.168.8.1 (192.168.8.1) 56(84) bytes of data.
^C
--- 192.168.8.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3107ms

This is from an ethernet connection, however the connection on my phone also drops in these instances.

Since I cannot access the router configuration page when this happens, the only recourse should be the reset button, but this hasn’t actually worked yet. After using it the router will just continuously flash blue, until I get tired of waiting and pull and reinsert the power cable - that will restart the router properly and give me access back within a minute or so.

Since I was suspecting this may have been caused by adguard I factory reset the router - I’m currently on firmware version 3.203, and the only configuration changes from the default are:

  • Admin Password set
  • WLAN
    • All SSIDs changed
    • 2.4G disabled
    • 5G password changed
    • 5G set to hidden
    • 5G set to channel 48
  • IGMP snooping enabled
  • Time Zone
  • DNS over TLS enabled

Before the factory reset I’d enabled the openwrt interface and had ssh access to the router, but of course logread is useless since it doesn’t log the previous boot - and I cannot access the router when it fails, so there doesn’t seem to be a way to figure out what’s causing it to freeze.

Any ideas on how I can debug?

There is a firmware version in MT1300/testing called 0826-oops-track that I suggest you try. I’m up almost 6 days without a hiccup. It has a new network driver addressing an ethernet problem–there is a thread about it here. That firmware version has a few broken dependencies (sqm for one) but I haven’t had any performance issues.

I have a very similar set-up to you, only difference is IGMP Snooping. I would suggest disabling that and see if that’s the problem.

Happi

What power supply are you using? My MT1300 has been up for over 3 weeks on stock FW, running over 20+ devices at home. Never a problem, and I have a very custom config too, lots of extra plugins, no issues whatsoever.

1 Like

Hm, thanks for the suggestions so far. I’ll flip off IGMP snooping for now, and look at the testing firmware.

I’m just running with the power supply provided with the router. I’ve had no freezes since that last one 24 hours or so ago, so who knows. Not using the micro SD slot yet, so I don’t have enough space for “lots of plugins” :wink:

I’d be happier with a way to get at least the final few hundred log lines from the previous boot, though, since that way I could probably figure out what’s actually going wrong.

If you are willing to get, we have a special firmware, which record log when the router crashes. You can get these log after the router reboot.

It is here https://dl.gl-inet.com/firmware/mt1300/testing/
Use this file
openwrt-mt1300-3.203-0826-oops-track.bin

When the router crash and reboot, then get the log using

dd if=/dev/mtd7 of=/last.log

Thanks! The router has been stable for about a week now, I’ll wait just a bit longer and try adding some add-ons then. If I still encounter issues I’ll try out that firmware.

1 Like

Issues are back after enabling (and subsequently disabling/removing) a VPN. I’ll try out the firmware and try to see what’s causing it.

Well, that was anti-climatic.

Router is now stuck with a permanent blue light. Using the reset button for a factory reset doesn’t solve this, though the blue light flashes for a very short while, the router turns off and then gets back to permanent blue lights. The other reset mode (3 seconds for a restart I presume) doesn’t do anything, and pulling the cable and reinserting results in the permanent blue light as well.

No wifi is set up, nor do I get an IP address via ethernet (though I do get a carrier, so something is alive):

Sep 12 21:02:32 yui dhcpcd[2347]: eno1: carrier acquired
Sep 12 21:02:32 yui dhcpcd[2347]: eno1: IAID 4d:39:64:87
Sep 12 21:02:32 yui dhcpcd[2347]: eno1: adding address fe80::1ac0:4dff:fe39:6487
Sep 12 21:02:33 yui dhcpcd[2347]: eno1: soliciting a DHCP lease
Sep 12 21:02:33 yui dhcpcd[2347]: eno1: soliciting an IPv6 router
Sep 12 21:02:38 yui dhcpcd[2347]: eno1: probing for an IPv4LL address
Sep 12 21:02:42 yui dhcpcd[2347]: eno1: using IPv4LL address 169.254.52.167
Sep 12 21:02:42 yui dhcpcd[2347]: eno1: adding route to 169.254.0.0/16
Sep 12 21:02:42 yui dhcpcd[2347]: eno1: adding default route
Sep 12 21:02:46 yui dhcpcd[2347]: eno1: no IPv6 Routers available

It seems to now be officially dead - or at least I don’t know how to revive it without getting a connection or a functioning reset button. Is there anything I can try before using my warranty on it?

Hold the reset in for at least 10 seconds to reset.

That doesn’t seem to work :slight_smile:

Ah, I ended up finding this page: Debrick via Uboot - GL.iNet Docs

This worked flawlessly, the router is back alive and I now have a /dev/mtd7 - currently it doesn’t contain anything useful of course, I’ll wait and see if I can reproduce the issue with this firmware. Thanks for the help so far!

Thanks and hope you can help to find more info

I just had another freeze, looks like a memory leak:

\0\0\0\0]\0]<4>[ 3175.210139] 145 total pagecache pages
<4>[ 3175.213783] 0 pages in swap cache
<4>[ 3175.217092] Swap cache stats: add 0, delete 0, find 0/0
<4>[ 3175.222289] Free swap  = 0kB
<4>[ 3175.225150] Total swap = 0kB
<4>[ 3175.228030] 65536 pages RAM
<4>[ 3175.230807] 0 pages HighMem/MovableOnly
<4>[ 3175.234618] 2253 pages reserved
<6>[ 3175.237753] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
<6>[ 3175.246283] [  689]     0   689      258        8       3       0        0             0 urngd
<6>[ 3175.254858] [ 2300]     0  2300      271        7       3       0        0             0 dropbear
<6>[ 3175.263706] [ 4998]     0  4998      944        0       4       0        0             0 gltertf
<6>[ 3175.272483] [17380]     0 17380      406        0       3       0        0             0 netifd
<6>[ 3175.281163] [17463]     0 17463      667        0       4       0        0             0 dnsmasq
<3>[ 3175.289931] Out of memory: Kill process 689 (urngd) score 0 or sacrifice child
<3>[ 3175.297164] Killed process 689 (urngd) total-vm:1032kB, anon-rss:32kB, file-rss:0kB, shmem-rss:0kB
<4>[ 3175.345927] procd invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null),  order=1, oom_score_adj=0
<4>[ 3175.357748] CPU: 1 PID: 1 Comm: procd Not tainted 4.14.221 #0
<4>[ 3175.363465] Stack : 00000000 0000000a 00000040 8007345c 80600000 8059f784 00000000 00000000
<4>[ 3175.371799]         8056912c 8fc2dc24 8fc3837c 805dc9c7 80563ec0 00000001 8fc2dbc8 ac07f5c4
<4>[ 3175.380137]         00000000 00000000 80740000 00010b20 00000000 000010f3 00000007 00000000
<4>[ 3175.388472]         00000000 805e0000 00057574 00000000 80000000 80600000 00000000 8056e910
<4>[ 3175.396810]         00000000 00200000 ffffffff 0000000a 00000000 802c34c0 00000004 80740004
<4>[ 3175.405146]         ...
<4>[ 3175.407589] Call Trace:
<4>[ 3175.410053] [<8000c7b0>] show_stack+0x58/0x100
<4>[ 3175.414493] [<8049ced4>] dump_stack+0xa4/0xe0
<4>[ 3175.418844] [<800d8690>] dump_header.isra.25+0x90/0x1e0
<4>[ 3175.424046] [<800d78a4>] oom_kill_process+0x10c/0x4c4
<4>[ 3175.429078] [<800d8528>] out_of_memory+0x3c0/0x430
<4>[ 3175.433856] [<800dd100>] __alloc_pages_nodemask+0x994/0xd0c
<4>[ 3175.439415] [<8002d2f8>] copy_process.part.87+0xe8/0x160c
<4>[ 3175.444790] [<8002e9d0>] _do_fork+0x100/0x380
<4>[ 3175.449131] [<8002ecb0>] sys_fork+0x24/0x30
<4>[ 3175.453312] [<80016578>] syscall_common+0x34/0x58
<4>[ 3175.458068] Mem-Info:
<4>[ 3175.460347] active_anon:164 inactive_anon:23 isolated_anon:0
<4>[ 3175.460347]  active_file:14 inactive_file:3 isolated_file:0
<4>[ 3175.460347]  unevictable:0 dirty:0 writeback:0 unstable:0
<4>[ 3175.460347]  slab_reclaimable:2340 slab_unreclaimable:18670
<4>[ 3175.460347]  mapped:1 shmem:125 pagetables:21 bounce:0
<4>[ 3175.460347]  free:5060 free_pcp:290 free_cma:0
<4>[ 3175.491958] Node 0 active_anon:656kB inactive_anon:92kB active_file:56kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4kB dirty:0kB writeback:0kB shmem:500kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
<4>[ 3175.513707] Normal free:20240kB min:16384kB low:20480kB high:24576kB active_anon:656kB inactive_anon:92kB active_file:56kB inactive_file:12kB unevictable:0kB writepending:0kB present:262144kB managed:253132kB mlocked:0kB kernel_stack:103600kB pagetables:84kB bounce:0kB free_pcp:1160kB local_pcp:336kB free_cma:0kB
<4>[ 3175.541307] lowmem_reserve[]: 0 0 0
<4>[ 3175.544791] Normal: 438*4kB (UMEH) 116*8kB (MH) 82*16kB (MH) 55*32kB (UMEH) 62*64kB (UMEH) 24*128kB (UMEH) 21*256kB (UMEH) 4*512kB (MEH) 0*1024kB 0*2048kB 0*4096kB = 20216kB
<4>[ 3175.560268] 145 total pagecache pages
<4>[ 3175.563911] 0 pages in swap cache
<4>[ 3175.567225] Swap cache stats: add 0, delete 0, find 0/0
<4>[ 3175.572422] Free swap  = 0kB
<4>[ 3175.575283] Total swap = 0kB
<4>[ 3175.578159] 65536 pages RAM
<4>[ 3175.580936] 0 pages HighMem/MovableOnly
<4>[ 3175.584747] 2253 pages reserved
<6>[ 3175.587886] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
<6>[ 3175.596408] [ 2300]     0  2300      271        7       3       0        0             0 dropbear
<6>[ 3175.605246] [ 4998]     0  4998      944        0       4       0        0             0 gltertf
<6>[ 3175.614023] [17380]     0 17380      406        0       3       0        0             0 netifd
<6>[ 3175.622705] [17463]     0 17463      667        0       4       0        0             0 dnsmasq
<3>[ 3175.631472] Out of memory: Kill process 2300 (dropbear) score 0 or sacrifice child
<3>[ 3175.639060] Killed process 2300 (dropbear) total-vm:1084kB, anon-rss:28kB, file-rss:0kB, shmem-rss:0kB
<4>[ 3175.685915] procd invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null),  order=1, oom_score_adj=0
<4>[ 3175.697734] CPU: 1 PID: 1 Comm: procd Not tainted 4.14.221 #0
<4>[ 3175.703451] Stack : 00000000 80604668 00000040 8007345c 80600000 8059f784 00000000 00000000
<4>[ 3175.711787]         8056912c 8fc2dc74 8fc3837c 805dc9c7 80563ec0 00000001 8fc2dc18 ac07f5c4
<4>[ 3175.720126]         00000000 00000000 80740000 000118d8 00000000 0000111b 00000007 00000000
<4>[ 3175.728463]         00000000 805e0000 000aa586 00000000 80000000 80600000 00000000 805dcac0
<4>[ 3175.736801]         00000001 00200000 ffffffff 80604668 00000000 802c34c0 00000004 80740004
<4>[ 3175.745134]         ...
<4>[ 3175.747572] Call Trace:
<4>[ 3175.750035] [<8000c7b0>] show_stack+0x58/0x100
<4>[ 3175.754474] [<8049ced4>] dump_stack+0xa4/0xe0
<4>[ 3175.758825] [<800d8690>] dump_header.isra.25+0x90/0x1e0
<4>[ 3175.764027] [<800d84e8>] out_of_memory+0x380/0x430
<4>[ 3175.768808] [<800dd100>] __alloc_pages_nodemask+0x994/0xd0c
<4>[ 3175.774364] [<8002d2f8>] copy_process.part.87+0xe8/0x160c
<4>[ 3175.779742] [<8002e9d0>] _do_fork+0x100/0x380
<4>[ 3175.784079] [<8002ecb0>] sys_fork+0x24/0x30
<4>[ 3175.788263] [<80016578>] syscall_common+0x34/0x58
<4>[ 3175.793016] Mem-Info:
<4>[ 3175.795294] active_anon:164 inactive_anon:23 isolated_anon:0
<4>[ 3175.795294]  active_file:0 inactive_file:19 isolated_file:0
<4>[ 3175.795294]  unevictable:0 dirty:0 writeback:0 unstable:0
<4>[ 3175.795294]  slab_reclaimable:2340 slab_unreclaimable:18670
<4>[ 3175.795294]  mapped:1 shmem:125 pagetables:21 bounce:0
<4>[ 3175.795294]  free:5060 free_pcp:302 free_cma:0
<4>[ 3175.826911] Node 0 active_anon:656kB inactive_anon:92kB active_file:0kB inactive_file:76kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4kB dirty:0kB writeback:0kB shmem:500kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
<4>[ 3175.848570] Normal free:20240kB min:16384kB low:20480kB high:24576kB active_anon:656kB inactive_anon:92kB active_file:0kB inactive_file:76kB unevictable:0kB writepending:0kB present:262144kB managed:253132kB mlocked:0kB kernel_stack:103600kB pagetables:84kB bounce:0kB free_pcp:1208kB local_pcp:336kB free_cma:0kB
<4>[ 3175.876084] lowmem_reserve[]: 0 0 0
<4>[ 3175.879570] Normal: 438*4kB (UMEH) 116*8kB (MH) 82*16kB (MH) 56*32kB (UMEH) 62*64kB (UMEH) 24*128kB (UMEH) 21*256kB (UMEH) 4*512kB (MEH) 0*1024kB 0*2048kB 0*4096kB = 20248kB
<4>[ 3175.895046] 145 total pagecache pages
<4>[ 3175.898703] 0 pages in swap cache
<4>[ 3175.902000] Swap cache stats: add 0, delete 0, find 0/0
<4>[ 3175.907213] Free swap  = 0kB
<4>[ 3175.910076] Total swap = 0kB
<4>[ 3175.912943] 65536 pages RAM
<4>[ 3175.915734] 0 pages HighMem/MovableOnly
<4>[ 3175.919548] 2253 pages reserved
<6>[ 3175.922669] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
<6>[ 3175.931197] [ 4998]     0  4998      944        0       4       0        0             0 gltertf
<6>[ 3175.939969] [17380]     0 17380      406        0       3       0        0             0 netifd
<6>[ 3175.948653] [17463]     0 17463      667        0       4       0        0             0 dnsmasq
<0>[ 3175.957421] Kernel panic - not syncing: Out of memory and no killable processes...

There’s also some garbage following that.