Flint Crash due to adguard Memory overprovision

Technical Support for Routers
1

GL-AX1800 Flint Firmware 3.214 AdGuardHome V 0.107.6 Router Uptime was 22 days

Sys log

Wed Jun 29 12:33:38 2022 user.notice AdGuardHome[9939]: 2022/06/29 16:33:38.930517 [info] saving filter 1650726409 contents to: /etc/AdGuardHome/data/filters/1650726409.txt
Wed Jun 29 12:33:38 2022 user.notice AdGuardHome[9939]: 2022/06/29 16:33:38.932547 [info] updated filter 1650726409: 9140810 bytes, 399547 rules
Wed Jun 29 12:33:39 2022 user.notice AdGuardHome[9939]: 2022/06/29 16:33:39.966540 [info] saving filter 1654614424 contents to: /etc/AdGuardHome/data/filters/1654614424.txt
Wed Jun 29 12:33:39 2022 user.notice AdGuardHome[9939]: 2022/06/29 16:33:39.968466 [info] updated filter 1654614424: 722805 bytes, 32089 rules
Wed Jun 29 12:33:40 2022 user.notice AdGuardHome[9939]: 2022/06/29 16:33:40.168100 [info] Updated filter #1650726409. Rules: 399548 → 399547
Wed Jun 29 12:33:40 2022 user.notice AdGuardHome[9939]: 2022/06/29 16:33:40.168209 [info] Updated filter #1654614424. Rules: 32089 → 32089
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: fatal error: runtime: out of memory
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]:
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime stack:
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.throw({0x84f7e8, 0x16})
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/panic.go:1198 +0x5c
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.sysMap(0xe800000, 0x1400000, 0x2387f58)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/mem_linux.go:169 +0xb0
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.(*mheap).grow(0x206a640, 0x94c)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/mheap.go:1393 +0x264
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.(*mheap).allocSpan(0x206a640, 0x94c, 0x0, 0x0)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/mheap.go:1179 +0x1ac
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.(*mheap).alloc.func1()
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/mheap.go:913 +0x78
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.systemstack()
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/asm_arm.s:318 +0x60
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]:
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: goroutine 9 [running]:
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.systemstack_switch()
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/asm_arm.s:275 +0x4 fp=0x48c6ba8 sp=0x48c6ba4 pc=0x7e4ec
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.(*mheap).alloc(0x206a640, 0x94c, 0x0, 0x1)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/mheap.go:907 +0x58 fp=0x48c6bcc sp=0x48c6ba8 pc=0x38598
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.(*mcache).allocLarge(0x76fae088, 0x1298000, 0x1, 0x0)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/mcache.go:227 +0x78 fp=0x48c6c00 sp=0x48c6bcc pc=0x2681c
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.mallocgc(0x1298000, 0x7be438, 0x1)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/malloc.go:1088 +0x6b4 fp=0x48c6c44 sp=0x48c6c00 pc=0x1c5f0
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.newarray(0x7be438, 0x22000)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/malloc.go:1256 +0x9c fp=0x48c6c58 sp=0x48c6c44 pc=0x1cb9c
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.makeBucketArray(0x788460, 0x11, 0x0)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/map.go:362 +0x17c fp=0x48c6c74 sp=0x48c6c58 pc=0x1dac8
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.hashGrow(0x788460, 0x45a1e20)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/map.go:1036 +0x8c fp=0x48c6c98 sp=0x48c6c74 pc=0x1f5e4
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.mapassign_fast32(0x788460, 0x45a1e20, 0xb535d48)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/map_fast32.go:156 +0xd4 fp=0x48c6cb4 sp=0x48c6c98 pc=0x202c8
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd(0x51a33b0, 0xb9cd290, 0x629f699800023469)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/urlfilter@v0.15.2/lookup/shortcutstable.go:81 +0x1dc fp=0x48c6d00 sp=0x48c6cb4 pc=0x3d6bfc
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/urlfilter.(*NetworkEngine).AddRule(0x40c4f30, 0xb9cd290, 0x629f699800023469)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/urlfilter@v0.15.2/networkengine.go:84 +0x7c fp=0x48c6d24 sp=0x48c6d00 pc=0x3d8be8
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/urlfilter.NewDNSEngine(0x4537380)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/urlfilter@v0.15.2/dnsengine.go:79 +0x4ac fp=0x48c6d60 sp=0x48c6d24 pc=0x3d77c8
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/filtering.(*DNSFilter).initFiltering(0x40dc380, {0x0, 0x0, 0x0}, {0x40dc7e0, 0x6, 0x8})
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/filtering/filtering.go:693 +0x98 fp=0x48c6d88 sp=0x48c6d60 pc=0x5da07c
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/filtering.(*DNSFilter).SetFilters(0x40dc380, {0x40dc7e0, 0x6, 0x8}, {0x0, 0x0, 0x0}, 0x0)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/filtering/filtering.go:326 +0xb4 fp=0x48c6df8 sp=0x48c6d88 pc=0x5d86bc
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home.enableFiltersLocked(0x0)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home/filter.go:741 +0x170 fp=0x48c6f44 sp=0x48c6df8 pc=0x6fabd8
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home.enableFilters(0x0)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home/filter.go:709 +0x6c fp=0x48c6f58 sp=0x48c6f44 pc=0x6faa30
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home.(*Filtering).refreshFiltersIfNecessary(0x2057318, 0x6)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home/filter.go:414 +0x2f8 fp=0x48c6fc4 sp=0x48c6f58 pc=0x6f8350
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home.(*Filtering).periodicallyRefreshFilters(0x2057318)
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home/filter.go:248 +0xd0 fp=0x48c6fe4 sp=0x48c6fc4 pc=0x6f75d8
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime.goexit()
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: runtime/asm_arm.s:838 +0x4 fp=0x48c6fe4 sp=0x48c6fe4 pc=0x80084
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: created by github.com/AdguardTeam/AdGuardHome/internal/home.(*Filtering).Start
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: github.com/AdguardTeam/AdGuardHome/internal/home/filter.go:52 +0x38
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]:
Wed Jun 29 12:33:54 2022 daemon.err AdGuardHome[9939]: goroutine 1 [chan receive, 60 minutes]:

Kernel log

[1887747.015986] Out of memory: Kill process 23722 (AdGuardHome) score 423 or sacrifice child
[1887747.024566] Killed process 23722 (AdGuardHome) total-vm:835748kB, anon-rss:187268kB, file-rss:0kB
[1909665.580982] AdGuardHome invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
[1909665.581013] AdGuardHome cpuset=/ mems_allowed=0
[1909665.591530] CPU: 1 PID: 23723 Comm: AdGuardHome Tainted: P 4.4.60 #1
[1909665.592997] Hardware name: Generic DT based system
[1909665.600738] [<80116d90>] (unwind_backtrace) from [<801136e4>] (show_stack+0x10/0x14)
[1909665.605763] [<801136e4>] (show_stack) from [<803159ac>] (dump_stack+0x7c/0x9c)
[1909665.613494] [<803159ac>] (dump_stack) from [<801aa738>] (dump_header+0x4c/0x1ac)
[1909665.621059] [<801aa738>] (dump_header) from [<801aac48>] (oom_kill_process+0x80/0x41c)
[1909665.628420] [<801aac48>] (oom_kill_process) from [<801ab2b8>] (out_of_memory+0x26c/0x300)
[1909665.636667] [<801ab2b8>] (out_of_memory) from [<801af218>] (__alloc_pages_nodemask+0x68c/0x768)
[1909665.644742] [<801af218>] (__alloc_pages_nodemask) from [<801a9d10>] (filemap_fault+0x284/0x464)
[1909665.653783] [<801a9d10>] (filemap_fault) from [<801c6504>] (__do_fault+0x3c/0x94)
[1909665.662619] [<801c6504>] (__do_fault) from [<801c8fa4>] (handle_mm_fault+0x5e4/0x10c8)
[1909665.670087] [<801c8fa4>] (handle_mm_fault) from [<8011d63c>] (do_page_fault+0x130/0x338)
[1909665.678420] [<8011d63c>] (do_page_fault) from [<80101214>] (do_DataAbort+0x34/0xb4)
[1909665.686506] [<80101214>] (do_DataAbort) from [<8010273c>] (__dabt_usr+0x3c/0x40)
[1909665.694384] Exception stack(0x86f89fb0 to 0x86f89ff8)
[1909665.701847] 9fa0: 00000000 04f243a0 06226ff0 06226ff0
[1909665.706985] 9fc0: 00000014 0000007c 02007ee8 00000000 06226ff0 00000000 04bfec30 06226fd8
[1909665.715308] 9fe0: 06226ff3 04f24cb4 003d1148 000ca31c 80000010 ffffffff
[1909665.723883] Mem-Info:
[1909665.730599] active_anon:57892 inactive_anon:114 isolated_anon:0
[1909665.730599] active_file:115 inactive_file:106 isolated_file:0
[1909665.730599] unevictable:0 dirty:0 writeback:0 unstable:0
[1909665.730599] slab_reclaimable:1083 slab_unreclaimable:28513
[1909665.730599] mapped:1615 shmem:540 pagetables:313 bounce:0
[1909665.730599] free:5381 free_pcp:310 free_cma:0
[1909665.744325] Normal free:20656kB min:16384kB low:20480kB high:24576kB active_anon:231568kB inactive_anon:456kB active_file:800kB inactive_file:876kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:445440kB managed:430756kB mlocked:0kB dirty:0kB writeback:0kB mapped:6572kB shmem:2160kB slab_reclaimable:4332kB slab_unreclaimable:114052kB kernel_stack:1456kB pagetables:1252kB unstable:0kB bounce:0kB free_pcp:1180kB local_pcp:292kB free_cma:0kB writeback_tmp:0kB pages_scanned:10616 all_unreclaimable? yes
[1909665.790070] lowmem_reserve[]: 0 0 0
[1909665.815639] Normal: 2774kB (UME) 2318kB (UME) 10116kB (UME) 14432kB (UME) 3464kB (UME) 8128kB (UE) 6256kB (UE) 5512kB (ME) 11024kB (M) 22048kB (M) 0*4096kB = 21596kB
[1909665.830744] 900 total pagecache pages
[1909665.831529] 111360 pages RAM
[1909665.835261] 0 pages HighMem/MovableOnly
[1909665.838408] 3671 pages reserved
[1909665.842482] 0 pages cma reserved
[1909665.845766] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[1909665.849284] [ 231] 0 231 210 11 5 0 0 0 ubusd
[1909665.857802] [ 681] 0 681 181 6 4 0 0 0 askfirst
[1909665.866721] [ 2267] 0 2267 1367 1026 7 0 0 -1000 cnssdaemon
[1909665.875480] [ 2725] 0 2725 275 44 5 0 0 0 logd
[1909665.884866] [ 2734] 0 2734 322 23 4 0 0 0 rpcd
[1909665.893359] [ 2791] 0 2791 1130 89 6 0 0 0 hostapd
[1909665.902138] [ 2795] 0 2795 1196 46 5 0 0 0 wpa_supplicant
[1909665.911076] [ 2879] 0 2879 388 36 5 0 0 0 netifd
[1909665.920722] [ 3585] 0 3585 333 12 4 0 0 0 udhcpc
[1909665.929594] [ 3870] 0 3870 375 55 4 0 0 0 sh
[1909665.938402] [ 3930] 0 3930 611 34 6 0 0 0 acfg_tool
[1909665.946932] [ 5376] 0 5376 334 13 4 0 0 0 crond
[1909665.956141] [ 5433] 0 5433 219 6 3 0 0 0 dropbear
[1909665.964887] [ 5456] 0 5456 602 19 6 0 0 0 gl_crond
[1909665.973653] [ 6506] 0 6506 663 48 5 0 0 0 smbd
[1909665.982681] [ 6507] 0 6507 669 24 5 0 0 0 nmbd
[1909665.991522] [ 6901] 0 6901 13692 8591 30 0 0 0 collectd
[1909666.000054] [ 7178] 0 7178 619 20 6 0 0 0 gl_ble_ubus
[1909666.009083] [ 7199] 0 7199 334 13 5 0 0 0 sh
[1909666.018451] [ 7285] 0 7285 334 14 3 0 0 0 ntpd
[1909666.027029] [ 7292] 0 7292 237 7 3 0 0 0 thermald
[1909666.035570] [ 7530] 0 7530 431 110 5 0 0 0 sh
[1909666.044595] [ 7684] 0 7684 232 18 4 0 0 0 mcsd
[1909666.053281] [ 7778] 0 7778 759 40 6 0 0 0 lbd
[1909666.062129] [ 8362] 0 8362 917 114 5 0 0 0 gltertf
[1909666.070446] [ 8383] 0 8383 1340 362 6 0 0 0 lighttpd
[1909666.079380] [ 8414] 0 8414 1318 958 7 0 0 0 gltraffic
[1909666.088315] [ 8525] 0 8525 745 38 6 0 0 0 api
[1909666.097813] [ 9105] 0 9105 357 38 5 0 0 0 sh
[1909666.106135] [ 9444] 0 9444 994 166 5 0 0 0 openvpn
[1909666.114714] [22023] 453 22023 647 43 6 0 0 0 dnsmasq
[1909666.123565] [22554] 0 22554 188 8 3 0 0 0 rngd
[1909666.132510] [23721] 0 23721 209001 46502 109 0 0 0 AdGuardHome
[1909666.141260] [14794] 0 14794 333 8 4 0 0 0 sleep
[1909666.150507] [ 2244] 0 2244 333 7 3 0 0 0 sleep
[1909666.159433] [ 2565] 0 2565 333 7 4 0 0 0 sleep
[1909666.168249] [ 2566] 0 2566 333 7 3 0 0 0 sleep
[1909666.176966] Out of memory: Kill process 23721 (AdGuardHome) score 419 or sacrifice child
[1909666.185764] Killed process 23721 (AdGuardHome) total-vm:836004kB, anon-rss:186008kB, file-rss:0kB

2

@hansome can you have a check?

3

I wouldn’t advise using adguard on the router if it’s as unstable as the other platforms. I use it on android and often it goes into “let’s block the Internet for no reason” mode.

Its safer to use adguard on the computers instead as it can be easily and quickly configured or reset.

4

It’s adguardhome has memleak, and finally get killed by oom-killer. will fix it asap.

5

The problem is that the product box itself promotes AdGuard Home inside. So people buying this router expect it to run AGH.

2 Likes
6

Not sure if changes have been made but the router crashes ever 6 hours even when running:
echo ‘3’ > /proc/sys/vm/drop_caches
This previously allowed the router to work for a 22 and 11 day period. Have a crash log from past 72hours.

7

Could you upload log or send to handongming#gl-inet.com.

8

Sent the crash log 24hours ago. If you see a multiple power interrrupts that is me pulling the plug because I could not get into Admin dashboard. I have got the router working for 24hours now but RAM is 95% used(seems really high to me)I have adguard home running, OpenVPN client running with VPN policies. I think when I added Netdata it used to much RAM. Side note it would be nice to maybe see a integration of Netdata into the GL.iNet dashboard

9

Hello,

After investigating your log and the thread, https://github.com/AdguardTeam/AdGuardHome/issues/4437
Seems the leak issue is not solved completely.
You can try upgrade to verion 107.8 to see if it works better.

In my last test setup, I also see memory usage increse:

One status of your log：
total-vm:836132kB, anon-rss:212720kB, file-rss:0kB

my setup intial：
total-vm:833456kB, anon-rss:11756kB, file-rss:10804kB

my setup 10 hours later：
total-vm:833712kB, anon-rss:16676kB, file-rss:11316kB

I trigger oom killer manully by command:

echo f >/proc/sysrq-trigger

As the issue 4437, user SiNONiMiTY tells,
“maybe there is something wrong with the garbage collection”.

You can try use crontab to restart adguard regularly,
Also, I’m trying to find a better way to handle this.