Wireguard client connection dropping and not reconnecting

I’m creating a dedicated topic on request of @alzhao here: Disappointed that VPN client & server cannot be active together - #78 by alzhao

Reminder of the problem

As explained here: Firmware 4.2.x is out as snapshot firmware - #191 by DuxBellorum
I’m using a GL-AX1800 Flint on 4.2.0 beta2 firmware. My Wireguard client’s connection is constantly dropping after a few hours and unable to reconnect by itself. It is stuck in yellow, at “Client starting, please wait…” status. The only way to fix this is to connect to the router interface, and manually stop (and eventually restart) the VPN client. In the meanwhile, Internet connection is KO.

During the last experiment, I turned WG client on at 20:11, at 21:39 I noticed that Internet is not working anymore and WG client is yellow, stuck at “client starting, please wait…”

Here are the logs:

Tue Jan 24 21:34:37 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=REKEY-TIMEOUT SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Tue Jan 24 21:34:37 2023 daemon.notice netifd: Interface ‘wgclient’ has lost the connection
Tue Jan 24 21:34:37 2023 daemon.notice netifd: Network device ‘wgclient’ link is down
Tue Jan 24 21:34:37 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Tue Jan 24 21:34:38 2023 user.notice mwan3[23591]: Execute ifdown event on interface wgclient (unknown)
Tue Jan 24 21:34:38 2023 daemon.notice netifd: wgclient (23599): udhcpc: started, v1.33.2
Tue Jan 24 21:34:38 2023 daemon.notice netifd: wgclient (23599): udhcpc: sending discover
Tue Jan 24 21:34:39 2023 user.notice firewall: Reloading firewall due to ifdown of wgclient ()
Tue Jan 24 21:34:40 2023 daemon.notice netifd: wgclient (23599): udhcpc: no lease, failing
Tue Jan 24 21:34:40 2023 daemon.notice netifd: wgclient (23599): udhcpc: started, v1.33.2
Tue Jan 24 21:34:41 2023 daemon.notice netifd: wgclient (23599): udhcpc: sending discover
Tue Jan 24 21:34:42 2023 daemon.notice netifd: Interface ‘wgclient’ is now down
Tue Jan 24 21:34:42 2023 daemon.notice netifd: Interface ‘wgclient’ is setting up now
Tue Jan 24 21:34:43 2023 user.notice mwan3[24372]: Execute ifdown event on interface wgclient (unknown)
Tue Jan 24 21:34:44 2023 user.notice firewall: Reloading firewall due to ifdown of wgclient ()
Tue Jan 24 21:34:51 2023 user.notice wgclient-up: env value:T_J_A1_1=object T_J_V_ifname=string USER=root T_J_A3_1=object ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=3 J_V_keep=1 T_J_V_ipaddr=array HOME=/ PROTO_IP6ADDR=fc00:bbbb:bbbb:bb01::1:9f10/128//// T_J_T2_mask=string HOTPLUG_TYPE=wireguard T_J_V_interface=string T_J_T4_mask=string J_A1_1=J_T2 CONFIG_wwan_dns= J_V_ifname=wgclient T_J_V_link_up=boolean T_J_T2_ipaddr=string J_A3_1=J_T4 LOGNAME=root DEVICENAME= T_J_V_action=int K_J_A1= 1 T_J_T4_ipaddr=string J_V_ipaddr=J_A1 K_J_A3= 1 TERM=linux SUBSYSTEM=wireguard T_J_V_ip6addr=array PATH=/usr/sbin:/usr/bin:/sbin:/bin J_T2_mask=32 CONFIG_LIST_STATE= J_V_interface=wgclient J_T4_mask=128 K_J_V= action ifname link_up keep ipaddr ip6addr interface J_V_link_up=1 J_T2_ipaddr=10.64.159.17 J_V_action=0 J_T4_ipaddr=fc00:bbbb:bbbb:bb01::1:9f10 N_J_V_link_up=link-up PROTO_IPADDR=10.64.159.17/32// T_J_V_keep=boolean J_V_ip6addr=J_A3 PWD=/ JSON_CUR=J_V K_J_T2= ipaddr mask CONFIG_SECTIONS=global AzireVPN Mullvad FromApp group_8259 grou
Tue Jan 24 21:34:51 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Tue Jan 24 21:35:20 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Tue Jan 24 21:37:22 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Tue Jan 24 21:39:24 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/

A few more precision already raised:

  • I’m using Mullvad VPN
  • my public IP is static
  • I did a firmware reset the day before the experiment, so I only configured what I need and the rest of the settings are at their factory default.
  • I tested with different VPN servers with the same behavior
  • My only WAN source is Repeater, here is my Multi-WAN settings (I didn’t touch anything there):
    image

I still have a pending suggestion of @K3rn3l_Ku5h to test switching the ping IPs in Multi-WAN configuration from Google DNS to Cloudflare or Quad9. I didn’t test it yet, but while I never experienced any problem pinging Google DNS with an effective Internet connection I’m not very optimistic on the success of this solution.

Firmware 4.2.0 snapshot compile time 1-21-2023

So I experienced a vpn disconnect last night I don’t think it is the same thing because I can still get into Admin panel and restart the vpn with no issue.

Part of the system log:

Thu Jan 26 01:12:47 2023 user.info mwan3track[8417]: Lost 6 ping(s) on interface wan (eth0)
Thu Jan 26 01:12:48 2023 daemon.warn dnsmasq[2330]: possible DNS-rebind attack detected: scribe.logs.roku.com
Thu Jan 26 01:12:49 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=REKEY-TIMEOUT SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:12:49 2023 daemon.notice netifd: Interface ‘wgclient’ has lost the connection
Thu Jan 26 01:12:49 2023 daemon.warn dnsmasq[2330]: possible DNS-rebind attack detected: scribe.logs.roku.com
Thu Jan 26 01:12:49 2023 daemon.notice netifd: Network device ‘wgclient’ link is down

The VPN log:

Thu Jan 26 01:12:49 2023 daemon.notice netifd: Network device ‘wgclient’ link is down
Thu Jan 26 01:12:49 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:12:49 2023 user.notice mwan3[25029]: Execute ifdown event on interface wgclient (unknown)
Thu Jan 26 01:12:50 2023 user.notice firewall: Reloading firewall due to ifdown of wgclient ()
Thu Jan 26 01:12:54 2023 daemon.notice netifd: Interface ‘wgclient’ is now down
Thu Jan 26 01:12:54 2023 daemon.notice netifd: Interface ‘wgclient’ is setting up now
Thu Jan 26 01:12:54 2023 user.notice mwan3[25881]: Execute ifdown event on interface wgclient (unknown)
Thu Jan 26 01:12:55 2023 user.notice firewall: Reloading firewall due to ifdown of wgclient ()
Thu Jan 26 01:12:55 2023 daemon.info netdata[14420]: RRDSET: chart name ‘net.wgclient’ on host ‘GL-AX1800’ already exists.
Thu Jan 26 01:12:55 2023 daemon.info netdata[14420]: RRDSET: chart name ‘net_operstate.wgclient’ on host ‘GL-AX1800’ already exists.
Thu Jan 26 01:12:55 2023 daemon.info netdata[14420]: RRDSET: chart name ‘net_carrier.wgclient’ on host ‘GL-AX1800’ already exists.
Thu Jan 26 01:12:55 2023 daemon.info netdata[14420]: RRDSET: chart name ‘net_mtu.wgclient’ on host ‘GL-AX1800’ already exists.
Thu Jan 26 01:12:55 2023 daemon.info netdata[14420]: RRDSET: chart name ‘net_packets.wgclient’ on host ‘GL-AX1800’ already exists.
Thu Jan 26 01:12:55 2023 user.notice wgclient-up: env value:T_J_A1_1=object T_J_V_ifname=string USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=3 J_V_keep=1 T_J_V_ipaddr=array HOME=/ CONFIG_mac_mac= T_J_T2_mask=string HOTPLUG_TYPE=wireguard T_J_V_interface=string J_A1_1=J_T2 J_V_ifname=wgclient T_J_V_link_up=boolean T_J_T2_ipaddr=string LOGNAME=root DEVICENAME= T_J_V_action=int K_J_A1= 1 J_V_ipaddr=J_A1 TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin J_T2_mask=16 CONFIG_LIST_STATE= J_V_interface=wgclient K_J_V= action ifname link_up keep ipaddr interface J_V_link_up=1 J_T2_ipaddr=10.14.0.2 J_V_action=0 N_J_V_link_up=link-up PROTO_IPADDR=10.14.0.2/16// T_J_V_keep=boolean PWD=/ JSON_CUR=J_V K_J_T2= ipaddr mask CONFIG_SECTIONS=global AzireVPN Mullvad FromApp group_9472 group_2439 group_5406 group_8195 peer_1545 peer_1140 peer_7956 peer_74 peer_7813 peer_6018 peer_7182 peer_7134 CONFIG_cfg030f15_ports=
Thu Jan 26 01:12:56 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:13:04 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:15:14 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:17:27 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:19:28 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Thu Jan 26 01:21:46 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/

Why not just change the ping to a local IP or localhost and remove MWAN from the equation entirely?

I was also going to recommend changing the debug level, but after a cursory search, it does not look like that’s an option from the CLI or configs for the WG client.

Since it disconnects frequently, if you have a spare router, you could always setup a local WG server in front of your flint temporarily (daisy-chain) and see if the issue is reproduced locally, narrowing it down to the WG client on the flint or pushing the troubleshooting to the inet connection or WG server.

I actually do not use WG, but I stayed in a Holiday Inn Express :face_in_clouds:

Jan 24 21:34:38 2023 user.notice mwan3[23591]: Execute ifdown event on interface wgclient (unknown)

I think this is the issue

I am pretty sure something in the MWAN3 settings is messed up or needs to be changed. You could change the available condition to a lower amount and/or change failure condition to higher
You ISP may have a specific number because of latency.

Still does not explain why it locks up.

Indeed, it seems quite a good catch.

I actually never asked to add MWAN to the equation, since I have only one source (well, I’m actually interested in the feature because I’m considering buying a pair of PLC to add up to the repeater bandwidth, but that’s another subject), and if it’s not fully functional, for now I don’t need it.

Apparently there’s no way to just turn it off ? I have the same values than in the screen so I’ll give it another try changeing these. Or may I just toggle off the Internet status tracking ?

Can you try this? Maybe mwan3 killed the wireguard interface.

I create a post here I am also experiencing similar issue.

I just tried this and it happened again. I have set off the Internet status tracking. I have turned on Wireguard client at 18:27. At 22:18, I notice Internet is not working anymore and WG client is yellow again.

Here are the logs (I’m on phone so I only paste them like they’re given):

Fri Jan 27 21:55:23 2023 daemon.notice netifd: wgclient (15950): udhcpc: no lease, failing
Fri Jan 27 21:55:23 2023 daemon.notice netifd: wgclient (15950): udhcpc: started, v1.33.2
Fri Jan 27 21:55:23 2023 daemon.notice netifd: wgclient (15950): udhcpc: sending discover
Fri Jan 27 21:55:24 2023 daemon.notice netifd: Interface 'wgclient' is now down
Fri Jan 27 21:55:24 2023 daemon.notice netifd: Interface 'wgclient' is setting up now
Fri Jan 27 21:55:24 2023 user.notice mwan3[16727]: Execute ifdown event on interface wgclient (unknown)
Fri Jan 27 21:55:25 2023 user.notice firewall: Reloading firewall due to ifdown of wgclient ()
Fri Jan 27 21:55:33 2023 user.notice wgclient-up: env value:T_J_A1_1=object T_J_V_ifname=string USER=root T_J_A3_1=object ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=3 J_V_keep=1 T_J_V_ipaddr=array HOME=/ PROTO_IP6ADDR=fc00:bbbb:bbbb:bb01::1:9f10/128//// T_J_T2_mask=string HOTPLUG_TYPE=wireguard T_J_V_interface=string T_J_T4_mask=string J_A1_1=J_T2 CONFIG_wwan_dns= J_V_ifname=wgclient T_J_V_link_up=boolean T_J_T2_ipaddr=string J_A3_1=J_T4 LOGNAME=root DEVICENAME= T_J_V_action=int K_J_A1= 1 T_J_T4_ipaddr=string J_V_ipaddr=J_A1 K_J_A3= 1 TERM=linux SUBSYSTEM=wireguard T_J_V_ip6addr=array PATH=/usr/sbin:/usr/bin:/sbin:/bin J_T2_mask=32 CONFIG_LIST_STATE= J_V_interface=wgclient J_T4_mask=128 K_J_V= action ifname link_up keep ipaddr ip6addr interface J_V_link_up=1 J_T2_ipaddr=10.64.159.17 J_V_action=0 J_T4_ipaddr=fc00:bbbb:bbbb:bb01::1:9f10 N_J_V_link_up=link-up PROTO_IPADDR=10.64.159.17/32// T_J_V_keep=boolean J_V_ip6addr=J_A3 PWD=/ JSON_CUR=J_V K_J_T2= ipaddr mask CONFIG_SECTIONS=global AzireVPN Mullvad FromApp group_8259 grou\nFri Jan 27 21:55:34 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 21:57:26 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 21:59:31 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:01:34 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:03:37 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:05:41 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:07:54 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:10:15 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:11:55 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:14:00 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:16:03 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/
Fri Jan 27 22:18:04 2023 user.notice wireguard-debug: USER=root ifname=wgclient ACTION=KEYPAIR-CREATED SHLVL=2 HOME=/ HOTPLUG_TYPE=wireguard LOGNAME=root DEVICENAME= TERM=linux SUBSYSTEM=wireguard PATH=/usr/sbin:/usr/bin:/sbin:/bin PWD=/

EDIT: I manually replaced the ‘\n’ with a line feed, please also make the logs directly pastable here !

Seems like mwan3 is still doing its thing, here is the multi-WAN configuration:

@hansome optimized the Wireguard watchdog can you pls try?

Snapshot from Jan 30th

https://dl.gl-inet.com/?model=ax1800&type=snapshot

I was reluctant at first to deploy a snapshot firmware on the router but I ended up giving it a try. I have installed the snapshot from Feb 4th, and for now it’s been working from 36 hours without any interruption. I’m still watching it but I’m rather optimistic that the problem is solved !

1 Like