Not communicating through VPN (L2TP/ipsec) default gateway (GL-M3000 Beryl AX)

ryu2012 · February 9, 2023, 7:44am

I am using GL-M3000 BerylAX 4.1.2.

I am using straongswan-full (5.9.2-2) and xl2tpd (1.3.16-2) to VPN tunnel to my home router.

The ipsec connection and xl2tpd authentication were successful and the default gateway is the l2tp tunnel.

root@GL-MT3000:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.32.2    0.0.0.0         UG        0 0          0 l2tp-VPN
121.95.xxx.xxx  192.168.1.1     255.255.255.255 UGH       0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.8.0     0.0.0.0         255.255.255.0   U         0 0          0 br-lan
192.168.32.2    0.0.0.0         255.255.255.255 UH        0 0          0 l2tp-VPN

However, the communication does not go through the default gateway (192.168.32.2) but through eth0 (192.168.1.1).


root@GL-MT3000:~# traceroute www.google.com
traceroute to www.google.com (142.250.206.196), 30 hops max, 46 byte packets
 1  192.168.1.1 (192.168.1.1)  0.680 ms  0.693 ms  0.775 ms
 2  210.227.xx.xx (210.227.xx.xx)  1.036 ms  1.125 ms  1.054 ms
 3  211.0.223.41 (211.0.223.41)  2.451 ms  2.329 ms  2.236 ms
 4  153.146.171.209 (153.146.171.209)  2.056 ms  2.020 ms  1.954 ms
 5  60.37.54.69 (60.37.54.69)  2.454 ms  122.1.245.49 (122.1.245.49)  2.752 ms  60.37.54.69 (60.37.54.69)  2.255 ms
 6  122.28.104.122 (122.28.104.122)  4.853 ms  2.430 ms  60.37.54.110 (60.37.54.110)  2.378 ms
 7  61.126.86.26 (61.126.86.26)  2.842 ms  61.126.86.30 (61.126.86.30)  2.506 ms  61.126.86.26 (61.126.86.26)  2.991 ms
 8  *  *  108.170.242.176 (108.170.242.176)  4.704 ms
 9  216.239.41.68 (216.239.41.68)  2.922 ms  108.170.237.92 (108.170.237.92)  2.676 ms  209.85.244.35 (209.85.244.35)  4.002 ms
10  142.250.58.92 (142.250.58.92)  10.439 ms  108.170.242.209 (108.170.242.209)  3.705 ms  108.170.242.208 (108.170.242.208)  5.594 ms
11  209.85.241.107 (209.85.241.107)  3.898 ms  209.85.244.2 (209.85.244.2)  4.002 ms  209.85.244.37 (209.85.244.37)  3.615 ms
12  142.250.58.20 (142.250.58.20)  20.736 ms  142.250.229.250 (142.250.229.250)  9.953 ms  142.250.58.20 (142.250.58.20)  11.479 ms
13  108.170.243.97 (108.170.243.97)  10.923 ms  11.406 ms  108.170.243.129 (108.170.243.129)  9.881 ms
14  142.250.236.53 (142.250.236.53)  10.199 ms  kix07s07-in-f4.1e100.net (142.250.206.196)  9.796 ms  142.250.236.35 (142.250.236.35)  9.756 ms
root@GL-MT3000:~#

I have tried similar VPN and network settings on an OpenWRT router from another manufacturer and it works fine, do I need any Beryl specific settings?

hansome · February 9, 2023, 11:54am

Can you print command output:

netstat -anp

and debug info here.
I may find some clue.

By the way, why not use wireguard? it’s simpler to configure and faster.

jdub · February 9, 2023, 1:00pm

Not my thread but I’ll answer.

WireGuard is not always faster, particularly when AES runs 5x faster than ChaCha.
It’s a disaster to configure and maintain if you’ve got more than about 10-15 nodes.
You may already have an IPSec stack set up and you want to plug into that.
You’re connecting to an enterprise network that may only allow IPSec and not WireGuard.

I will grant that IPSec can be a pain to configure**, but it’s just not true that WireGuard is universally a better solution. WireGuard is very good for a very specific set of use cases, but if you need or want to do anything more complicated it can fall apart quickly. There are other reasons to avoid using it in production applications (e.g., no support for cipher negotiation***), but we’ll leave that aside for now.

** (or it can be quite easy! I would take the bet that I could set up a point-to-point IPSec tunnel between two pfSense nodes as fast or faster than you can set up a WireGuard tunnel between two MT3000s - particularly if anything complicated needed to happen.)

*** What happens if there’s a serious break in ChaCha20 that’s published tomorrow? You have to push a kernel update to all of your devices because WireGuard is “opinionated” (to use the white paper’s term) about its cipher suite. Not so big of a deal if you have 2 travel routers that you can update at your leisure. Major deal if you’re using it in an industrial control system where you can’t reboot without taking the plant down for maintenance. Or you need to update 1,000 endpoints basically simultaneously. Oops. With the capacity to automatically switch between ciphers, you uncheck a box (or delete a string from the config file) and everything just works.

ryu2012 · February 10, 2023, 12:54am

Here are the netstat results.

root@GL-MT3000:~# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3695/nginx.conf -g 
tcp        0      0 192.168.32.132:53       0.0.0.0:*               LISTEN      5061/dnsmasq
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      5061/dnsmasq
tcp        0      0 192.168.1.123:53        0.0.0.0:*               LISTEN      5061/dnsmasq
tcp        0      0 192.168.8.1:53          0.0.0.0:*               LISTEN      5061/dnsmasq
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2530/dropbear
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3695/nginx.conf -g 
tcp        0    864 192.168.8.1:22          192.168.8.104:49526     ESTABLISHED 28724/dropbear
tcp        0      0 :::80                   :::*                    LISTEN      3695/nginx.conf -g 
tcp        0      0 ::1:53                  :::*                    LISTEN      5061/dnsmasq
tcp        0      0 fe80::9683:c4ff:fe27:952a:53 :::*                    LISTEN      5061/dnsmasq
tcp        0      0 :::22                   :::*                    LISTEN      2530/dropbear
tcp        0      0 :::443                  :::*                    LISTEN      3695/nginx.conf -g 
udp        0      0 0.0.0.0:4500            0.0.0.0:*                           7144/charon
udp        0      0 0.0.0.0:1701            0.0.0.0:*                           3359/xl2tpd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           3480/avahi-daemon: 
udp        0      0 0.0.0.0:500             0.0.0.0:*                           7144/charon
udp        0      0 192.168.32.132:53       0.0.0.0:*                           5061/dnsmasq
udp        0      0 127.0.0.1:53            0.0.0.0:*                           5061/dnsmasq
udp        0      0 192.168.1.123:53        0.0.0.0:*                           5061/dnsmasq
udp        0      0 192.168.8.1:53          0.0.0.0:*                           5061/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           5061/dnsmasq
udp     4608      0 0.0.0.0:68              0.0.0.0:*                           7144/charon
udp        0      0 :::4500                 :::*                                7144/charon
udp        0      0 :::5353                 :::*                                3480/avahi-daemon: 
udp        0      0 :::500                  :::*                                7144/charon
udp        0      0 ::1:53                  :::*                                5061/dnsmasq
udp        0      0 fe80::9683:c4ff:fe27:952a:53 :::*                                5061/dnsmasq
raw   214272      0 0.0.0.0:17              0.0.0.0:*               17          7144/charon
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING       7438 3480/avahi-daemon:  /var/run/avahi-daemon/socket
unix  14     [ ]         DGRAM                      5656 2056/logd           /dev/log
unix  2      [ ACC ]     STREAM     LISTENING       6957 3175/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING      12849 7144/charon         /var/run/charon.xml
unix  2      [ ACC ]     STREAM     LISTENING      12851 7144/charon         /var/run/charon.wlst
unix  2      [ ACC ]     STREAM     LISTENING      12853 7144/charon         /var/run/charon.dck
unix  2      [ ACC ]     STREAM     LISTENING      13116 7600/usbmuxd        /var/run/usbmuxd
unix  2      [ ACC ]     STREAM     LISTENING      12872 7144/charon         /var/run/charon.ctl
unix  2      [ ACC ]     STREAM     LISTENING      12874 7144/charon         /var/run/charon.vici
unix  2      [ ACC ]     STREAM     LISTENING       4990 1758/fcgiwrap       /var/run/fcgiwrap.socket
unix  2      [ ACC ]     STREAM     LISTENING       4336 1043/ubusd          /var/run/ubus/ubus.sock
unix  3      [ ]         STREAM     CONNECTED       7652 3695/nginx.conf -g  
unix  2      [ ]         DGRAM                      7162 3359/xl2tpd         
unix  3      [ ]         STREAM     CONNECTED       7650 3695/nginx.conf -g  
unix  2      [ ]         DGRAM                      4340 1043/ubusd          
unix  3      [ ]         STREAM     CONNECTED       5658 2056/logd           
unix  2      [ ]         DGRAM                      7433 3480/avahi-daemon:  
unix  2      [ ]         DGRAM                      5875 2530/dropbear       
unix  2      [ ]         DGRAM                      7237 2758/netifd         
unix  3      [ ]         STREAM     CONNECTED       1413 1/procd             
unix  3      [ ]         STREAM     CONNECTED       6212 2108/rpcd           
unix  3      [ ]         STREAM     CONNECTED      11765 1043/ubusd          /var/run/ubus/ubus.sock
unix  3      [ ]         STREAM     CONNECTED       7651 3695/nginx.conf -g  
unix  3      [ ]         STREAM     CONNECTED       6105 2888/odhcpd         
unix  2      [ ]         DGRAM                      6043 2758/netifd         
unix  3      [ ]         STREAM     CONNECTED       6178 1043/ubusd          /var/run/ubus/ubus.sock
unix  3      [ ]         STREAM     CONNECTED       6213 1043/ubusd          /var/run/ubus/ubus.sock
unix  3      [ ]         STREAM     CONNECTED       6045 2758/netifd         
unix  3      [ ]         STREAM     CONNECTED       4343 1043/ubusd          /var/run/ubus/ubus.sock
unix  3      [ ]         STREAM     CONNECTED      11764 6699/lua            
unix  3      [ ]         STREAM     CONNECTED       6772 1043/ubusd          /var/run/ubus/ubus.sock
unix  3      [ ]         STREAM     CONNECTED       7649 3695/nginx.conf -g  
unix  3      [ ]         STREAM     CONNECTED       6046 1043/ubusd          /var/run/ubus/ubus.sock
unix  2      [ ]         DGRAM                     13122 7647/pppd           
unix  3      [ ]         STREAM     CONNECTED       6362 1043/ubusd          /var/run/ubus/ubus.sock
unix  2      [ ]         DGRAM                     16665 1/procd             
unix  2      [ ]         DGRAM                      7188 3070/crond          
unix  3      [ ]         STREAM     CONNECTED       7440 3480/avahi-daemon:  
unix  3      [ ]         STREAM     CONNECTED       6361 2283/lua            
unix  3      [ ]         STREAM     CONNECTED       7247 3175/dbus-daemon    
unix  3      [ ]         STREAM     CONNECTED       7246 3175/dbus-daemon    
unix  2      [ ]         DGRAM                      6364 2283/lua            
unix  3      [ ]         STREAM     CONNECTED       7441 3175/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                      9126 4291/starter        
unix  2      [ ]         DGRAM                     10936 5061/dnsmasq        
unix  3      [ ]         STREAM     CONNECTED      10933 5061/dnsmasq        
unix  3      [ ]         STREAM     CONNECTED       5068 1868/lua            
unix  3      [ ]         STREAM     CONNECTED      10934 1043/ubusd          /var/run/ubus/ubus.sock
unix  2      [ ]         DGRAM                     14251 7144/charon         
unix  3      [ ]         STREAM     CONNECTED       5069 1043/ubusd          /var/run/ubus/ubus.sock
unix  2      [ ]         DGRAM                     10939 5061/dnsmasq        
root@GL-MT3000:~#

Result of debug info.
ipt.zip (15.5 KB)

firmware

hansome · February 14, 2023, 11:48am

I notice there is a ip rule, that’s added by your setting, it may get modified by mwan3.

220:	from all lookup 220

Try to disable mwan3 for any conflict with ip rule and route added by L2TP/ipsec.

/etc/init.d/mwan3 disable
uci set mwan3.globals.enabled='0'
uci commit mwan3
reboot

ryu2012 · February 15, 2023, 2:08am

Thank you!
I disabled mwan3 and everything worked fine.

root@GL-MT3000:~# traceroute www.google.com
traceroute to www.google.com (216.58.220.100), 30 hops max, 46 byte packets
 1  192.168.32.2 (192.168.32.2)  5.472 ms  5.352 ms  5.655 ms
 2  xxxxxx.kddnet.ad.jp (118.155.xxx.xxx)  12.179 ms  10.581 ms  9.826 ms
 3  72.14.242.145 (72.14.242.145)  14.301 ms  14.988 ms  14.808 ms
 4  *  *  *
 5  64.233.175.42 (64.233.175.42)  15.354 ms  108.170.235.120 (108.170.235.120)  10.853 ms  108.170.236.126 (108.170.236.126)  10.105 ms
 6  108.170.242.209 (108.170.242.209)  11.013 ms  142.250.226.59 (142.250.226.59)  13.016 ms  108.170.242.208 (108.170.242.208)  9.682 ms
 7  nrt12s30-in-f4.1e100.net (216.58.220.100)  13.830 ms  72.14.234.67 (72.14.234.67)  14.345 ms  nrt12s30-in-f4.1e100.net (216.58.220.100)  13.735 ms
root@GL-MT3000:~#

jdub · February 19, 2023, 1:54am

FYI this also works to make Tailscale actually function as an exit node.

Also, paging @jsr to this thread.