Tailscale VPN install: why must I stop and remove the mwan3 package? GL-MT1300

Strider3000 · 2022-01-03T09:58:02.951Z

Is there a way I can configure the network or firewall settings on my Gl.inet device (GL-MT1300) so that Tailscale will work gracefully with mwan3?

I’ve successfully enabled Tailscale (1.18.2) on my gl.inet device by installing the Tailscale package on an SD card (loosely following the guide here: Running Tailscale on a Mango* Router - 🄱🄸🅃🅂🄽🄱🄾🄱🅂 Blog).

One thing bugs me, however. I’ve had to stop and remove the mwan3 package in order to be able to ping other Tailscale hosts or to route to remote subnets via the Tailscale interface.

Why must I remove mwan3? I would prefer to keep that package (if possible) so the gl.inet device can gracefully failover from one WAN connection to another (e.g. from WAN to tethering to wifi, etc).

Besides the details in the guide (linked above) I’ve added the following lines to /etc/config/network:

config interface ‘tailscale’
option proto ‘none’
option ifname ‘tailscale0’

I’ve added the following lines to /etc/config/firewall:

config zone ‘vpn_zone’
option name ‘tailscale’
option input ‘ACCEPT’
option forward ‘REJECT’
option output ‘ACCEPT’
option device ‘tailscale0’
option masq ‘1’
option mtu_fix ‘1’

config forwarding
option dest ‘tailscale’
option src ‘lan’

config forwarding
option dest ‘lan’
option src ‘tailscale’

And finally I had to remove mwan3: mwan3 stop && opkg remove mwan3 --force-depends
To re-iterate: Is there a way I can configure the network or firewall settings on my Gl.inet device (GL-MT1300) so that Tailscale will work gracefully with mwan3? Any ideas are appreciated.

alzhao · 2022-01-03T10:17:40.872Z

If you do not remove mwan3, Tailscale cannot connect or you do not have Internet?

Can you give more details?

Strider3000 · 2022-01-03T12:15:37.666Z

Internet always works.
Setup tailscale without disabling and removing mwan3: no ping to tailscale host or network.
Remove mwan3: can ping tailscale host and network.
See the following:

root@GL-MT1300:~# ping google
PING google.com (142.250.113.139): 56 data bytes
64 bytes from 142.250.113.139: seq=0 ttl=105 time=42.625 ms
64 bytes from 142.250.113.139: seq=1 ttl=105 time=41.989 ms
64 bytes from 142.250.113.139: seq=2 ttl=105 time=27.670 ms
^C
— google.com ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 27.670/37.428/42.625 ms

root@GL-MT1300:~# tailscale version
1.18.2
tailscale commit: cc23fba40ac7ec4b60fe2210999a2f20101e0c7c
other commit: e21a681f4a9ffdef994234e4c3d857e8dc3bbb19
go version: go1.17.2-ts7037d3ea51

root@GL-MT1300:~# tailscale ping 100.125.5.11
pong from (100.125.5.11) via :41641 in 169ms

root@GL-MT1300:~# ping 100.125.5.11
PING 100.125.5.11 (100.125.5.11): 56 data bytes
^C
— 100.125.5.11 ping statistics —
24 packets transmitted, 0 packets received, 100% packet loss

root@GL-MT1300:~# ping 192.168.10.200
PING 192.168.10.200 (192.168.10.200): 56 data bytes
^C
— 192.168.10.200 ping statistics —
29 packets transmitted, 0 packets received, 100% packet loss

root@GL-MT1300:~# mwan3 stop && opkg remove mwan3 --force-depends
Removing package mwan3 from root…

reboot here

root@GL-MT1300:~# ping 100.125.5.11
PING 100.125.5.11 (100.125.5.11): 56 data bytes
64 bytes from 100.125.5.11: seq=0 ttl=128 time=761.909 ms
64 bytes from 100.125.5.11: seq=1 ttl=128 time=81.799 ms
64 bytes from 100.125.5.11: seq=2 ttl=128 time=90.931 ms
64 bytes from 100.125.5.11: seq=3 ttl=128 time=81.920 ms
^C
— 100.125.5.11 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 81.799/254.139/761.909 ms

root@GL-MT1300:~# ping 192.168.10.200
PING 192.168.10.200 (192.168.10.200): 56 data bytes
64 bytes from 192.168.10.200: seq=0 ttl=63 time=224.051 ms
64 bytes from 192.168.10.200: seq=1 ttl=63 time=288.800 ms
64 bytes from 192.168.10.200: seq=2 ttl=63 time=258.725 ms
64 bytes from 192.168.10.200: seq=3 ttl=63 time=308.566 ms
64 bytes from 192.168.10.200: seq=4 ttl=63 time=251.003 ms
^C
— 192.168.10.200 ping statistics —
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 224.051/266.229/308.566 ms

ryanc · 2022-01-03T18:14:58.058Z

There’s a conflict in the policy routing rules.

Here’s a script to build the latest version on Linux (requires git and go to be installed):

#!/bin/bash
set -uo pipefail
trap 's=$?; echo ": Error on line "$LINENO": $BASH_COMMAND"; exit $s' ERR
IFS=$'\n\t'


function build_dist {
    echo Build: tailscale-${VERSION}_$1
    env GOOS=linux GOARCH=$1 ./build_dist.sh tailscale.com/cmd/tailscale  && mv tailscale{,-${VERSION}_$1}
    echo Build: tailscaled-${VERSION}_$1
    env GOOS=linux GOARCH=$1 ./build_dist.sh tailscale.com/cmd/tailscaled && mv tailscaled{,-${VERSION}_$1}
}

cd ~/code/tailscale
CURRENT_TAG="$(git describe --tags)"
git fetch --tags || true
REV="$(git rev-list --tags --max-count=1)"
LATEST_TAG="$(git describe --tags $REV)"
if [ "$CURRENT_TAG" != "$LATEST_TAG" ]; then
    git reset --hard
    git checkout "$LATEST_TAG"
    # patch policy routing rule priorities
    grep -lrP '\b52[1357]0\b' . \
        | xargs -n1 sed -Ei 's/\b52([1357])0\b/13\10/g'
fi
export VERSION=$(cat VERSION.txt)
build_dist amd64
build_dist arm
build_dist arm64
export GOMIPS=softfloat
build_dist mips

For the GL-MT1300, I believe you’ll want the mips version.

Here’s a patched one I compiled:

drive.google.com

tailscale-1.18.2_mips_gl.inet_ryanc.zip

Google Drive file.

Strider3000 · 2022-01-03T19:34:28.778Z

Hmmmmm ask and you shall receive - h/t ryanc!
My architecture is mipsle but it was an easy edit.
Here’s my compiled version for DL for mipsle architecture: tailscale-1.18.2_mipsle_gl.inet.zip - Google Drive
Ping is high (300ms compared to 80ms) but it now works with mwan3 enabled.
Can you share what the specific issue was? Have you submitted an issue to the Tailscale project?

For those who wish to compile themselves:
apt install git
snap install go --classic
mkdir /root/code
cd /root/code
git clone GitHub - tailscale/tailscale: The easiest, most secure way to use WireGuard and 2FA.
./mipsle.sh (script as follows):

#!/bin/bash
set -uo pipefail
trap 's=$?; echo ": Error on line "$LINENO": $BASH_COMMAND"; exit $s' ERR
IFS=$'\n\t'

function build_dist {
    echo Build: tailscale-${VERSION}_$1
    env GOOS=linux GOARCH=$1 ./build_dist.sh tailscale.com/cmd/tailscale  && mv tailscale{,-${VERSION}_$1}
    echo Build: tailscaled-${VERSION}_$1
    env GOOS=linux GOARCH=$1 ./build_dist.sh tailscale.com/cmd/tailscaled && mv tailscaled{,-${VERSION}_$1}
}

cd ~/code/tailscale
CURRENT_TAG="$(git describe --tags)"
git fetch --tags || true
REV="$(git rev-list --tags --max-count=1)"
LATEST_TAG="$(git describe --tags $REV)"
if [ "$CURRENT_TAG" != "$LATEST_TAG" ]; then
    git reset --hard
    git checkout "$LATEST_TAG"
    # patch policy routing rule priorities
    grep -lrP '\b52[1357]0\b' . \
        | xargs -n1 sed -Ei 's/\b52([1357])0\b/13\10/g'
fi
export VERSION=$(cat VERSION.txt)
build_dist amd64
build_dist arm
build_dist arm64
export GOMIPS=softfloat
build_dist mipsle

Strider3000 · 2022-01-04T11:35:19.042Z

2021.01.04 update: Ping is back down to normal and things work perfectly with above patched binaries.
@ryanc submitted a github issue with the tailscale project here: Compat issue: GL.iNet/mwan3 · Issue #3659 · tailscale/tailscale · GitHub

PeterE · 2022-09-28T14:35:13.658Z

Hi Strider
Is it possible for you to explain the steps you took to install Tailscale on the Mango router. Do you just replace the 2 directories, but follow BitsnBobs guide elsewhere?

Strider3000 · 2022-10-13T14:09:19.030Z

I’m happy to report that as of Tailscale version 1.32.0 this issue is now solved and we no longer need patched binaries.

Strider3000 · 2022-10-13T14:13:39.937Z

Here are my complete notes:

Install SD card

bitsnbobs.kertho.me – 4 Aug 21

Running Tailscale on a Mango* Router

Instructions on how to get Tailscale working on a GLiNet Mango router

cat /etc/openwrt_release

opkg update
opkg install gdisk e2fsprogs kmod-fs-ext4 curl ca-bundle kmod-tun kmod-usb-storage kmod-usb-storage-uas usbutils block-mount

blkid
block detect | uci import fstab
uci set fstab.@mount[-1].enabled=‘1’
uci commit fstab
uci show fstab
reboot

Note, the storage will sometimes be mmcblk0 or sda1

#mkdir -p /mnt/mmcblk0/opt/tailscale/state
mkdir -p /mnt/sda1/opt/tailscale/state
#ln -s /mnt/mmcblk0/opt/ /opt
ln -s /mnt/sda1/opt/ /opt

copy binaries and folders from package downloaded off tailscale to /opt/tailscale

chmod +x /opt/tailscale/tailscale and tailscaled

ln -s /opt/tailscale/tailscale /usr/sbin/tailscale
ln -s /opt/tailscale/tailscaled /usr/sbin/tailscaled
vim /etc/init.d/tailscale

paste the following script:

#!/bin/sh /etc/rc.common

Copyright 2020 Google LLC.

SPDX-License-Identifier: Apache-2.0

/bin/sleep 30
USE_PROCD=1
START=80

start_service() {
local state_file
local port
local std_err std_out

config_load tailscale
config_get_bool std_out “settings” log_stdout 1
config_get_bool std_err “settings” log_stderr 1
config_get port “settings” port 41641
config_get state_file “settings” state_file /opt/tailscale/state/tailscaled.state

/usr/sbin/tailscaled --cleanup

procd_open_instance
procd_set_param command /usr/sbin/tailscaled

Set the port to listen on for incoming VPN packets.

Remote nodes will automatically be informed about the new port number,

but you might want to configure this in order to set external firewall

settings.

procd_append_param command --port “$port”

OpenWRT /var is a symlink to /tmp, so write persistent state elsewhere.

procd_append_param command --state “$state_file”

procd_set_param respawn
procd_set_param stdout “$std_out”
procd_set_param stderr “$std_err”

procd_close_instance
}

stop_service() {
/usr/sbin/tailscaled --cleanup
}

end paste

chmod +x /etc/init.d/tailscale

vim /etc/config/tailscale

paste the following:

config settings ‘settings’
option log_stderr ‘1’
option log_stdout ‘1’
option port ‘41641’
option state_file ‘/opt/tailscale/state/tailscaled.state’

end paste

vi /etc/config/network

config interface ‘tailscale’
option proto ‘none’
option ifname ‘tailscale0’

vi /etc/config/firewall

config zone ‘vpn_zone’
option name ‘tailscale’
option input ‘ACCEPT’
option forward ‘REJECT’
option output ‘ACCEPT’
option device ‘tailscale0’
option masq ‘1’
option mtu_fix ‘1’

config forwarding
option dest ‘tailscale’
option src ‘lan’

config forwarding
option dest ‘lan’
option src ‘tailscale’

cd /etc/init.d
service tailscale start
tailscale up --advertise-routes=192.168.8.0/24 --accept-routes

check things work

service tailscale enable
/etc/init.d/firewall restart
reboot

Also for WAN access to SSH and HTTP:

/etc/config/firewall:

config rule
option dest_port ‘22’
option src ‘wan’
option name ‘AllowSSH’
option target ‘ACCEPT’
list proto ‘tcp’

config rule
option dest_port ‘80’
option src ‘wan’
option name ‘AllowHTTP’
option target ‘ACCEPT’
list proto ‘tcp’

Upgrade tailscale binaries:

backup old state folders and binaries

upload new ones

chmod +x

reboot

If this is done after a system upgrade, then you need to rebuild scripts and services

PeterE · 2022-10-16T18:46:57.409Z

Thanks
I have got tailscale installed on my MT300N router and it appears on my tailscale admin panel
Another question…if I attempt to route all my router traffic using an exit node, I use the command
tailscale up --exit-node=
However, the router hangs when I enter this command and I have to factory reset it.
Any idea why?
I would like to use the router/tailscale combination as a full tunnel VPN. So any device I connect the router has all the traffic directed to an exit node.

jdub · 2022-12-05T17:41:26.384Z

The reason appears to be related to the various things GL.iNet are doing in their own firewall rules, as far as I can tell. If we look on a device running V4.1, we see:

0:	from all lookup local
51:	from all fwmark 0x100000/0x100000 lookup 51
52:	from all fwmark 0x80000/0x80000 lookup 52
53:	from all fwmark 0x60000/0x60000 lookup 53
1001:	from all iif eth0 lookup 1
2001:	from all fwmark 0x100/0x3f00 lookup 1
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

where a stock device would show… well, nothing:

root@OpenWrt:~# ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
root@OpenWrt:~#

Tailscale normally inserts rules in table 52 with priorities in the 5200s, seen below from a custom MT1300 OpenWRT image running Tailscale:

0:	from all lookup local
5210:	from all fwmark 0x80000 lookup main
5230:	from all fwmark 0x80000 lookup default
5250:	from all fwmark 0x80000 lookup unspec unreachable
5270:	from all lookup 52
32766:	from all lookup main
32767:	from all lookup default

The previous fix was intended to move the Tailscale rules in front of the mwan3 rules - which it does correctly on GLI devices. See below (the rules inserted in the 1300 range instead of 5200 range, as per the patch):

0:	from all lookup local
51:	from all fwmark 0x100000/0x100000 lookup 51
52:	from all fwmark 0x80000/0x80000 lookup 52
53:	from all fwmark 0x60000/0x60000 lookup 53
1001:	from all iif eth0 lookup 1
1002:	from all iif apclix0 lookup 2
1310:	from all fwmark 0x80000/0xff0000 lookup main
1330:	from all fwmark 0x80000/0xff0000 lookup default
1350:	from all fwmark 0x80000/0xff0000 unreachable
1370:	from all lookup 52
2001:	from all fwmark 0x100/0x3f00 lookup 1
2002:	from all fwmark 0x200/0x3f00 lookup 2
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

The problem appears to be that GLI’s firmware seems to be inserting stuff in tables 51 and 52 (and presumably 53, though that crashes):

root@GL-AXT1800:~# ip route show table 51
default via 192.168.20.1 dev eth0 proto static src 192.168.20.196
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
192.168.20.0/24 dev eth0 proto kernel scope link src 192.168.20.196
root@GL-AXT1800:~# ip route show table 52
default via 192.168.20.1 dev eth0 proto static src 192.168.20.196
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
192.168.20.0/24 dev eth0 proto kernel scope link src 192.168.20.196
root@GL-AXT1800:~# ip route show table 53
Dump terminated

(with 192.168.20.0 being the internal subnet the GLI router is on, and 192.168.20.196 being its ip address).

Tailscale tries to insert rules in table 52 - for exit nodes, specifically:

default dev tailscale0
throw 100.100.100.100 dev tailscale0
throw 127.0.0.0/8
throw 192.168.8.0/24

(for a router with a 192.168.8.0/24 lan where --exit-node-allow-lan-access has been enabled)

The problem, as near as I can tell, is that 1) GLI have inserted the table 51-53 lookup rules with priority 51-53, and 2) The fact that GLI have inserted the various rules into Table 51/52 mean that instead of having a clean table 52 like above, you also get something that looks like this:

default dev tailscale0
default via 192.168.20.1 dev eth0 proto static src 192.168.20.196 metric 20
throw 100.100.100.100 dev tailscale0
throw 127.0.0.0/8
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
throw 192.168.8.0/23
192.168.20.0/24 dev eth0 proto static scope link metric 20
throw 192.168.20.0/24

Basically the extra rules and table lookups break --exit-node in Tailscale. It would be great if GLI could issue some sort of fix for this, as their new routers (the AXT1800 in my case, and presumably the new MT3000 as well) are more than powerful enough to run this in a road-warrior setup, which is fantastic. Unfortunately for now the only solution I can see is building a stock luci build using the gl-infra-builder, which works great - but it would be nicer if it were possible to use with stock.

(honestly it would be nicer if GLI just integrated tailscale in, since it’s a) dead simple and b) wouldn’t require too much of a UI to make things work… but there are a lot of priorities and my guess is it’s pretty far down on the list, either way perhaps @alzhao can comment or bring to developers attention).

eric · 2022-12-05T23:48:50.011Z

I agree that with the market penetration for Tailscale, it should just be part of the GL iNet supported packages like Wireguard and OpenVPN. It would be nice to also include and have support for ZeroTier. Having both of these packages supported by GI iNet would make it so much easier for non-tech people to setup a router at home and take one with them.

The problem may be that Tailscale and ZeroTier may be seen as a competitors to Gl iNet’s AstroRelay.

jdub · 2022-12-05T23:54:18.304Z

True. I would settle for “first do no harm” though. It’s fine if they don’t natively support it, but it would be nice if they didn’t make design choices that prevented it from working at all

alzhao · 2022-12-06T06:57:47.857Z

Just an update 4.2.2 will add Tailscale and Zerotier support.

jdub · 2022-12-06T10:46:13.845Z

Excellent! While in development, can we ask that it also support being able to use your own headscsle server (login server) as well as choosing exit nodes?

Strider3000 · 2022-12-06T16:35:30.714Z

I’d also be curious to know if the Zerotier support includes bridging support (Layer2).

alzhao · 2022-12-07T09:36:56.185Z

I am calling developer to talk directly.