Hi,
Thanks to @bryan for providing remote access.
We’ve identified the root cause of the issue—an update to Tailscale introduced a problem with firewall marks.
opened 08:03AM - 19 Apr 24 UTC
connectivity
OS-linux
P1 Nuisance
bug
pod/network-features
### What is the issue?
`nftables` firewall backend on Linux seems not using f… wmark `0x00040000/0x00ff0000` like its correspondent in `iptables`, instead it uses fwmark `0x00000400/0x0000ff00` for firewall rules.
### Steps to reproduce
On systems that using `nftables` backend
```bash
sudo nft -s list ruleset ip
```
```nft
table ip filter {
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter jump ts-forward
}
chain INPUT {
type filter hook input priority filter; policy accept;
counter jump ts-input
}
chain ts-forward {
iifname "tailscale0*" counter meta mark set meta mark & 0xffff04ff | 0x00000400
meta mark & 0x0000ff00 == 0x00000400 counter accept
oifname "tailscale0*" ip saddr 100.64.0.0/10 counter drop
oifname "tailscale0*" counter accept
}
chain ts-input {
iifname "lo*" ip saddr <local_ts_ip> counter accept
iifname != "tailscale0*" ip saddr 100.115.92.0/23 counter return
iifname != "tailscale0*" ip saddr 100.64.0.0/10 counter drop
iifname "tailscale0*" counter accept
udp dport <ts_lis_port> counter accept
}
}
table ip nat {
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
counter jump ts-postrouting
}
chain ts-postrouting {
meta mark & 0x0000ff00 == 0x00000400 counter masquerade
}
}
```
On systems with `iptables`
```bash
sudo iptables -nvL -t filter
```
```
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ts-input all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ts-forward all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain ts-forward (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- tailscale0 * 0.0.0.0/0 0.0.0.0/0 MARK xset 0x40000/0xff0000
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x40000/0xff0000
0 0 DROP all -- * tailscale0 100.64.0.0/10 0.0.0.0/0
0 0 ACCEPT all -- * tailscale0 0.0.0.0/0 0.0.0.0/0
Chain ts-input (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- lo * <ts_local_ip> 0.0.0.0/0
0 0 RETURN all -- !tailscale0 * 100.115.92.0/23 0.0.0.0/0
0 0 DROP all -- !tailscale0 * 100.64.0.0/10 0.0.0.0/0
0 0 ACCEPT all -- tailscale0 * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:<ts_lis_port>
```
### Are there any recent changes that introduced the issue?
Seems a3c7b21 forgot to take endianness into consideration, as shown [here](https://github.com/tailscale/tailscale/commit/a3c7b21cd18866cd8ceeee9b102c9e4171fcd719#diff-24d56f3ddeab24b98059a573652492d4aea06a7a23299f6a622bb4b4be3368ddR69)
### OS
Linux
### OS version
Arch Linux
### Tailscale version
1.64.0
### Other software
firewalld
### Bug report
_No response_
If users manually updated to affected version, the firewall marks may not be applied correctly, causing the traffic to match the IP rules blackhole rule.
For affected users, we recommend rolling back the update for now.
(If you updated via @admon ’s script, you can run sh update-tailscale.sh --restore.)
For more details, please refer to the fix submitted by our R&D team to @admon ’s repository:
main ← handongming:fix/nft-fwmark-endian-11803
opened 02:17AM - 06 May 26 UTC
## Problem
On **little-endian** hosts (arm64, amd64), Tailscale’s **nftables*… * netfilter backend built fwmark **mask/value** bytes for `meta mark` / `Bitwise` expressions using the **wrong byte order**. Rules that should match **`0x40000 / 0xff0000`** (third byte) instead behaved like **`0x00000400 / 0x0000ff00`** — i.e. the mask effectively hits the **wrong 8-bit lane**.
That collides with **policy-routing marks in the `0x8000 / 0xf000` nibble** (e.g. OpenWrt **fw4 / `vpn_table`** style marks on GL.iNet firmware). **Tailscale’s `0x80000/0xff0000` and GL’s `0x8000/0xf000` are meant to be non-overlapping in `ip rule`**, but the nft bug shifts the bitfield so they **fight the same mark bits**.
Typical symptom: **first packet of a flow passes, later packets break** (e.g. LAN ping gets one reply then silence) once **connmark save/restore** paths run on **ESTABLISHED** traffic — consistent with [tailscale#11803](https://github.com/tailscale/tailscale/issues/11803).
Workaround users may try: `tailscale set --netfilter-mode=off` (trade off Tailscale’s netfilter integration). Proper fix: **`encoding/binary.NativeEndian`** in `util/linuxfw` — same direction as [tailscale#19601](https://github.com/tailscale/tailscale/pull/19601).
## This PR
- Adds `patches/0001-fix-nftables-fwmark-endianness.patch` (from the #11803 / #19601 fix).
- In `build-tailscale.yaml`, after checking out `tailscale/tailscale`, checks out this repo under `repo/` and runs `patch -p1 < repo/patches/0001-...` **before** `go mod download` / cross-compile.
**No change** to feature tags, UPX publishing, or release flow — only the Tailscale sources are patched before build.
## When to drop this
Remove the patch + checkout/apply steps after **#19601** is merged **and** the Tailscale release you build already contains the fix.