Zerotier routed VPN

Router
1

site 1 GL-BE9300, 4.7.11
site 2 GL-X3000, 4.7.4 (UI shows 4.0 0704release5)
zerotier enabled on both from UI

For routed setup, I added static route for each router's LAN with its zerotier IP as nexthop (on zerotier central), and disabled Masquerading in openwrt firewall on zerotier zone

It sort of works, except large packets keep getting lost, on wired the problem might not be obvious, on wireless it is completely unusable (can't finish TLS handshake)

I changed zerotier MTU to 1280 zerotier API, both devices picked up MTU change. With MTU clamping enabled on both ends, site 1 > site 2 MSS 1460 (clamping not working), site 2 to site 1 MSS 1240 (clamping working)

Can gl-inet provide a usable default settings for routed zerotier VPN? why doing SNAT on VPN which seems quite convoluted / unnecessary.

2

Even with 1280 MTU on zerotier network and working PMTUD (getting ICMP PTB with DF), I am still seeing high packet loss (retransmission) and corrupted packets (ERR_SSL_PROTOCOL_ERROR)

3

Hello,

Are you using Zerotier as a VPN?
I guess the role of X3000 is client, and BE9300 is server (or nexthop for X3000).

This usage scenario seems to have not been tested.

4

This is a zerotier site to site VPN setup over Internet (no client / server for zerotier), zerotier managed routes for each router's LAN with nexthop being the router's zerotier IP

I suspect this has something to do with BE9300's hardware acceleration (out of order packets?). I never had this issue with X3000.

5

yep, this part I known.
But I did not clear that issue about you encounter in the Flint 3 Zerotier, ping [X3000 LAN IP] in Flint3 some time the ICMP keep getting lost?

6

no issues with ICMP, but TCP packets keep getting lost especially larger packets, even after MSS clamping to 1260 bytes

7

@bruce can gl-inet add an option / test for routed zerotier prefixes? the NAT setup is problematic for some use cases (e.g. IP cameras), having routed prefix work without NAT simplifies a lot of things.

8

It looks like the Zerotier of 2 devices didn't make a hole successfully, and they access via relay server, probably server was not very stable.

Is it possible to change a different WAN source to test again?

Do you want to build a bridged layer 2 network through Zerotier?

9

@bruce

zerotier-cli peers shows all peers are DIRECT, not a NAT traversal issue

not bridged, this is routed

router 1 LAN <> router 1 ZT <> router 2 ZT <> router 2 LAN, so both router 1 and router 2 as intermediate IP hops, no NAT involved for LAN 1 <> LAN 2

10

So the NAT question of IP cam mentioned above should not exist, since it is actually LAN 1 <-> ZT <-> LAN 2 through the routing table.

However, the large TCP packets lost is very strange.
Have you tried to disable the hardware acceleration of BE9300 to check?
But I think hardware acceleration should only involve for NAT, but no impact on routing.

11

Is there a way to confirm hardware acceleration state directly? toggling it in web UI made no difference in CPU utilization for NAT traffic

12

The BE9300 (and other QSDK models) seem to be no way to see if the packet is processed by hardware acceleration, only statistics are seen:
cat /sys/kernel/debug/ecm/ecm_ppe_ipv4/accelerated_count

The X3000 (and other MTK_SDK models), you can check the process logs for HA:
cat /sys/kernel/debug/hnat/all_entry
state=BIND means process by HA, state=UNBIND means no process.

13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.