Flint 3 (GL-BE9300) 4.10.1 – ECM IPv4 acceleration silently kills new TCP connections over DS-Lite (workaround: front_end_ipv4_stop)

Setup

  • Device: GL-BE9300 (Flint 3), firmware 4.10.1 (OpenWrt 23.05-SNAPSHOT r0-3601c2a49, kernel 5.4.213)
  • ECM packages: kmod-qca-nss-ecm-premium / -wifi-plugin / kmod-qca-nss-sfe / kmod-qca-nss-ppe 5.4.213+gd34b66c-dirty
  • ECM config: ecm.global.enabled=1, acceleration_engine='auto'
  • WAN: German cable ISP (DOCSIS, own cable modem in bridge mode), DS-Lite. wan6 is DHCPv6 with iface_dslite=1, which creates the wan6_4 dslite interface (ds-wan6_4, MTU 1280, AFTR via DHCPv6 option 64). There is no native IPv4; IPv4 only runs through the DS-Lite tunnel. mtu_fix=1 / MSS clamping active. IPv6 LAN mode native (prefix delegation, no NAT66).
  • DPI, AdGuard Home, VPN, QoS and SQM are all off. No custom VLANs.

Symptom

On all LAN clients (iPhone, iPad, MacBook over Wi-Fi, and a wired Raspberry Pi), some web pages don't load or only load after several retries. When it happens, a single new IPv4 TCP connection hangs until the client gives up. IPv6 is not affected. Traffic generated by the router itself (curl on the Flint) is not affected.

Reproduction

Sequential HTTPS requests from a LAN client to about 23 popular sites (apple.com, google.com, amazon.de, spiegel.de, microsoft.com, …), 3 rounds, IPv4 forced, 10 s timeout:

for r in 1 2 3; do while read h; do

curl -4 -s -o /dev/null -m 10 -w "%{time_connect} $h\n" "https://$h/" || echo "FAIL $h"

done < sites.txt; done

Condition Mac (Wi-Fi 5 GHz) Raspberry Pi (wired) Router itself
ECM on (default) 82/100, 61/69, 66/69 OK 67/69 OK 80/80 OK
ECM stopped (/etc/init.d/qca-nss-ecm stop) 69/69 69/69 –
ECM on, echo 1 > /sys/kernel/debug/ecm/front_end_ipv4_stop 69/69, 69/69, 69/69 69/69 –

The same A/B/A result was also seen on an earlier date (11/100 vs. 100/100 OK, with TLS timeouts after a successful TCP handshake). Every failure is curl exit 28 (timeout). Several different CDNs are affected (Akamai, Fastly, CloudFront, Google).

Packet-level observation (nftables log on the Flint, prerouting -450 / postrouting 500)

For every failed connection:

  1. The client SYN arrives on br-lan (logged once).
  2. The SYN leaves via ds-wan6_4.
  3. The SYN-ACK arrives on ds-wan6_4 and is forwarded out of br-lan to the client.
  4. The client sends no SYN retransmissions, so it received the SYN-ACK, but the connection never progresses (no TLS / no data delivered).

So the handshake completes through the slow path. The flow dies right after it is handed to the accelerated path. This looks like ECM/SFE/PPE mishandling IPv4-in-IPv6 (DS-Lite / ip6tnl) flows for some connections, possibly a wrong rule for the tunnel or the MSS/MTU (1280) handling in the offloaded path.

Workaround

Disable only the IPv4 front end. IPv6 stays accelerated:

echo 1 > /sys/kernel/debug/ecm/front_end_ipv4_stop

echo 1 > /sys/kernel/debug/ecm/ecm_db/defunct_all

This does not persist across reboots or ECM restarts, so it is re-applied via rc.local and cron. Throughput afterwards: IPv6 approx. 680 Mbit/s over Wi-Fi (accelerated). IPv4 approx. 350–500 Mbit/s over Wi-Fi (CPU forwarding, approx. 20–30 % softirq).

Related

  • Flint 3 (GL-BE9300) Network Acceleration drops connection shows the same symptom class (website timeouts, retransmissions, fixed by disabling acceleration). In that thread it was PPPoE, and it went away after MTU changes (PPPoE MTU 1492 / WAN MTU 1508). The common factor may be a WAN MTU below 1500 (PPPoE 1492, DS-Lite 1280) that the accelerated path does not handle the way the slow path does (MSS clamping / fragmentation / ICMP frag-needed).

Request

  • Please check ECM/NSS handling of DS-Lite (dslite / ip6tnl) IPv4 flows on the BE9300.
  • Please consider an option in the UI to disable acceleration for IPv4 only, or to exclude DS-Lite tunnels from acceleration automatically.

I'm happy to provide ecm_dump.sh / sfe_dump ipv4 / ppe_flow_dump output taken during a failure, or to test a beta firmware.

Hi,

Thank you for your detailed testing and feedback.

So far, we have not been able to reproduce the issue locally. To better understand your DS-Lite setup and investigate further, could you please share your router with our support team through GoodCloud for remote troubleshooting?

After sharing the device, please send us its MAC address and a temporary router admin password via private message.

Thank you for your understanding and cooperation.

How to send private messages:

Same router and firmware (GL-BE9300, 4.10.1), and the same symptom, but a different trigger and no DS-Lite. This may give you something that's easier to reproduce locally.

Trigger here: a custom VLAN on the LAN side. My WAN is plain DHCP IPv4 on the 2.5G WAN port (MTU 1500). Wi-Fi 7 / MLO is off.

Setup (made over SSH with uci, the same way as the Flint 3 IoT VLAN guide on this forum):

  • switch_vlan with vlan='10' and ports='3t 7t'.
  • eth1.10 bridged into br-iot.
  • Its own firewall zone, with iot -> wan forwarding and masquerade.
  • /etc/config/ecm stock (acceleration_engine 'auto').

Symptom: a wired client on VLAN 10 gets DHCP and DNS and can ping. A new IPv4 TCP connection to the internet completes the handshake, then stalls: TLS ClientHello and nothing back, 0-byte downloads, even plain HTTP. Wired clients on br-lan (eth1.1) are fine. Wi-Fi clients in the same br-iot are fine too.

Reproduction without any external switch: make LAN3 an untagged VLAN 10 port:

swconfig dev switch1 vlan 1 set ports "3t 4 6 7"
swconfig dev switch1 vlan 10 set ports "3t 5 7t"
swconfig dev switch1 port 5 set pvid 10
swconfig dev switch1 set apply

Then plug a laptop into LAN3 and run curl ``https://1.1.1.1/.

A/B tests (same client; curl https://1.1.1.1/ plus a 50 MB HTTPS download):

acceleration_engine / setting Managed switch port, VLAN 10 tagged on LAN1 LAN3 untagged VLAN 10
auto stalls stalls
ppe-sfe stalls not tried
auto + /proc/sys/ppe/bridge_mgr/fdb_disabled=1 stalls not tried
auto + vlan_as_vp_interface=eth1 (eth1.10 recreated afterwards) not tried stalls
auto + front_end_ipv4_stop=1 + defunct_all works not tried
sfe works works

Routed traffic from br-lan to a host on VLAN 10 (no NAT) works under auto. Only NAT to WAN breaks.

ecm_dump.sh for a stuck flow:

conns.conn.13441.from_interfaces.0.ethernet.iface.name=eth1
conns.conn.13441.from_interfaces.1.vlan.iface.name=eth1.10
conns.conn.13441.from_interfaces.1.vlan.tag=a
conns.conn.13441.from_interfaces.2.bridge.iface.name=br-iot
conns.conn.13441.to_interfaces.0.ethernet.iface.name=eth0
conns.conn.13441.ppe_v4.ported.can_accel=1

Taken with this thread and the Slate 7 WireGuard report that staff reproduced on 4.10.0 ([4.10.0] Slate 7 (GL-BE3600): WireGuard client passes no LAN TCP traffic while Network Acceleration (ECM) is on – worked on 4.9.2), it looks like 4.10.x PPE mishandles IPv4 NAT flows whenever the path includes something beyond a plain port: a tunnel, a VPN or a VLAN.

Workaround that keeps most of the speed: uci set ecm.global.acceleration_engine='sfe'; uci commit ecm, then restart qca-nss-ecm. That gives about 870/360 Mbps over Wi-Fi here, versus 886/416 under auto. It persists across reboots, unlike front_end_ipv4_stop. @inquisitive, it may be worth trying on DS-Lite too.

Happy to run ppe_flow_dump / sfe_dump ipv4 while reproducing, or to test a beta.

Thanks @charles2, the router is shared via GoodCloud and I've sent the MAC + temporary admin password by PM.

@Zurandis, thanks for the hint – switching acceleration_engine to sfe fixes it on DS-Lite as well. I ran a larger interleaved A/B test today to make sure it's not just luck.

Method

  • 4 cycles of auto → sfe → cpu (= sfe + front_end_ipv4_stop=1), interleaved so time-of-day effects cancel out. ECM restarted on every engine change, defunct_all on every mode change.
  • Connect test per phase: 49 popular HTTPS sites, IPv4 forced, HEAD request, 10 s timeout, 2 rounds in random order. 6 parallel workers on a MacBook (Wi-Fi, 5 GHz HE80) + 1 worker on a wired Raspberry Pi → 2,744 connections per mode.
  • Throughput per phase: 2 × 15 s, 6 parallel IPv4 downloads, one stream per server spread across 6 different providers (Hetzner fsn1/nbg1/hel1, OVH, thinkbroadband, wtnet) to avoid server-side rate limiting.

Results

Mode Stalled connections¹ Throughput IPv4 (median / range) Router CPU idle
auto (PPE + SFE) 17 / 2,744 (0.62 %) 954 Mbit/s (889–962), 2 of 48 download streams died ~85 %
sfe 0 / 2,744 668 Mbit/s (622–689), 0 streams died ~68 %
cpu (IPv4 front end stopped) 0 / 2,744 553 Mbit/s (484–620), 0 streams died ~63 %

¹ curl exit 28 with time_connect=0 – the same signature as in my first post. Fisher one-sided auto vs sfe: p ≈ 7·10⁻⁶.

(All three modes also show a few curl exit 92 – HTTP/2 stream errors from www.microsoft.com after a completed TLS handshake. They happen equally in cpu mode and on the wired client, so they're server-side and unrelated.)

Possible easy repro for your lab: 13 of the 17 stalls were www.microsoft.com – 13 of 56 attempts (23 %) failed under auto, 0 of 56 under sfe and cpu. The others were ebay.de, stern, youtube and booking (1 each). So with acceleration_engine=auto and a DS-Lite WAN (wan6_4 / ds-wan6_4, MTU 1280), something like this should show it within a minute:

for i in $(seq 1 50); do
  curl -4 -I -s -o /dev/null -m 10 -w "%{time_connect}\n" https://www.microsoft.com/ || echo FAIL
done | grep -c FAIL

Note that the overall rate varies a lot over time – on 10.11 it was 5–20 % of all sites, this morning only ~0.2 % before the rate went up again during the test. So a single short run can look clean.

Takeaways

  • It's the PPE path: with auto, ECM put IPv4 flows on both PPE and SFE (debugfs ecm_ppe_ipv4/accelerated_count and ecm_sfe_ipv4/accelerated_count both increasing), and only auto produces stalls. sfe alone is clean.
  • That matches Zurandis' VLAN+NAT case – PPE seems to mishandle IPv4 NAT flows whose egress isn't a plain port (here the DS-Lite ip6tnl interface; qca_nss_ppe_tunipip6 is loaded).
  • sfe costs ~30 % IPv4 throughput vs. auto (668 vs. 954 Mbit/s over Wi-Fi), but is still ~20 % faster than disabling the IPv4 front end, and it persists via uci without the cron hack.
  • The router is now running acceleration_engine=sfe with front_end_ipv4_stop=0; my old cron/rc.local workaround is removed. If you want to see the bug live during remote troubleshooting, just set it back to auto and restart qca-nss-ecm, then run the loop above from a LAN client.