Flint 3 (GL-BE9300) - PPE ECM bug causes packet deaths

[Deco]

The following information has been reverse engineered to the best of my ability to diagnose, this took several hours to mitigate and pin down, if you need anything else i’m happy to provide information.

On firmware 4.9.0, ECM's PPE front-end performs zero conntrack stats syncs, while the SFE front-end on the same router performs roughly one per second with no failures. Because ECM refreshes a flow's nf_conntrack entry from those stats, any PPE-accelerated flow is never refreshed: its packet counters freeze and its timeout decays monotonically until the entry is destroyed while the flow is still passing traffic. The NAT mapping disappears, return traffic is dropped, and the application stalls.

This gives every accelerated flow a hard lifetime equal to its conntrack timeout, regardless of activity:

• UDP: 180 seconds (nf_conntrack_udp_timeout_stream)

• TCP: 7440 seconds (nf_conntrack_tcp_timeout_established)

Games and other long-lived UDP sessions therefore break roughly every 3 minutes, while ping, DNS, PPPoE and link state all test perfectly clean, ICMP is not PPE-accelerated, so every conventional diagnostic passes.

Affected

Model	GL-BE9300 (Flint 3)
Firmware	4.9.0
Base	OpenWrt 23.05-SNAPSHOT, kernel 5.4.213
WAN	PPPoE (~900/100)
Setting	ecm.global.acceleration_engine='auto' (the default, which selects PPE)

Symptoms of this bug, there might be more, but are some of the following:

• Online games report a "disconnection" window and recover on their own

• Long-running uploads fail intermittently

• Individual applications stall while the rest of the machine stays online

• The link never drops, the WAN never renegotiates, ping and DNS are flawless throughout

Root cause

Compare the two front-ends on the same router at the same moment, sampled over 90 seconds under normal load:

t=0s   PPE: accel=60   stats=[success=0     fail=0 nack=0]   SFE: accel=6  stats=[success=845017]
t=30s  PPE: accel=73   stats=[success=0     fail=0 nack=0]   SFE: accel=6  stats=[success=845045]
t=60s  PPE: accel=82   stats=[success=0     fail=0 nack=0]   SFE: accel=8  stats=[success=845074]
t=90s  PPE: accel=102  stats=[success=0     fail=0 nack=0]   SFE: accel=8  stats=[success=845103]

Read from:

/sys/kernel/debug/ecm/ecm_ppe_ipv4/stats_request_counter
/sys/kernel/debug/ecm/ecm_sfe_ipv4/stats_request_counter
/sys/kernel/debug/ecm/ecm_ppe_ipv4/accelerated_count
/sys/kernel/debug/ecm/ecm_sfe_ipv4/accelerated_count

PPE is carrying the large majority of flows and has never attempted a stats request, fail=0 and nack=0 show it is not failing, it simply never runs. SFE, handling a fraction of the flows, has 845,000 successes.

Reproduction: watching a live flow get killed

Four concurrent UDP flows from one LAN host to the same game server, sampled every ~15 s. to= is the conntrack timeout, pkts= are conntrack's packet counters:

:51937  to=169 →124 →109 → 94     pkts FROZEN at 722/595
:59902  to=109 → 63 → 48 → 33     pkts FROZEN at 419/815
:53955  to= 49 → 19 →  4 → GONE   pkts FROZEN at 308/738
:55275  to=179 →179 →179 →169     pkts RISING 61/174 → 235/1214

The first three are PPE-accelerated: counters frozen because traffic bypasses the kernel, timeouts decaying with no refresh. :53955 was destroyed while still live. :55275 was not accelerated, so its packets traverse conntrack normally and its timeout keeps resetting to 180, the control case in the same capture.

A flow with 724,610 / 2,915,693 packets was also observed at to=16, and was gone 15 seconds later.

Consequence, from nft list chain inet fw4 accept_to_wan:

oifname "pppoe-wan" ct state invalid counter packets 30597 bytes 1611928 drop

30,597 outbound packets silently dropped as invalid, no RST, no ICMP, because their conntrack entry had been destroyed underneath them.

This is not conntrack exhaustion

Other reports of Flint 3 instability mention nf_conntrack: table full. This is a different failure with the same origin, and it is worth distinguishing:

count: 208    max: 28672
insert_failed=0   drop=0   early_drop=0    (all CPUs)

The table was nowhere near full and had never dropped a packet. The entries are not accumulating, they are being deleted too early.

Why it is easy to misdiagnose

Conntrack packet counters cannot be used as a health signal while PPE holds a flow. They freeze, so a perfectly healthy accelerated connection reads as out=11 pkts / back=1 pkt and looks like a stall. The reliable indicator is the timeout field: pinned near maximum means it is being refreshed; decaying monotonically means it is not.

Additionally, /sys/kernel/debug/ecm/ecm_state/ returns zero bytes at any output mask on this build, so ECM's accelerated-flow list cannot be enumerated from userspace. That rules out diagnosing per-flow engine assignment, and also rules out a userspace conntrack -U refresh daemon as a workaround.

Workaround that works

uci set ecm.global.acceleration_engine='ppe-sfe'
uci commit ecm
reboot

Mode 5 loads both front-ends. PPE remains available for the Wi-Fi / PPE Direct Switch path, while SFE takes routed WAN flows and refreshes their conntrack entries correctly.

Verified after this change:

• UDP flow timeouts hold at 175–179 against a 180 maximum, with rising counters (previously decaying to zero)

• TCP holds 7438–7439 against 7440

• MLO client healthy, router→client latency 4.2 ms (was 14–66 ms)

• Throughput unchanged: 551–630 Mbps vs 608–650 on auto

PPE still reports success=0 under this mode and still holds ~100 flows, so the underlying defect is unfixed, the hybrid simply routes the flows that need conntrack onto the engine that maintains it.

Important: acceleration_engine='sfe' is not a valid workaround

Switching to SFE alone fixes conntrack but completely breaks MLO Wi-Fi clients. The client associates normally at strong signal (−46 dBm, 648 Mbps negotiated) but traffic times out.

Confirmed on both a hot swap and a clean cold boot, so it is not a module load-order artifact. qca_nss_ppe_ds (PPE Direct Switch) is bound to wifi_3_0, and MLD clients on mld0 appear to require the PPE front-end to be loaded.

Throughput for reference, same 1 GB file and path throughout:

Engine	Throughput	Wired UDP flows	MLO Wi-Fi
auto (PPE)	608–650 Mbps	reaped at 180 s	works
sfe	603–627 Mbps	correct	broken
ppe-sfe	551–630 Mbps	correct	works
acceleration off	365–378 Mbps	correct	works

[will.qiu]

Hi,

Thank you for the detailed report. We attempted to reproduce this on a Flint 3 running firmware 4.9.0, but were unable to reproduce the reported UDP failure.

Our test topology was:

PC → Flint 3 → untagged PPPoE → OpenWrt PPPoE server
   → veth → network namespace acting as a remote Internet host

ECM was left at the default:

ecm.global.acceleration_engine='auto'

We verified from ecm_state that the tested UDP flow was actually accelerated by PPE:

ppe_v4.ported.accel_mode=2
ae_nack=0
driver_fail=0

We then maintained one fixed UDP five-tuple for 240 seconds, exceeding the reported 180-second conntrack timeout:

elapsed=240.1
sent=2137
received=2137
lost=0

While traffic was active:

• The conntrack timeout remained at approximately 179 seconds.
• Conntrack packet counters continued increasing.
• At more than 180 seconds, the flow still existed with 1809/1809 packets.
• The flow remained PPE-accelerated; SFE was not used.
• After traffic stopped, the timeout began decreasing normally.

Interestingly, PPE still reported:

stats_request_counter:
success=0 fail=0 nack=0

Therefore, on our unit, a zero stats_request_counter did not mean that the conntrack entry was not being refreshed. This counter alone may not be sufficient to determine whether PPE/ECM statistics synchronization is functioning.

Could you please provide a more exact reproduction procedure, including:

1. Whether the client is wired, regular Wi-Fi, or MLO Wi-Fi.
2. Whether PPPoE uses a VLAN ID.
3. The packet rate, destination, ports, and expected traffic pattern.
4. Whether multiple simultaneous UDP flows are required.
5. The exact commands or script used to generate and monitor the flows.
6. How the affected flows were confirmed to be PPE-accelerated.
7. Whether the problem occurs after a clean factory reset with only PPPoE configured.
8. Relevant anonymized output from:

uci show ecm
ubus call network.interface.wan status
cat /proc/net/pppoe
cat /sys/kernel/debug/ecm/ecm_ppe_ipv4/stats_request_counter
cat /sys/kernel/debug/ecm/ecm_ppe_ipv4/accelerated_count

A packet capture from both the LAN and WAN sides covering the moment the flow stops would also be very helpful. Please remove any public IP addresses or other sensitive information before posting.

With a deterministic reproduction procedure, we can repeat the test under the same conditions and investigate further.

Thanks again for taking the time to document this.

[ecmfix]

I hit the same thing (Zoom calls kept dropping out), but got Claude to ssh in, fix the issues, and provide answers to your questions! Hope this can get fixed in new firmware soon, I was about to throw out the router...

Flint 3 (GL-BE9300) 4.9.0 — answers to the ten diagnostic questions, plus a one-command repro. ppe-sfe does not fix it.

Re: "unable to reproduce the reported UDP failure."

Different reporter, same hardware and firmware. Answering @will.qiu's ten questions below as far as my setup allows, and flagging clearly where I can't.

Important difference from the original report: my WAN is DHCP, not PPPoE. The fault is present anyway, so PPPoE is not required to trigger it.

Also: catching a UDP session dying is hard. There is a kernel-level symptom that is binary, instant, and needs no traffic pattern — see §Repro check. It is present continuously on my unit and gone the moment ECM is stopped.

---

Answers to the ten questions

1. Exact reproduction procedure — Partial. I cannot give a deterministic UDP-failure recipe. I can give a deterministic check for the underlying kernel state (§Repro check), which on my unit is 100% correlated with the fault.

2. Client type (wired / Wi-Fi / MLO Wi-Fi) — Wired. macOS client → Thunderbolt dock → unmanaged 2.5G switch → router LAN. Negotiated 2500Base-T, zero interface errors across 131M packets. All three radios enabled but no Wi-Fi client involved in any measurement here.

3. Does PPPoE use a VLAN ID — N/A — no PPPoE. WAN is proto=dhcp, untagged, public IP from Comcast, ~1.1 Gbps down / 42 Mbps up.

4. Packet rate, destination, ports, traffic pattern — Three patterns, all reproduced it:

• ICMP, 2 pkt/s continuous to 4 targets: 192.168.1.1 (router), 192.168.1.12 (NAS, same switch), 1.1.1.1, 8.8.8.8

• TCP 443 bulk, 8 concurrent HTTPS streams to speed.cloudflare.com

• UDP 8801 (Zoom media), ~1.7 Mbps, 20-minute session — the original user-visible failure

5. Are multiple simultaneous UDP flows required — No. The D-state wedge appears under ordinary mixed household traffic from ~5 clients, with no UDP flood and no synthetic load. It was present while total host throughput was ~1.7 Mbps.

6. Exact commands used to generate and monitor flows — In §Commands below, copy-pasteable.

7. How flows were confirmed PPE-accelerated — Indirectly, by two signals. Stating the method honestly since it is not a direct PPE query:

• /sys/kernel/debug/ecm/ecm_db/connection_count non-zero and tracking (350–409 during tests)

• Conntrack counters for active flows reset to 0 and freeze — e.g. a TCP flow read packets=57 bytes=73528 during the kernel-handled handshake, then packets=0 bytes=0 while still transferring. Counters going to zero while data flows indicates the kernel is no longer seeing the packets.

• If there is a direct way to enumerate PPE-accelerated flows, tell me the command and I'll run it.

8. Does it occur after a clean factory reset with only PPPoE — Not tested. This is a production router; I can't factory-reset it. Also cannot test PPPoE — my ISP is DHCP.

9. uci show ecm output — Below, at time of fault:

ecm.global=ecm
ecm.global.acceleration_engine='auto'
ecm.global.enabled='1'

(stock defaults — not modified before the fault appeared)

10. Packet capture, LAN + WAN, covering the moment the flow stops — Not captured yet. This is the gap. I can produce it if useful: it needs ECM re-enabled and the fault allowed to recur, which degrades a working household network, so I'd want to know the capture you want (interfaces, filter, duration) before doing it once rather than repeatedly.

---

Repro check

cat /proc/loadavg

for p in /proc/[0-9]*; do
    s=$(awk '{print $3}' $p/stat 2>/dev/null)
    [ "$s" = "D" ] && cat $p/comm
done

Condition present when:

• ecm_sfe_ipv4_workqueue / ecm_sfe_ipv6_workqueue appear in D state

• Load sits at ~2.0 while top shows ~97% idle — load is uninterruptible sleep, not work

Observed continuously on my unit:

D kworker/u8:3+ecm_sfe_ipv4_workqueue
D kworker/u8:2+ecm_sfe_ipv6_workqueue
load average: 2.12 2.09 2.03
CPU: 2% usr 0% sys 0% nic 97% idle 0% io 0% irq 0% sirq

The IPv6 workqueue wedges even though no IPv6 address is assigned on WAN (wan6 = dhcpv6 configured, zero inet6 addresses on eth0).

Environment

• GL.iNet BE9300, IPQ5332/AP-MI01.6 — firmware 4.9.0

• OpenWrt 23.05-SNAPSHOT, kernel 5.4.213, Tainted: P C

• WAN DHCP, untagged, no VLAN

• Running: AdGuard Home, netifyd, gl-dpi (all stock). ~5 clients.

• Uptime when first observed: 17h45m

State comparison

acceleration_engine	D-state	load	throughput	loss on router's own LAN IP
auto (stock)	wedged	2.10	~1100 Mbps	32/hr, peak 141/min
ppe-sfe	re-wedged in ~7 min	2.06	836 Mbps	—
disabled	clean	0.04	1077 Mbps	0

Key findings

• ppe-sfe does not fix it. Suggested in this thread as the workaround. Both workqueues re-wedged within ~7 minutes and throughput was lower than disabling ECM outright.

• Restart does not clear it; only stop does. After /etc/init.d/qca-nss-ecm restart a worker re-wedged in 57 seconds. After stop: load 2.10 → 1.38 (20s) → 0.04 (6 min), clean since.

• Not one frozen thread. kworker/u8:N PIDs rotate (u8:0,2,3,4,5,6 observed) — the pool spawns replacements as each wedges. Ages when sampled: 26 min and 96 min.

• Disabling costs nothing on a ~1 Gbps WAN — 1077 vs ~1100 Mbps.

Kernel warning

Triggered by nft on firewall reload, twice, on two different CPUs:

WARNING: CPU: 0 PID: 27163 at kernel/workqueue.c:3049 __flush_work.isra.0+0x4c/0x1a0
CPU: 0 PID: 27163 Comm: nft Tainted: P C 5.4.213 #0
pc : __flush_work.isra.0+0x4c/0x1a0
lr : flush_work+0xc/0x14

Measured impact

Continuous 2 pings/sec, four targets. Client, router and NAS all on the same 2.5G switch.

ECM enabled, one hour:

• NAS 192.168.1.12 (same switch as router) — 0 losses

• Router 192.168.1.1 — 32 losses, four episodes of 4–5s where it stopped answering its own LAN address

• 1.1.1.1 — 36, 8.8.8.8 — 36

• 33 seconds where both public resolvers spiked within a median 0.9 ms of each other (60–220 ms)

ECM stopped, same test: NAS 0, router 0, 1.1.1.1 1, 8.8.8.8 2.

A NAS on the same switch answering every packet while the router drops 32 isolates the fault to the router — not cabling, switch, or client.

Client-side (macOS TCP connection summaries) during a failing 20-min video call:

13:03–13:10 ~9–10 retransmits/min (baseline)
13:11 29
13:13 218
13:15 144

Unrelated processes to unrelated destinations degraded in the same minutes — video, mail and cloud-sync clients spiked together at ~1.7 Mbps total host throughput (~4% link utilisation).

Not congestion.

Commands

Monitor (from a wired LAN client) — logs which targets lose packets concurrently:

ping -i 0.5 192.168.1.1 & # router's own LAN IP
ping -i 0.5 192.168.1.12 & # NAS on same switch — control
ping -i 0.5 1.1.1.1 &
ping -i 0.5 8.8.8.8 &

Conntrack counter / timeout behaviour during a sustained TCP flow:

# on a LAN client:
curl -s -o /dev/null --limit-rate 300k "https://speed.cloudflare.com/__down?bytes=100000000" &

# on the router, sample the same flow repeatedly:
for i in 1 2 3 4 5 6; do
    date +%H:%M:%S
    grep "src=<CLIENT_IP>" /proc/net/nf_conntrack | grep dport=443 | grep ESTABLISHED \
    | awk '{print "timeout="$5, $9, $11, $12}' | head -6
    sleep 20
done

Bulk throughput (8 concurrent streams):

for i in $(seq 1 8); do
    curl -s -o /dev/null "https://speed.cloudflare.com/__down?bytes=50000000" &
done
wait

Ask

• Is ecm_sfe_* wedging in D state expected under any configuration?

• Is there a direct way to enumerate PPE-accelerated flows? (for Q7 above)

• Given ppe-sfe did not work here, is there a supported config that keeps acceleration?

• If you want the LAN+WAN packet capture (Q10), specify interfaces, filter and duration and I'll re-enable ECM and capture it.

Unit is available for testing.

[will.qiu]

Hi,

Thank you for the additional testing and measurements.

We repeated the test internally on firmware 4.9.0 with Hardware Acceleration enabled, AdGuard Home enabled, and DPI disabled. IPv6 was enabled but had not obtained an address. We tested continuous pings, long-running bidirectional UDP traffic, and multi-stream TCP traffic, but could not reproduce the reported packet loss.

We did observe the SFE workqueue entering D state and increasing the load average, without corresponding CPU utilization or packet loss.

---

1. SFE workqueue in D state

An ecm_sfe_* worker appearing in D state is expected when SFE is enabled and is not, by itself, evidence that the worker is wedged.

The worker may enter uninterruptible sleep while waiting for device I/O. Linux includes such tasks in the load average, even when they are not actively consuming CPU. This explains why the load average can increase while CPU usage remains low.

More details are available here.

---

2. DPI status

Could you confirm whether DPI was enabled during your test?

• Enabling DPI through the Admin Panel should automatically disable Hardware Acceleration.
• If ECM was manually re-enabled through SSH while DPI remained enabled, this is not a supported configuration and may cause unexpected behavior.
• Please also confirm whether you manually changed or restarted any ECM-related settings.

---

3. Enumerating PPE-accelerated flows

On the stock v4.9.0 firmware, the following commands provide per-flow information:

ppe_flow_dump
sfe_dump ipv4
ecm_dump.sh

• ppe_flow_dump directly lists PPE hardware flow-table entries, including the five-tuple, NAT information, interfaces, and hw_flow_index.
• sfe_dump ipv4 lists flows handled by SFE.
• ecm_dump.sh shows the ECM connection and its assigned front end. An active entry may contain:

ppe_v4.ported.accel_mode=2

or:

sfe_v4.ported.accel_mode=2

The files under /sys/kernel/debug/ecm/ecm_state/ are not the flow list itself; please use ecm_dump.sh to read the ECM state device.

These outputs contain IP addresses, MAC addresses, and ports, so please redact them before posting publicly.

[ecmfix]

Following up on my earlier posts — I need to withdraw most of what I reported.

Withdrawn

**1. The D-state "repro check". ** @will.qiu was right. I captured the blocked-task backtrace with sysrq-w:

kworker/u8:3 D

Workqueue: ecm_sfe_ipv4_workqueue ecm_sfe_ipv4_init [ecm]

Call trace:

__schedule / schedule / schedule_timeout

schedule_timeout_uninterruptible

ecm_sfe_ipv4_init+0x53c/0x78c [ecm]

That is the driver deliberately sleeping in uninterruptible state, not a wedged worker. The load-average inflation follows from it exactly as you described. **Please disregard that section of my earlier post ** — it should not be used as a reproduction signal by anyone.

**2. My packet-loss evidence. ** My "control" device sat on the same switch as the test client, so that traffic was switched locally and never crossed the router. It could not distinguish a router fault from a fault anywhere else in the path. The comparison was invalid.

**3. My claim that flows were confirmed PPE-accelerated. ** I inferred that from conntrack counters reading zero. On this unit those counters are zeroed on a regular ~15 second cycle (netifyd, presumably), so zero counters mean nothing. accel_mode=2 via ecm_dump.sh is the correct check — thank you for the commands.

Re-test

Acceleration re-enabled, ~50 minutes: continuous 2 pings/sec to four targets plus a long-lived synthetic UDP flow. **Zero packet loss. Could not reproduce. **

The symptoms I originally attributed to acceleration turned out to be three unrelated faults found afterwards: a failed switch port, a stale static default gateway on another host, and the interface issue below.

**I have no evidence that hardware acceleration caused anything on my unit. ** GL-BE9300, firmware 4.9.0, WAN is DHCP (not PPPoE).

One thing that may be a real firmware issue

Unrelated to ECM, but observed twice on this unit and worth a look.

**After a reboot, **eth1.1** can come up with a false no-carrier, killing the entire wired LAN while Wi-Fi works normally. **

eth1 : carrier=1 operstate=up

eth1.1 : carrier=0 operstate=lowerlayerdown

: bridge port state 0 (disabled), 0 MACs learned

swconfig: ports 6 and 7 link up at 2500baseT

The parent interface has carrier and the physical ports are up, but the VLAN interface never follows, so br-lan has no working wired member. Clients get link but no DHCP. Wi-Fi is unaffected because wlan0/1/2 join br-lan directly.

Recovery without a reboot:

ip link **set** eth1.1 down; ip link **set** eth1.1 up

That restored carrier=1, bridge state 3 (forwarding), and MAC learning within seconds.

This may be related to a boot failure I had earlier where the radios came up broadcasting but layer 3 was dead and a factory reset was needed to recover — same shape, network init not completing correctly on boot.

Apologies for the noise on the ECM side.

[will.qiu]

Hi,

Thank you for the follow-up and for confirming that you found no evidence of an ECM/PPE issue after retesting.

The new eth1.1 behavior may be related to this previously reported issue:

[4.9.0] GL-BE9300: link-down on one LAN port disables br-lan's only wired port (eth1.1), disconnecting all wired clients

Hi, Thank you for the detailed report, troubleshooting information, and the workaround you shared. We have reproduced the same issue locally on a GL-BE9300 running v4.9.0. We have recorded this issue and submitted it to the relevant team for further analysis. If there are any further updates, we will continue to follow up with you in this thread. Thank you again for your patience and assistance.

Could you please confirm:

1. Have you created any additional or custom VLANs, including through LuCI or SSH?
2. Does the issue appear to be related to the link state of LAN 1—for example, a connected device being powered on or off, or its Ethernet cable being unplugged or reconnected?

Please also let us know whether a device was connected to LAN 1 when the router booted and the issue occurred. We will investigate it further.