kmod feed built for 4.7 breaks modules on 4.8/4.9 — root cause of the BBR/Web-UI bug

There is an earlier report that enabling BBR breaks the Web UI on Flint 3 — routing, internet and SSH keep working, but the admin UI hangs, and reverting to cubic fixes it instantly. That thread is now closed: BBR breaks Web UI on Flint 3 (4.8.4)

I reproduced it on a Slate 7 (BE3600), firmware 4.9.0, kernel 5.4.213 — same ipq53xx platform and the same package feed as Flint 3 — and tracked down the cause.

This is not a BBR problem. kmod-tcp-bbr from the GL feed is ABI-incompatible with the running kernel, and there is no mechanism that detects it.

The decisive test

cubic     (built into kernel)   →  works
reno      (built into kernel)   →  works
bbr       (loadable module)     →  broken
hybla     (loadable module)     →  broken
scalable  (loadable module)     →  broken

Three algorithms with completely different designs failing identically, while both built-ins work, rules out "the algorithm is bad". The discriminator is built-in vs loadable module. (Tested with hardware acceleration both on and off — no difference.)

Why

The configured feed is:

https://fw.gl-inet.com/releases/qsdk_v12.5/kmod-4.7/be3600-ipq53xx
                                            ^^^^^^^^

It is built for firmware 4.7. My device runs 4.9.0, and there is no kmod-4.8 or kmod-4.9 feed — I probed both, for be3600 and be9300, and only kmod-4.7 exists.

Between those builds, struct tcp_sock shifted by 8 bytes. Normally CONFIG_MODVERSIONS catches this via symbol CRCs, but it is disabled on both sides (no module_layout reference in the .ko, no __crc_ symbols in /proc/kallsyms). vermagic only encodes SMP/preempt/arch — it does not check struct layout. So the module loads silently and reads/writes the wrong offsets.

Concretely, by disassembling the running kernel and the module:

kernel:  snd_ssthresh=1700   snd_cwnd=1704   snd_cwnd_clamp=1712
module:  writes cwnd to 1712   ← that is snd_cwnd_clamp, not snd_cwnd

BBR never touches the real snd_cwnd. After any loss event the kernel sets cwnd to 1, BBR thinks it is restoring it but writes to the wrong field, and cwnd stays at 1 for the life of the connection. ss -ti shows the contradiction directly:

bbr  cwnd:1  ...  bbr:(bw:2811472bps, mrtt:175.311)  pacing_rate 2783360bps
     send 50902bps   notsent:445701   busy:444430ms

BBR's own model is correct (2.8 Mbps), but cwnd:1 caps the socket at one packet per RTT ≈ 50 kbps.

Why the Web UI specifically

Loopback gets hit hardest:

loopback throughput, 20 MB:   cubic ~500 MB/s     broken bbr ~18 KB/s

GL's nginx proxies parts of the admin UI over loopback TCP (proxy_pass http://127.0.0.1:3000). Static files that nginx serves from disk are unaffected, and SSH never touches loopback — which is exactly the reported symptom pattern: UI spins, everything else looks fine.

How to check if you are affected

With BBR active, from the router:

curl -o /dev/null http://127.0.0.1/<some large local file>

Healthy: hundreds of MB/s. Broken: tens of KB/s.

Workaround

Rebuild tcp_bbr.ko against headers whose struct layout matches the running kernel. I did this in an arm64 container with upstream 5.4.213 source, verifying with an offsetof probe that the build produces snd_cwnd=1704 / icsk_ca_priv=1264 before installing. Do not just install the feed package.

Results after fixing it

Upload through the proxy, 10 samples each, hardware acceleration on:

cubic   mean  725 KB/s   median  742   range  542–924
bbr     mean 1763 KB/s   median 1763   range 1599–1892

2.4x throughput, and far more consistent — BBR's worst sample beats cubic's best. TLS handshake latency was the same for both once DNS noise was excluded, so the gain is throughput and stability, not latency.

Request to GL

  1. Please publish a kmod feed matching 4.8/4.9, or point the feed at a build made from the shipped kernel tree.
  2. Consider enabling CONFIG_MODVERSIONS so mismatches fail loudly instead of silently corrupting memory.

This is not limited to congestion control. Any module in kmod-4.7 that touches struct sock / inet_connection_sock / tcp_sock will read and write wrong offsets on 4.8/4.9, with no warning.

1 Like

Hi,

Thank you for the report.

We will further analyze this issue with our development team.

Hi,

We confirmed with the development team, and currently:

  1. kmod-4.7 is only the name of the directory used to store the packages. The actual packages are matched based on the kernel version and hash, so there should not be any mismatch issue.
  2. We will ask the team to further check and fix the packages related to struct sock, inet_connection_sock, and tcp_sock.

Regarding the BBR issue, we have applied the fix based on your suggestion, rebuilt the package, and verified locally that it appears to be working properly.

Could you please help verify whether the following package also works correctly in your environment?

kmod-tcp-bbr_5.4.213-1_aarch64_cortex-a53_neon-vfpv4.ipk (5.6 KB)

Once you confirm that it is working properly as well, we will update it to the repository.