MT1300 won't do packet shaping because traffic stats are wrong

markus.laker · August 7, 2021, 12:07pm

Hi! I’ve just bought a GL-MT1300 in order to do packet shaping on our Internet connection. But it doesn’t work, because most of the traffic that should go through the br-wan bridge isn’t counted in traffic stats, and so no packet-shaping occurs. In addition, if I use tcpdump to examine the traffic on br-wan, some packets that go through the MT1300 show up, but most don’t. Despite this, network traffic flows normally across the MT1300. Connectivity is fine; it’s just that I can’t do packet shaping, which is what I bought the device to do.

The MT1300 sits between the ADSL router and the Lan – no machine, including our wireless access point, can access the Internet without going through the MT1300. The ADSL router is on 192.168.1.1, all client machines are on 192.168.0.x/24, and the MT1300 is the gateway between them, with its Wan port on 192.168.1.4 and its Lan port (middle socket) on 192.168.0.3. Pardon my terrible drawing skills!

My desktop PC has only one route to the ADSL router:

[msl@localhost ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.3     0.0.0.0         UG    100    0        0 enp9s0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 enp9s0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
[msl@localhost ~]$ traceroute -In 192.168.1.1
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets
 1  192.168.0.3  0.964 ms  1.130 ms  1.332 ms
 2  192.168.1.1  2.083 ms  2.278 ms  2.475 ms
[msl@localhost ~]$

On the MT1300, I have these bridges, which were preconfigured when I bought the device:

root@GL-MT1300:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-guest                7fff.000000000000       no
br-lan          7fff.9483c40da929       no              eth0.1
br-wan          7fff.9483c40da928       no              eth0.2
root@GL-MT1300:~#

The MT1300’s routing table looks like this – note, the only path to the ADSL router on 192.168.1.1 is through the br-wan bridge:

root@GL-MT1300:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 br-wan
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br-wan
192.168.9.0     0.0.0.0         255.255.255.0   U     0      0        0 br-guest
root@GL-MT1300:~#

There’s also a vlan on eth0. I confess that I know almost nothing about vlans. Again, this was preconfigured when I bought the device.

root@GL-MT1300:~# ip -d link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff promiscuity 3 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
root@GL-MT1300:~# ip -d link show eth0.1
14: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP mode DEFAULT group default qlen 1000
    link/ether 94:83:c4:0d:a9:29 brd ff:ff:ff:ff:ff:ff promiscuity 1 
    vlan protocol 802.1Q id 1 <REORDER_HDR> 
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 7fff.94:83:C4:0D:A9:29 designated_root 7fff.94:83:C4:0D:A9:29 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress off vlan_tunnel off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
root@GL-MT1300:~# ip -d link show eth0.2
16: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-wan state UP mode DEFAULT group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff promiscuity 1 
    vlan protocol 802.1Q id 2 <REORDER_HDR> 
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 7fff.94:83:C4:0D:A9:28 designated_root 7fff.94:83:C4:0D:A9:28 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress off vlan_tunnel off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
root@GL-MT1300:~#

Here’s a demonstration of the problem. I list the traffic stats on all interfaces; I email myself an incompressible 4MiB file using SMTP; I wait for my desktop computer to stop sending data, and then leave an extra few seconds for the MT1300 to send the last few packets to the ADSL router; and then I list the traffic stats again. Here are the stats before the transfer; for brevity, I’ve removed stats for ifb0, ifb1, teql0, ra0 and br-guest, which are all zero.

root@GL-MT1300:~# ip -s a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    4641       46       0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    4641       46       0       0       0       0       
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9683:c4ff:fe0d:a928/64 scope link 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    45615662   86789    0       21      0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    46698563   77590    0       0       0       0       
13: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:29 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.3/24 brd 192.168.0.255 scope global br-lan
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    2436457    25421    0       2631    0       5778    
    TX: bytes  packets  errors  dropped carrier collsns 
    3940709    15090    0       0       0       0       
14: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:29 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    2450851    25715    0       79      0       5859    
    TX: bytes  packets  errors  dropped carrier collsns 
    3940709    15090    0       0       0       0       
15: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.4/24 brd 192.168.1.255 scope global br-wan
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    1473416    7241     0       0       0       80      
    TX: bytes  packets  errors  dropped carrier collsns 
    1185804    8015     0       0       0       0       
16: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-wan state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    1473416    7241     0       0       0       80      
    TX: bytes  packets  errors  dropped carrier collsns 
    1185804    8015     0       0       0       0       
root@GL-MT1300:~#

Here are the stats after the transfer:

root@GL-MT1300:~# ip -s a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    4641       46       0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    4641       46       0       0       0       0       
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9683:c4ff:fe0d:a928/64 scope link 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    51946640   93912    0       21      0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    53054822   84599    0       0       0       0       
13: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:29 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.3/24 brd 192.168.0.255 scope global br-lan
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    2456376    25720    0       2667    0       5844    
    TX: bytes  packets  errors  dropped carrier collsns 
    3962172    15266    0       0       0       0       
14: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:29 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    2471092    26021    0       80      0       5926    
    TX: bytes  packets  errors  dropped carrier collsns 
    3962172    15266    0       0       0       0       
15: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.4/24 brd 192.168.1.255 scope global br-wan
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    1479256    7303     0       0       0       80      
    TX: bytes  packets  errors  dropped carrier collsns 
    1192931    8093     0       0       0       0       
16: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-wan state UP group default qlen 1000
    link/ether 94:83:c4:0d:a9:28 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    1479256    7303     0       0       0       80      
    TX: bytes  packets  errors  dropped carrier collsns 
    1192931    8093     0       0       0       0       
root@GL-MT1300:~#

You can see that the byte-counts for br-wan are 1185804 before and 1192931 after. So br-wan claims to have sent only 7,127 bytes while the 4MiB file was being emailed. The byte-counts for the underlying interface, eth0.2, are identical.

Eth0 claims to have sent 6,356,259 byes, which is about what I’d expect when emailing a 4MiB file. But the TX count goes up by a similar amount when I download the big file I’ve just emailed myself. That makes it useless for packet-shaping – I only want to control data going out to the Internet.

Unsurprisingly, when I attempt packet-shaping on br-wan, nothing useful happens. The traffic still flows out to the ADSL router at full speed, and we get buffer bloat that slows down interactive use. Enabling OpenWrt’s SQM doesn’t help, either, because the traffic stats are (wrongly) so low than no shaping occurs.

In contrast, when I pipe data directly into and out of the MT1300 using socat, both on the Wan side and on the Lan side, the byte counts on br-lan and br-wan increase as you’d expect. But, if I place a laptop on the Wan side and use socat to piipe data from the Lan side to the Wan side, using the MT1300 as a gateway, the MT1300’s traffoc stats don’t increase. So the problem seems to be restricted to times when the MT1300 is used as a gateway, and not to all network traffic flowing into and out of the device.

How can I make Linx report accurate traffic stats and use them in packet-shaping when the MT1300 is routing?

markus.laker · August 7, 2021, 3:57pm

A bit more experimentation:

I temporarily connected a laptop to a spare port on the ADSL router (the box at the top of the diagram above). I checked that the Wi-Fi access port on the router was disabled (it was), and I disabled the Wi-Fi connection on the laptop. The only way packets could pass from the laptop to the main network was through the ADSL router, then through the MT1300, and then through the main network switch.

I used socat to set up a constant 100Mbit/sec transfer from my desktop machine (on the main network) to the laptop (connected to the ADSL router). I brought up a network monitor on the laptop. It showed a constant 11.8Mbyte/sec transfer in and about 260kbyte/sec out (presumably ACKs).

I ran tcpdump on the MT1300, specifying any interface of any. It showed a heavy stream of traffic coming into the machine on eth0, but it didn’t show the same traffic being passed on to the laptop.

I installed darkstat, and it didn’t show much traffic passing in either direction.

ip -s a showed heavy traffic through eth0 but no significant traffic through br-wan.

Unplugging either the Wan cable or the Lan cable on the MT1300 caused the data transfer to stop – the value on the laptop’s traffic meter fell to zero. Plugging the cable in quickly enough caused the transfer to resume, because TCP is clever like that.

Disabling the br-wan port caused (with ip link set br-wan down) caused the transfer to stop.

These two simple tests showed that the traffic was definitely passing through the MT1300, and indeed though the br-wan bridge. So I don’t understand why the traffic doesn’t show up in traffic stats or how to fix it.

markus.laker · August 7, 2021, 6:12pm

Researching further, I found this question https://forum.openwrt.org/t/what-is-eth0-1-and-eth0-2/82918 and this backgrounder https://openwrt.org/docs/guide-user/network/vlan/switch_configuration on the OpenWrt Web site. Does this mean that the MT1300 has only one CPU interface, connected to a switch? Does this in turn mean that qdiscs (Linux queue disciplines) can never get the information they need to run on an MT1300?

markus.laker · August 8, 2021, 6:25pm

I’ve made further progress. Recall that my Lan is on 192.168.0.0/24 and my ADSL router is on 192.168.1.1. Running

tcpdump -U -i eth0 'not net 192.168.0'

shows that outgoing traffic is sent from eth0 (and not br-wan, and not eth0.2). So the problem isn’t that traffic stats are wrong, as I thought: the problem is that most traffic doesn’t go out on the interface I expected.

I may be able to do packet-shaping on eth0 if I start with a filter that lets everything destined for my Lan, i.e. 192.168.0.0/24, go out to the the network at lightspeed, and then I packet-shape everything else (i.e. all packets destined for the Internet) as I originally planned to.

I haven’t time to get this working today, but I’ll post back with a packet-shaping script if I succeed. I’ll also edit the subject line of this thread if the board lets me.

I still wouldn’t mind knowing why most of my outgoing traffic goes out on eth0 and not br-wan.

markus.laker · August 9, 2021, 4:50pm

A friend and I solved this together. The problem was that flow offloading was turned on by default when I bought the MT1300, and flow offloading is incompatible with QoS and SQM.

It’s easy to turn off software offloading in LuCI: go to “Network: Firewall” and unset “Software flow unloading”.

JerryZhao · August 11, 2021, 3:23am

This is due to we open the offload by default.
root@GL-MT1300:~# uci show firewall.@defaults[0]
firewall.cfg01e63d=defaults
firewall.cfg01e63d.syn_flood=‘1’
firewall.cfg01e63d.input=‘ACCEPT’
firewall.cfg01e63d.output=‘ACCEPT’
firewall.cfg01e63d.forward=‘REJECT’
firewall.cfg01e63d.flow_offloading=‘1’
firewall.cfg01e63d.flow_offloading_hw=‘1’

elorimer · August 11, 2021, 5:54pm

Thanks. And look, it says right there, “Not fully compatible with Qos and SQM”.