In that case a don't know what else to tell you, I have op24 my wifi radios are working flawless and I test my 2.5 ports they working good too 
Well you have probably not understood the problem.
2.5G LAN works fine on op24 firmware. It’s the non op24 firmware that has issues.
Yeah I got you, and then your wifi it's mess up on op24 right?
Looks like I was wrong about the WiFi. I had 160 MHz DFS set previously (on non op24) and forgot about it after I installed op24 which explained the massive difference in speed. Once I enabled 160 MHz the speeds on WiFi 5G were “fixed”
Good,
I was wondering because I'm telling you, I'm more than happy with op24
Now on the other hand the close source one sucks big time for me and my network
Similarly strange, this is on 4.6.4-op24.
iperf3 from the Flint 2 to internet (4Gbit symmetrical, so given the ethernet ports on the Flint 2, the upper bound is 2.5Gbit/s):
# iperf3 -c speedtest.ams1.nl.leaseweb.net -p 5201-5210
Connecting to host speedtest.ams1.nl.leaseweb.net, port 5201
[...]
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.01 GBytes 1.73 Gbits/sec 3 sender
[ 5] 0.00-10.05 sec 2.01 GBytes 1.72 Gbits/sec receiver
# iperf3 -c speedtest.ams1.nl.leaseweb.net -p 5201-5210 -R
Connecting to host speedtest.ams1.nl.leaseweb.net, port 5201
Reverse mode, remote host speedtest.ams1.nl.leaseweb.net is sending
[...]
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.05 sec 1.41 GBytes 1.21 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.39 GBytes 1.20 Gbits/sec receiver
iperf3 from my Mac, going through the Flint 2 as the router (over ethernet, as @WizMeister already said, the WiFi speed was a red herring):
% iperf3 -c speedtest.ams1.nl.leaseweb.net -p 5201-5210
Connecting to host speedtest.ams1.nl.leaseweb.net, port 5201
[...]
[ 7] 0.00-10.01 sec 2.63 GBytes 2.26 Gbits/sec sender
[ 7] 0.00-10.01 sec 2.63 GBytes 2.25 Gbits/sec receiver
~ % iperf3 -c speedtest.ams1.nl.leaseweb.net -p 5201-5210 -R
Connecting to host speedtest.ams1.nl.leaseweb.net, port 5201
Reverse mode, remote host speedtest.ams1.nl.leaseweb.net is sending
[...]
[ ID] Interval Transfer Bitrate Retr
[ 7] 0.00-10.01 sec 2.69 GBytes 2.31 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 2.67 GBytes 2.29 Gbits/sec receiver
So, the Flint 2 is perfectly capable of routing close to 2.5Gbit (this is even over PPPoE with VLAN tagging), however performs worse when running iperf on the Flint 2 itself. This is not a huge issue of course, since we usually generate traffic from clients, but curious nonetheless.
Like I said upthread, I think this caused by something becoming CPU-bound. See the htop screenshot below made while running iperf. You can see iperf using roughly 80% CPU, but the load is much higher than just that iperf process. Also, all the bars are red, meaning that the time is spent in kernel threads. So I am still wondering if running iperf on the device itself somehow triggers a (routing?) path that is not hardware-accelerated.
2 Likes
The negotiated rate is 1200 Mbps. The actual speed is around 900 Mbps. That's already not bad.
1 Like
Still waiting on the SFP+ modules so can't test directly from this machine as, although it's connected to a 2.5Gbit port there's still only a 1Gbit link between this switch and the other one.
But I spun up a fedora rawhide machine to test something else (please excuse the screenshots I didn't have it setup for SSH/Remote access yet)
1 Connection
4 Connections
Reverse
1 Connection
4 Connections
This is inter vlan so it will go in/out the 2.5G port on the Flint2
Direct (I moved the VM onto the same VLAN as the Iperf3 server)
Librespeed tops out at around 1.8Gbit/s Down / 1.5 Up even when direct, however I think the Nuc hosting the vm runs out of CPU grunt (They're 4500u based systems)
Hmm I don't think network acceleration was properly enabled it was set to auto in the GL-inet ui but I think i'd actually disabled it when messing with LUCI settings
With it on I get 2.3Gbit/s but a load of retried/dropped packets
Edit:
After some more testing it looks like for whatever reason my offloading may have either been disabled or not working fully when I did the initial tests, I can pretty much max 2.5Gbit/s with it on/working.
Edit2:
Forget to say I'm on the 4.7 Beta.
I found your two comments quite hard to follow. What is the network layout? Where is the 1Gbit link (why are you getting more than 1GBit with a 1GBit link)?
Also, where is the iperf3 server running, on the Flint 2 or another machine?
Finally, are you trying to confirm what @WizMeister and I are seeing or do you see the full speed where we see reduced speed (when the router is the receiver)?
To summarize the findings so far.
- @WizMeister sees with the 4.6.4 (non-op24) and 4.7 beta when testing between the Flint 2 and a 2.5Gbe machine that is directly connected:
- Expected speed when the Flint 2 is the sender.
- Reduced speed when the Flint 2 is the receiver.
This issue does not occur on 4.6.4 op24.
-
I see a similar issue on 4.6.4 op24. It's the same testing setup with the modification that the other machine is now not on the LAN, but on the WAN (Internet):
- Expected speed when the Flint 2 is the sender.
- Reduced speed when the Flint 2 is the receiver.
In the reduced speed case, I also see a high CPU load in kernel threads.
I found your two comments quite hard to follow. What is the network layout? Where is the 1Gbit link (why are you getting more than 1GBit with a 1GBit link)?
Also, where is the iperf3 server running, on the Flint 2 or another machine?
Finally, are you trying to confirm what @WizMeister and I are seeing or do you see the full speed where we see reduced speed (when the router is the receiver)?
Edit: TLDR version: Router should be able to forward traffic at around 2.5Gbit/s providing at least Software Flow offloading is enabled.
If it's disabled then router seems to struggle to forward traffic as such speeds.
Sorry, I'll clarify
I have a cluster of 3 NUC machines running Proxmox connected to a 2.5G switch (which is itself connected to the 2.5G port in the flint2)
I was testing from a Fedora Rawhide VM on NUC1 to an iperf3 server in a NixOS VM on NUC2.
In order to make sure the traffic transits the Flint2 I put the Rawhide VM on a different VLAN to the NixOS VM requiring the traffic to transit the router (which in this case is the Flint2).
The "Direct" tests were where I switched the VLAN of the rawhide VM so it was on the same VLAN as the iperf3 VM meaning the traffic was simply able to transit the switch without touching the router.
I then moved the Rawhide VM back to the other VLAN disabled/re-enabled acceleration on the flint2 and was able to get ~2.3Gbit/s between VLANs, I think when I was playing around in Luci It had disabled HW-Accel but the GL-Inet UI was showing it as enabled, I suspect the lack of HW acceleration explains the sub 1Gbit results.
I was going to post some librespeed tests as well, but the Rawhide VM runs out CPU horsepower and bottlenecks.
Furthermore, I have some more 2.5Gbit capable devices one of which I'd have preferred to have used for the iperf3 client instead of a VM, but I can't at the moment as they're in a different room of the house and whilst I've also upgraded that switch to 2.5Gbit there's currently a 1Gbit Fibre link between that switch and the one connected to the flint2.
I'll be replacing this interswitch link with a 10Gbit link once the SFP+ modules I ordered to go with the 2.5Gbit switches arrive. (Only reason for going 10Gbit on that link is because the switch has a couple of 10G ports, and it's actually easier for me to get 10Gbit optics)
I haven't bothered doing Iperf3 to the router itself because I don't really see the point, it's a router the thing we should be concerned with is its packet forwarding performance. I'm also unsure if HW acceleration will be able to accelerate things when the router itself is the destination.
Thanks for the clarification! Yes, this works fine. Heck, with hardware acceleration it can even do PPPoE on top of using VLAN tagging (see above).
Agreed. I think we got here because @WizMeister doesn't have the configuration yet to test routing speeds. But I think testing the speed from/to the router is quite a common way of testing routers. Suppose that you don't get good routing speeds due to some issue, one way of narrowing the possible issue is to test router <-> LAN or router <-> WAN speed as a simplification. But due these issues (depending on the firmware version), you might get the impression that something is wrong with the router, whereas in reality the routing speed is fine (if you use hardware acceleration). So, I think it is either worth solving or putting this into the documentation somewhere if it is hard to solve.
The latter point brings me to another annoyance: very few router vendors list the possible/expected speeds. E.g. a lot of routers cannot do PPPoE at line speed and I only found out pre-sale that the Flint 2 can thanks to another member of this community.
That's good to know for if/when someone gets around to running fibre here as a lot of the ISP's still use PPPoE.
Although It's likely the fastest service I'll be able to get is 1.8Gbit down 125Mbit up (I'm still not sure if it's worth the extra cost vs 1Gbit/125)
When using iPerf3 to test wireless connections I've found that adding the "-w 4M" option yields much better results. Worth giving a try if you're not getting the wireless speeds you're expecting.
I’m still looking for a good, affordable router that can do more than 2.5Gbit PPPoE, since our current connection is 4Gbit symmetric. The only routers that I could find so far as candidates are the QHora301W/322 and BPi-R4, but I’m not sure they are as stable/well supported as the Flint 2, and I don’t want to put a x86 PC in our utility closet.
I was able to run routing speed tests on firmware 4.6.4-op24
The set up is:
Flint 2
└ --- 2.5G WAN port (set up as LAN) --> MacBook M1 Max with USBC 2.5Gbe NIC (192.168.1.104)
└ --- 2.5G LAN port --> Dell machine with 2.5G NIC (192.168.1.69)
Test A:
Iperf server: MacBook with USBC 2.5Gbe NIC (192.168.1.104)
Iperf client: Dell with 2.5Gbe NIC
If I run iperf3 -c 192.168.1.104 from the Dell machine (sender is Dell and receiver is MacBook) then I have I have expected speed (about 2.3Gbps)
If I run iperf3 -c 192.168.1.104 -R from the Dell machine (reverse; sender is MacBook and receiver is Dell) then I have slower speed (about 1.6Gbps)
Test B:
Iperf server: Dell with 2.5Gbe NIC (192.168.1.69)
Iperf client: MacBook with USBC 2.5Gbe NIC
If I run iperf3 -c 192.168.1.69 from the MacBook (sender is MacBook and receiver is Dell) then I have slower speed (about 1.6Gbps)
If I run iperf3 -c 192.168.1.69 -R from the MacBook (reverse; sender is Dell and receiver is MacBook) then I have expected speed (about 2.3Gbps)
It looks like even though 4.6.4-op24 works better for me when running iperf3 directly on the Flint, when routing, speeds are not bidirectional.
PS: I have network acceleration enabled and set to “Hardware”
I will be installing 1Gbps fiber at home (where the Flint 2 currently is) and I have also ordered a second Flint 2 for my office space where I already have 1Gbps fiber currently handled (PPPoE over VLAN) by an ASUS RT-AX53U / RT-AX1800U to which I have installed OpenWRT. Given the fact that the ASUS gives me about 850Mbps with software flow offloading (no hardware support AFAIK) along with a secondary VLAN for VOIP, I believe the Flint 2 will not even break a sweat.
Having said that, I’m worried because it is marketed as supporting 2 x 2.5Gbe but lack of bidirectional performance is a bummer.
This is really weird, since my test in Flint 2 (MT6000) 2.5Gbe issues - #28 by nonagoninf was with
Mac Studio (PCIe 10Gbe NIC)
<-> Zyxel 2.5Gbe switch
<-> Flint2
<-> Genexis XGS-PON ONT
<-> ISP
<-> speedtest.ams1.nl.leaseweb.net
So many more hops than your setup and ~2.3Gbit in both directions.
Let me test from a MacBook in both directions.