Disclaimer: I'm not a network expert and I don't own many devices for more thorough test, but I'll describe problem as much as I can.
I'm currently using Beryl7 as a home router in repeater mode. My network consists of my laptop (connected via WiFi) and my NAS (connected via LAN port).
While pushing ~1.5TB files to my NAS I noticed, that few files can't be copied (process hangs for a minute and reports a timeout error). Error is reproducible for same files at the same cutoff point. Later I was able to isolate small chunk of the file (~200kb) which repeatedly fails for me.
Error persists for multiple interfaces on the NAS, multiple protocols (samba, nfs) and multiple clients in multiple operating systems (e.g. windows, linux).
Interesting fact 1: error vanished when I switched my NAS from LAN port to WAN port (wan configured as second LAN, as I'm using wifi-as-wan).
Interesting fact 2: error vanished when I initiated a copy from behind NAT, from a container in windows WSL2, which is not on the same network as host (byte stream was modified from behind nat?)
Originally tested on firmware 4.8.7. Tried to upgrade to 4.9.0 beta3 -- error persists.
Router syslog does not show anything.
I don't have multiple workstations, network adapters or anything else to test more. Hopefully this can be helpful to identify the culprit.
if it is not hw accerelation, can I ask if you use any ports with a different negotiation?
I don't think that 1gbps negotiations make a big issue, but I learned by tons of error fixing with googles ai that flowcontrol can be a very mean one, especially if one port is 100M is negotiated, you may want to disable flowcontrol via ethtool only for this port.
What the AI told me, was that the port can be overwhelmed and start sending pause frames, which happen to kill half of the internal switch, in my case the driver started to lock down because it received a packet with the same source address even when no connection crossed that port, the checksum became invalid due to how flow control handled it only at peak congestion.
Another issue I also found, was that some devices leverage their own propetairy offloading/flow control properties... Mediatek for example can do this to, and I learned on those newer B560 motherboards you may want to look into disabling the offloading, green ethernet, and the super packets (may named different), when I did this the network did not go down anymore, I had it with steam, and since steam uses multithreaded download very aggresively it only there crashed, it seems that some of these nic features can be really broken to routers.
My other question I have is, does the network go down completely?, can wan still get dhcp or fails that too in this state?, you mentoid you could connect to it... so it seems its not this issue I'm thinking off, did you mean that the port got changed randomly?
My NAS has 2.5G+5G ports, so it was always working in 2.5G mode (router native) on any port. During investigation I attempted to force negotiation gown to 1G on the NAS side – didn’t help.
>> My other question I have is, does the network go down completely?
Not the network, only the session is dropped. You’re copying the file then it becomes stuck and timeouts in a minute. You can re-connect and use connection as usual (with other files).
My problem is not random. It is repeatedly occurring when I try to transfer exactly this file and it fails at exactly same transfer progress. I have a few such files isolated. I tried to replace leading bytes with zeroes in the file – no effect. Then I cut out the small (~200kb) part of the original failing file – and it repeatedly fails on its own.
It looks, like when specific byte-sequence is transferred through the port – it crashes the session (that’s the best way I can describe the experience)
I don’t have any more devices to test how failing file is affected by other devices. Maybe the byte-sequence is dependent on the device fingerprint/hash/address or something else. All I know, is that the issue is not random, but 100% repeatable. And it goes away if I switch to using WAN port in LAN mode.
Hmm I'm interested into conntrack tracking to see if this somehow could affect this in your network.
If you are able to ssh in your router can you post the contents of:
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
When this happens?
If you still have access a look into conntrack -L can show also the probability cause if a device is creating alot of new connections very fastly.
In that case, the solution would be more to create either a firewall traffic rule to limit new connections per hour for this nas, or to not track the connection as action, if my founding is true then the nat table can become exhausted this then also drop packets.
I don’t know what more to tell you. Maybe I have a faulty unit. Or maybe issue depends on some environment state. Problem is persistent, not dependent on file size or file type. I faced the issue when copying >1.5TB of data from my laptop to NAS. Only 4 or 5 files failed. Different sizes and types. The attached 200kb chunk is a part of one such file, that fails on its own.
I also eliminated the laptop from the chain. I copied the file to the router over ssh and initiated a copy to NAS directly from router.
# cable is plugged into LAN PORT
root@GL-MT3600BE:/copy-failure-test# smbclient //192.168.8.112/downloads -U user%SECRET -c 'put /copy-failure-test/repro-fail.bin repro-fail.bin'
lpcfg_do_global_parameter: WARNING: The "null passwords" option is deprecated
cli_push returned NT_STATUS_IO_TIMEOUT
NT_STATUS_IO_TIMEOUT closing remote file \repro-fail.bin
# cable is unplugged from LAN port and plugged into WAN port
root@GL-MT3600BE:/copy-failure-test# smbclient //192.168.8.112/downloads -U user%SECRET -c 'put /copy-failure-test/repro-fail.bin repro-fail.bin'
lpcfg_do_global_parameter: WARNING: The "null passwords" option is deprecated
putting file /copy-failure-test/repro-fail.bin as \repro-fail.bin (17755.5 kb/s) (average 17755.7 kb/s)
