I have a Samsung USB flash drive tested to achieve these speeds when used directly;
110.5 MB/s Write
381.3 MB/s Read
When Testing via SMB on the MT2500, I get these speeds
29.4 MB/s Write
49.8 MB/s Read
Anyone that can confirm this is due to CPU bottleneck ?
Samba isn't designed to run on lower power hardware. So yep, I would say this is the problem.
I could swear I've seen better performance when using Samba to Brume 2 with a USB HDD attached than what you're seeing, but I haven't had the time to do any rigorous testing which would eliminate interference from buffering, etc.
In my case I found, when initially testing with scp and rsync via ssh, that encryption was causing a CPU bottleneck, and switching to Samba resulted in noticeably higher performance, so I stopped testing at that point.
Out of curiosity, I just did a backup of a VM big enough to exceed the size of any buffer from a Proxmox host to Brume 2 (running firmware 4.5.16) with attached USB HDD via Samba share. The two machines are connected via wired ethernet.
Result: the transfer rate was about 83 MB/sec.
So, I would suspect that something else may be going on in OP's case.
Did some further testing on this, I have formatted the USB to other filesystems (FAT32 for example) and done local test from the terminal to take SMB and Ethernet out of the picture.
On my PC
Testing write throughput...
Write throughput: 32 MB/s
Testing read throughput...
Read throughput: 278 MB/s
On the Router
Testing write throughput...
Write throughput: 30 MB/s
Testing read throughput...
Read throughput: 73 MB/s
From dmesg on the router
[ 7176.019764] usb 2-1: new SuperSpeed Gen 1x2 USB device number 3 using xhci-mtk
[ 7176.045364] usb-storage 2-1:1.0: USB Mass Storage device detected
[ 7176.053205] usb-storage 2-1:1.0: Quirks match for vid 090c pid 1000: 400
[ 7176.060030] scsi host0: usb-storage 2-1:1.0
[ 7177.477940] scsi 0:0:0:0: Direct-Access Samsung Flash Drive 1100 PQ: 0 ANSI: 6
[ 7177.486861] sd 0:0:0:0: [sda] 125313283 512-byte logical blocks: (64.2 GB/59.8 GiB)
[ 7177.494691] sd 0:0:0:0: [sda] Write Protect is off
[ 7177.499491] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 7177.504674] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 7177.530316] sda: sda1 sda2
[ 7177.535175] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 7178.176085] FAT-fs (sda2): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Plugged in a much faster drive and can't seem to get past 73MB/s
Testing write throughput...
Write throughput: 73 MB/s
Testing read throughput...
Read throughput: 73 MB/s
From dmesg on the router
[ 8634.095554] usb 2-1: new SuperSpeed Gen 1x2 USB device number 4 using xhci-mtk
[ 8634.121160] usb-storage 2-1:1.0: USB Mass Storage device detected
[ 8634.127601] usb-storage 2-1:1.0: Quirks match for vid 090c pid 1000: 400
[ 8634.134449] scsi host0: usb-storage 2-1:1.0
[ 8635.557400] scsi 0:0:0:0: Direct-Access Samsung Flash Drive 1100 PQ: 0 ANSI: 6
[ 8635.566444] sd 0:0:0:0: [sda] 501253132 512-byte logical blocks: (257 GB/239 GiB)
[ 8635.574080] sd 0:0:0:0: [sda] Write Protect is off
[ 8635.578879] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 8635.584064] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 8635.600357] sda: sda1 sda2 sda3 sda4
[ 8635.606265] sd 0:0:0:0: [sda] Attached SCSI removable disk
How do you test on router / PC?
This shell script I put on the USB and run from terminal
#!/bin/ash
# Set the file name and maximum size
file_name="test.file"
max_size=1073741824 # 1GB in bytes
max_count=1024 # 1024 blocks of 1MB each
# Function to test write throughput
test_write_throughput() {
echo "Testing write throughput..."
start_time=$(date +%s)
dd if=/dev/zero of="./$file_name" bs=1M count=$max_count conv=fsync 2>/dev/null
end_time=$(date +%s)
elapsed_time=$(expr "$end_time" - "$start_time")
data_written=$(expr $max_count \* 1) # Data written in MB
write_speed=$(expr $data_written / "$elapsed_time")
echo "Data written: $data_written MB (Elapsed time: $elapsed_time seconds)"
echo "Write throughput: $write_speed MB/s"
}
# Function to test read throughput
test_read_throughput() {
echo "Testing read throughput..."
start_time=$(date +%s)
dd if="./$file_name" of=/dev/null bs=1M iflag=direct 2>/dev/null
end_time=$(date +%s)
elapsed_time=$(expr "$end_time" - "$start_time")
data_read=$(expr $max_count \* 1) # Data read in MB
read_speed=$(expr $data_read / "$elapsed_time")
echo "Data read: $data_read MB (Elapsed time: $elapsed_time seconds)"
echo "Read throughput: $read_speed MB/s"
}
# Test write throughput
test_write_throughput
# Test read throughput
test_read_throughput
# Clean up the test file
rm "./$file_name"
I'm starting to think the USB controller is limited on this device ~584mbps
I pasted the shell script into my router and tested it on the aforementioned USB HDD:
# ./test.sh
Testing write throughput...
Data written: 1024 MB (Elapsed time: 5 seconds)
Write throughput: 204 MB/s
Testing read throughput...
Data read: 1024 MB (Elapsed time: 4 seconds)
Read throughput: 256 MB/s
I then modified it to write 4GB instead of 1GB, just to be absolutely sure to exceed any possible buffer size on Brume 2:
# ./test.sh
Testing write throughput...
Data written: 4096 MB (Elapsed time: 17 seconds)
Write throughput: 240 MB/s
Testing read throughput...
Data read: 4096 MB (Elapsed time: 16 seconds)
Read throughput: 256 MB/s
Could the cable be the problem?
I don't remember the exact details, but don't USB 3.x controllers have a way to detect whether they're using a cable capable of 5Gbps, 10Gbps etc?? Maybe something is going awry with this.
Edit: and if you're using the same cable on PC and router, it could be that the cable is being misdetected, rather than it being defective or a USB 2 only cable.
Thanks for taking the time to run the tests on your device.
In my case, there are no cable used, it's a USB stick plugged straight in the USB port.
I wish I could achieve those speeds you're seeing.
I'm on the same version of firmware, I'm just wondering if there was a hardware revision ?
If you could also check the output of dmesg straight after plugging in the USB drive and compare.
[55608.218102] usb 2-1: new SuperSpeed Gen 1x2 USB device number 2 using xhci-mtk
[55608.243747] usb-storage 2-1:1.0: USB Mass Storage device detected
[55608.250170] usb-storage 2-1:1.0: Quirks match for vid 090c pid 1000: 400
[55608.257503] scsi host0: usb-storage 2-1:1.0
[55609.696084] scsi 0:0:0:0: Direct-Access Samsung Flash Drive 1100 PQ: 0 ANSI: 6
[55609.707386] sd 0:0:0:0: [sda] 501253132 512-byte logical blocks: (257 GB/239 GiB)
[55609.715204] sd 0:0:0:0: [sda] Write Protect is off
[55609.720047] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[55609.725430] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[55609.742456] sda: sda1 sda2 sda3 sda4
[55609.748753] sd 0:0:0:0: [sda] Attached SCSI removable disk
From this I can see it's connecting at USB3.0 Gen1 speeds 5 Gbps, and driver xhci-mtk may be causing the issue ?
It would be interesting to see if you have some message similar or additional messages like this
usb 2-1: SuperSpeed enabled
Just remembered I have a second Brume 2, will test on that one too.
Tried another brand USB Stick
[69651.252081] usb 2-1: new SuperSpeed Gen 1x2 USB device number 3 using xhci-mtk
[69651.278277] usb-storage 2-1:1.0: USB Mass Storage device detected
[69651.284802] scsi host0: usb-storage 2-1:1.0
[69652.308230] scsi 0:0:0:0: Direct-Access UFD 3.0 Silicon-Power128 PMAP PQ: 0 ANSI: 6
[69652.318026] sd 0:0:0:0: [sda] 246546432 512-byte logical blocks: (126 GB/118 GiB)
[69652.326337] sd 0:0:0:0: [sda] Write Protect is off
[69652.331138] sd 0:0:0:0: [sda] Mode Sense: 2b 00 00 08
[69652.336999] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[69655.369745] sda: sda1 sda2
[69655.376636] sd 0:0:0:0: [sda] Attached SCSI removable disk
And I'm getting better speeds now
Testing write throughput...
Data written: 1024 MB (Elapsed time: 20 seconds)
Write throughput: 51 MB/s
Testing read throughput...
Data read: 1024 MB (Elapsed time: 5 seconds)
Read throughput: 204 MB/s
So maybe it doesn't like the Samsung USB Drives with message Quirks match for vid 090c pid 1000: 400 ?
In my case I boot with the device already attached.
Early in the boot process:
[ 0.703984] xhci-mtk 11200000.xhci: xHCI Host Controller
[ 0.709305] xhci-mtk 11200000.xhci: new USB bus registered, assigned bus number 1
[ 0.719759] xhci-mtk 11200000.xhci: hcc params 0x01403f99 hci version 0x110 quirks 0x0000000000210010
[ 0.729005] xhci-mtk 11200000.xhci: irq 84, io mem 0x11200000
[ 0.735211] hub 1-0:1.0: USB hub found
[ 0.738991] hub 1-0:1.0: 1 port detected
[ 0.743150] xhci-mtk 11200000.xhci: xHCI Host Controller
[ 0.748469] xhci-mtk 11200000.xhci: new USB bus registered, assigned bus number 2
[ 0.755955] xhci-mtk 11200000.xhci: Host supports USB 3.2 Enhanced SuperSpeed
[ 0.763139] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[ 0.771562] hub 2-0:1.0: USB hub found
[ 0.775343] hub 2-0:1.0: 1 port detected
[ 0.779746] usbcore: registered new interface driver uas
[ 0.785145] usbcore: registered new interface driver usb-storage
a bit later:
[ 1.258665] usb 2-1: new SuperSpeed Gen 1x2 USB device number 2 using xhci-mtk
[ 1.285407] scsi host0: uas
[ 1.288959] scsi 0:0:0:0: Direct-Access Inateck ASM225CM 0 PQ: 0 ANSI: 6
[ 1.298912] sd 0:0:0:0: [sda] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 1.306749] sd 0:0:0:0: [sda] 4096-byte physical blocks
[ 1.312063] sd 0:0:0:0: [sda] Write Protect is off
[ 1.316853] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 1.322056] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.331197] xhci-mtk 11200000.xhci: ERROR Transfer event for unknown stream ring slot 1 ep 2
[ 1.339620] xhci-mtk 11200000.xhci: @00000000432315c0 7e203100 00000000 1a000200 01038001
[ 1.347893] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
[ 12.298330] sda: sda1
[ 12.301830] sd 0:0:0:0: [sda] Attached SCSI disk
I never did see a usb 2-1: SuperSpeed enabled message.
Interesting links which reveal that users have experienced poor performance when this quirk is active (and some ideas about how to disable the quirk, no idea if these methods will need to be modified for OpenWRT).
Note that depending on your particular USB drive, the quirk may be preventing some other bad behavior from the device; however, it would appear from these reports that some devices don't need it to be in place.
I have no idea if there's a fix for this in present-day kernels, which (if it exists) may not have made it into OpenWRT kernels yet.
https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1746447.html