How-To: Installing Vanilla OpenWRT on GL X3000

Technical Support for Routers
1

Recommendations:
In case your modem’s module firmware does not run the latest Quectel firmware, I highly recommend to flash it to the latest stable version RM520NGLAAR03A03M4G using the steps shown below. Please note that this is not the router’s OpenWRT firmware, rather the module one. A full reference can be found at: Upgrade Quectel Module Software - GL.iNet Router Docs 4

  • To upgrade the Quectel modem firmware, SSH into the modem as root, and issue the following commands. Please verify that the calculated md5 hashes of the downloaded Qfirehose and the firmware zip file match the ones shown below within the lines!
opkg update && opkg install unzip
cd /usr/bin/ 
wget https://fw.gl-inet.com/tools/quectel_tool/QFirehose-mtk7981a-sha256-c0b944
chmod 775 QFirehose-mtk7981a-sha256-c0b944 
md5sum QFirehose-mtk7981a-sha256-c0b944 
42c934c8be6c860afc8b08c1cdb3fffd  QFirehose-mtk7981a-sha256-c0b944
wget https://fw.gl-inet.com/download/RM520GL-modem_firmware/RM520NGLAAR03A03M4G_01.201.01.201.zip -P /
md5sum  /RM520NGLAAR03A03M4G_01.201.01.201.zip 
6deb45893ce2088729708c9790e4d742  /RM520NGLAAR03A03M4G_01.201.01.201.zip
unzip /RM520NGLAAR03A03M4G_01.201.01.201.zip -d /RM520NGLAAR03A03M4G_01.201.01.201
QFirehose-mtk7981a-sha256-c0b944 -f /RM520NGLAAR03A03M4G_01.201.01.201
echo 1 > /sys/devices/platform/11280000.pcie/pci0001:00/0001:00:00.0/0001:01:00.0/remove
echo 1 > /sys/devices/platform/11280000.pcie/pci0001:00/0001:00:00.0/rescan
gl_modem AT AT+QGMR
rm /RM520NGLAAR03A03M4G_01.201.01.201* -rf
reboot
  • Now, familiarize yourself also with the Uboot flashing method, which you will need to use when flashing the downloaded vanilla OpenWRT firmware and the GL’s stock one: Debrick via Uboot - GL.iNet Router Docs 4

Downloading and Flashing a Vanilla OpenWRT firmware on a GL-X3000 router:

  • Go to the URL: OpenWrt Firmware Selector
  • Click on Customize installed packages and/or first boot script and insert the following packages in addition to the listed ones:

kmod-usb-serial-qualcomm luci-proto-mbim luci-proto-modemmanager luci-proto-qmi luci-ssl minicom kmod-usb-net-cdc-mbim picocom umbim usb-modeswitch pciids pciutils usbids usbmuxd kmod-usb-acm kmod-usb-ehci kmod-usb-net-huawei-cdc-ncm kmod-usb-net-ipheth kmod-usb-net-rndis kmod-usb-ohci kmod-usb-serial-ch341 kmod-usb-serial-ch348 kmod-usb-storage-uas kmod-usb-uhci kmod-usb2

  • Click on REQUEST BUILD and wait till the build is successful.
  • Once the build is finished, download it to your your computer by clicking on SYSUPGRADE.
    • You may also want to verify the downloaded image sha256sum hash prior to flashing it!
  • The computer at this stage should be connected to the router using an Ethernet cable!
  • Flash the router using the Uboot method:
    • Power off the router by unplugging the power cord
    • Press and hold the reset button while powering on the router
    • Wait for the Internet led to blink 5 times
    • Release the reset button
    • The Uboot rescue page is now accessible via http://192.168.1.1
      • Make sure your computer’s IP/MASK is configured correctly as per the Uboot instructions!
    • Select the OpenWrt sysupgrade image and start the upgrade process
    • Wait for the router to finish flashing and rebooting ...
  • Login to LuCI at https://192.168.1.1/cgi-bin/luci/ and set a password for root.
  • SSH into the modem as root, and execute the following commands:
echo -e "AT+QCFG=\"usbnet\",0\r\n" > /dev/ttyUSB2
echo -e "AT+QCFG=\"data_interface\",0,0\r\n" > /dev/ttyUSB2
reboot
  • Next browse to https://192.168.1.1/cgi-bin/luci/admin/network/network and create an Internet connection using the following steps:
    • Click on Add new interface
    • Choose any name for the connection and set the protocol to QMI Cellular and then click Create interface
    • set Modem device to /dev/cdc-wdm0
    • Enter your APN in the correspondence field
    • Click on the Firewall Settings tab and then set Create / Assign firewall-zone to WAN
    • Click Save and then Save & Apply
    • Refresh the page and now you should now see an established connection. Congrats!

Yet, with the vanilla OpenWRT, the maximum connection speed I could reach is 300-350 Mbps . By contrast, I was getting over 800 Mbps on the GL.iNet firmware – that is more than double the speed! The reason is that GL.iNet uses a proprietary Quectel protocol/drivers that utilize the modem’s PCIe over MHI.

“QMI: A protocol for controlling the modem that is exactly the same as for the USB variant but is instead routed over PCIe/MHI.”

Although a vanilla OpenWRT has some pcie-mhi drivers, I could not get them to work at all after tinkering for a few days! Thus, if you want to triple your connection speed, you need to compile these proprietary drivers by yourself and then install it in your vanilla installation. The next section is for advanced users!

Compiling Quectel PCIe-MHI Drivers (advanced users):
After you have successfully installed a vanilla OpnWRT, perform the following steps in sequence to avoid compiling errors and other nasty issues:

  • You need to use any Debian variants (Mint/Ubutrnu ...etc) to compile the required drivers/packages. Note, you may use a virtual machine, but it’s going to be very slow. Let’s first install the prerequisites:
sudo apt update
sudo apt install build-essential clang flex bison g++ gawk gcc-multilib g++-multilib gettext git libncurses5-dev libssl-dev python3-setuptools rsync swig unzip zlib1g-dev file wget
  • Preparing the build system:
git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git pull
git checkout master
./scripts/feeds update -a 
./scripts/feeds install -a
./scripts/feeds install libpam liblzma
./scripts/feeds install -a
sed -i -e 's/libpcre/libpcre2/g' package/feeds/telephony/freeswitch/Makefile
./scripts/feeds install -a
wget https://downloads.openwrt.org/snapshots/targets/mediatek/filogic/config.buildinfo -O .config
make defconfig
make menuconfig
  • Go to Target Profile and select GL.iNet GL-X3000
    ◦ hit Esc twice and when prompted to save the new configurations choose Yes and hit Enter.
  • Now lets build the toolchain – BE PATIENT – it takes about 30 minutes to finish:
make -j $(($(nproc)+1)) toolchain/install
make -j $(($(nproc)+1)) target/linux/compile

unzip Quectel_Linux_PCIE_MHI_Driver_V1.3.7.zip -d  package/kernel/
unzip Quectel_QConnectManager_Linux_V1.6.7.zip -d package/utils/
  • Make sure the new packages are properly selected in the menu:
    make menuconfig
    ◦ Enter the Utilities menu and select the quectel-CM-5G by hitting space
    ◦ Then enter the Kernel modules > Wireless Drivers menu and select kmod-pcie_mhi by hitting space
    ◦ Both selected packages should have the symbol <M>
    ◦ hit Esc twice and when prompted to save the new configurations choose Yes and hit Enter.
  • Compiling the packages:
make package/utils/quectel-CM-5G/{clean,compile} V=sc
make package/kernel/pcie_mhi/{clean,compile} V=sc
  • If you have successfully compiled both of them without errors, they should be located at:
ls $HOME/openwrt/bin/packages/aarch64_cortex-a53/base/quectel-CM-5G*.ipk
ls $HOME/openwrt/bin/targets/mediatek/filogic/packages/kmod-pcie_mhi_*.ipk
  • Let’s copy both packages to the X3000 device:
scp $HOME/openwrt/bin/packages/aarch64_cortex-a53/base/quectel-CM-5G*.ipk root@192.168.1.1:/root
scp $HOME/openwrt/bin/targets/mediatek/filogic/packages/kmod-pcie_mhi_*.ipk root@192.168.1.1:/root
  • SSH into the X3000 router as root and and install both:
opkg install quectel-CM-5G_1.6.7-r1_aarch64_cortex-a53.ipk
opkg install kmod-pcie_mhi_6.6.43.1.3.7-r1_aarch64_cortex-a53.ipk
  • reboot the device.
  • Once the device is rebooted, SSH into the box and create a Cellular connection using the Quectel connection manager tool and you correct string of your ISP:
    quectel-CM -4 -n 1 -s <apn> &
  • If you see that udhcpc has obtained an IP from the ISP, then you have successfully established connection!
  • To test the Internet accessibility, add the new interface to the WAN zone and insert a DNS server into the correct file:
sed -i -e "/[[:blank:]]*option[[:blank:]]*name[[:blank:]]*'wan'.*/a\\\tlist device 'rmnet_mhi0.1'" /etc/config/firewall 
service firewall restart
echo 'nameserver 1.1.1.1' >  /tmp/resolv.conf.d/resolv.conf.auto
echo 'nameserver 1.1.1.1' >  /tmp/resolv.conf
  • Perform a speed test few times and write down the maximum speed you get. We will significantly improve the speed by the next tweaks as there are a lot of dropped packets behind the scene!
  • We still need to tweak few things more to prevent packets from being dropped (a bug in the pcie_mhi driver when using HW offloading) as well as improve the speed of the Internet connection on a vanilla OpneWRT:
echo 'pcie_mhi poll_weight=128 debug_mode=0 mhi_mbim_enabled=0 bridge_mode=0 qmap_mode=1' > /etc/modules.d/90-pcie_mhi
echo -e 'net.core.rmem_max = 16777216\nnet.core.wmem_max = 16777216\nnet.ipv4.tcp_rmem = 4096 262144 21367520\nnet.ipv4.tcp_wmem = 4096 24576 21367520\nnet.core.netdev_max_backlog = 4096\nnet.core.somaxconn = 8192' >>  /etc/sysctl.conf
sed -i -e 's/\/tmp\/resolv.conf.d\/resolv.conf.auto/\/etc\/dnsmasq.servers/'  /etc/config/dhcp
echo 'nameserver 1.1.1.1' >  /etc/dnsmasq.servers
  • Login into LuCI web interface and enable the use of all cpu cores as well as Hardware flow offloading:
    image
    image1846×1446 183 KB

    image
    image1400×1092 111 KB
  • reboot the router.
  • SSH into the router and start the connection again:
    quectel-CM -4 -n 1 -s <apn> &
  • Perform speed test again and you should get triple the speed without any packets being dropped! To verify whether there are dropped packets, run ifconfig while doing repetitive speed tests.
  • To disable IPv6 stack, add the following lines into the /etc/rc.local file and reboot:
sysctl -w net.ipv4.conf.all.send_redirects=0
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

Note! If you do not use the WAN port (eth0) of the GL-X3000, which is a 2.5 Gbps, then you can get even a better speed by utilizing it. To do so, remove the port from the WAN and add it to the bridge.

Re-flashing the stock GL.iNet firmware:
These steps are needed when you want to revert back to stock GL.iNet firmware.

gl_modem AT 'AT+QCFG="pcie/mode",0'
gl_modem AT 'AT+QCFG="data_interface",1,0'
gl_modem AT 'AT+QCFG="usbnet",0'
  • Afterwards, you can upgrade again to latest GL.iNet firmware. Thanks to ywp of GL.iNet for the recovery hint!

Quectel_QConnectManager_Linux_V1.6.7.zip (227.3 KB)
Quectel_Linux_PCIE_MHI_Driver_V1.3.7.zip (142.2 KB)

8 Likes
2

The speed I got after performing the above steps:

image
image1856×1446 150 KB

image
image2600×838 193 KB

I am sure there are more room for tweaks and optimizations but I don't really have a time for it right now;)

4 Likes
3

https://dl.gl-inet.com/router/x3000/
4.4.10 pcie Verification has been Removed
It's currently in beta, but should it be okay

1 Like
4

This is great news so people can revert back to this version without experiencing the endless reboot loop👍

5

Out of interest, when you tried this, were the symptoms you saw that the /dev/wwan0mbim0, /dev/wwan0qcdm0 and /dev/wwan0qmi0 character devices were correctly created, but the corresponding wwan0 network device didn't get created?

If so, I've just reported exactly this behaviour on the linux-mhi list with current mainline linux and the upstream mhi wwan drivers. It happens on both 6.10.2 and the head of Linus' tree as of last night. I'll let you know if I get it debugged, because the fix will presumably be identical for openwrt kernels.

Amusingly, I can bring up the modem fine, getting IPv4 and IPv6 address, default routes and nameservers, but there's no network device for me to configure them on! USB mbim works fine, and I'm lucky to get 80Mbps out here in the sticks, so the lack of PCIe isn't a bottleneck that's urgent for me to fix, but it's on my 'idle project' list. :slight_smile:

6

Yes. I spent days trying to get it work using ModeManager and qmicli. All tries have failed .

Please keep me posted if you managed to get it work with the Linux stock drivers. I should point out that there are currently two minor issues with Quectel drivers: if a speed test reach around 800-900 mbps then some packets begin to get dropped. And the latency is 20-24 compared to 14 on openwrt driver.

Same thing I have experienced.

Yeah but for me it’s an issue; because I can reach on my iPhone using the same sim and spot 1.2 gbps. So getting right now 800 to 900 mbps using compiled drivers is an achievement!

7

The thread on linux-mhi is PCIe MBIM modem not creating a wwan netdev in case you want to follow it. I'll reply here to let you know if/when I get it going properly too.

I will try not to be too jealous of your 5G bandwidth! :rofl:

8

Thanks for sharing the thread. I read it all but I see no one has responded to you yet.

Haha :joy: but too much radio waves in the home bother me :joy:

9

If you own x3000 then you can compile Quectel drivers and investigate the dropped packets :disguised_face:

10

For those who are still experiencing dropped packets, please add the following kernel parameters to your /etc/sysctl.conf


# http://www.nateware.com/linux-network-tuning-for-2013.html
# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960

# cloudflare uses this for balancing latency and throughput
# https://blog.cloudflare.com/the-story-of-one-latency-spike/
net.ipv4.tcp_rmem = 4096 1048576 2097152

net.ipv4.tcp_wmem = 4096 65536 16777216

# Also increase the max packet backlog
net.core.netdev_max_backlog = 100000
net.core.netdev_budget = 50000

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

# If your servers talk UDP, also up these limits
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192

source: Alarm "system.softnet_stat" is very strict. · Issue #1076 · netdata/netdata · GitHub

11

Even if the wwan0 were created, would you be able to connect using PCIe over MHI? I checked Quectel forum and their engineers mentioned that a connection can’t be made unless using their tool quectel connect manager ?

12

Yes, I can bring the modem up by MBIM over /dev/wwan0mbim0 just fine: it negotiates v4 and v6 addresses, routers and nameservers. It's very close to working over PCIe: the control channels work correctly exactly like USB, but the mbim data channel isn't bound for some reason - hence the lack of netdev. There's specific support for RM520N in the mhi pci generic driver too, so it's probably something quite minor, but I've not meddled with mhi much and haven't found time to hack more at this myself yet I'm afraid, given that I'm limited by reception before USB2 bandwidth.

Generally low-quality manufacturer support staff do say that kind of thing (if they understand the question at all): "you have to use our cobbled together pile of crap to use our hardware rather than a clean upstream driver" and so on. The reality is their out-of-tree drivers remain out-of-tree for a reason - usually shocking code quality!

13

Agree :+1: . I am reading the source code of quectel driver and editing a lot of things ! I managed to get 1.1 gbps hahaha I am trying to match that of iPhone speed, specifically Qualcomm modem.

14

I don't know if there's a community of PCIe modem users+developers elsewhere in the OpenWrt world who might be worth you chatting with about the mainline driver? They may have a little more clue than me about what the right early steps to debug are. I'm starting from a standing start so to speak, and still have to get up to speed with mhi basics. I'm fitting around other projects too so it may take a while for me to get properly around to it. For what it's worth, the SDX65 in the RM520N is extremely mainstream, so lots of people will want to use them.

For me the main impetus for seeing PCIe work is just to verify that my tinymbim tool does the correct thing with deducing the primary wwan network interface name from the character device by looking in /sys/dev/char/MAJ:MIN/device/device/net for the interface. I've never been 100% happy with this and it'll report all the network interfaces rather than just the primary with multiple data connections, but I don't know of a better method.

15

A quiet Sunday and the cows were behaving themselves so I debugged it. The RM520N-GL in the X3000 has a different PCI id than the kernel is expecting, so it ends up with the wrong mhi channel config by default. The following works for me and gets it working with the mainline driver:

diff --git a/drivers/bus/mhi/host/pci_generic.c b/drivers/bus/mhi/host/pci_generic.c
index 08844ee7..7043d366 100644
--- a/drivers/bus/mhi/host/pci_generic.c
+++ b/drivers/bus/mhi/host/pci_generic.c
@@ -620,6 +620,9 @@ static const struct pci_device_id mhi_pci_id_table[] = {
 		.driver_data = (kernel_ulong_t) &mhi_telit_fn980_hw_v1_info },
 	{ PCI_DEVICE(PCI_VENDOR_ID_QCOM, 0x0306),
 		.driver_data = (kernel_ulong_t) &mhi_qcom_sdx55_info },
+	/* RM520N-GL variant with Qualcomm vendor and subvendor ID */
+	{ PCI_DEVICE_SUB(PCI_VENDOR_ID_QCOM, 0x0308, PCI_VENDOR_ID_QCOM, 0x5201),
+		.driver_data = (kernel_ulong_t) &mhi_quectel_rm5xx_info },
 	/* Telit FN990 */
 	{ PCI_DEVICE_SUB(PCI_VENDOR_ID_QCOM, 0x0308, 0x1c5d, 0x2010),
 		.driver_data = (kernel_ulong_t) &mhi_telit_fn990_info },

(You might need to apply by hand to an older kernel as the diff context may be wrong if other cards have been added between 6.6.x and 6.10.x. The only thing that really matters is that the PCI_DEVICE_SUB() entries for PCI_VENDOR_ID_QCOM, 0x0308 (like this one) need to come before the default PCI_DEVICE() entry for PCI_VENDOR_ID_QCOM, 0x0308 as they're the more specific matches.)

I haven't done much testing with it yet, but unmodified tinymbim brought it up with both IPv4 and IPv6 exactly as with the USB interface. Probably means umbim will be fine with it too.

[Edit] I'm using the PCIe mode for my internet access as I write this now. Speed is identical to USB, just over 100Mbps because I'm limited by 4G reception. No obvious packet loss but then my maximum download speed is unlikely to stress anything.

I saw regular warnings of the form sequence number glitch prev=38 curr=0 (for various values of prev, but curr is always zero) which I will probably suppress. These are discontinuities in a sequence field only intended for debugging purposes. Nothing relies on it and the USB cdc-ndm driver logs the same issue with debug severity instead of warning. We should probably do the same in mhi_wwan_mbim. I've included this in the update patch I pushed to GitHub - arachsys-hosts/gl-x3000: Build recipe for GL-X3000 router.

2 Likes
16

This is great news! Thanks for sharing the patch. To test PCIe over mhi what are the required drivers I need besides the generic one ?

17

Ah, that's a good point, it isn't obvious at all and I puzzled over it when I was trying it out first time. In the end, I had to compile in

CONFIG_MHI_BUS=y
CONFIG_MHI_BUS_PCI_GENERIC=y
CONFIG_WWAN=y
CONFIG_MHI_WWAN_CTRL=y
CONFIG_MHI_WWAN_MBIM=y

but you don't need CONFIG_MHI_NET for MBIM wwan.

The WWAN_CTRL is what gives you the /dev/wwan0mbim0, etc. character devices, and the WWAN_MBIM is what gives you the network interface.

18

The reason I am asking because I am trying to build only those drivers that are needed; not the whole Linux as it takes time.

I will also try MBIM to be able compare it with QMI's speed. So, probably I will compile all of them at once.

19

I have edited teh generic driver based on your commit. Then I have loaded all the following drivers. Yet, I could not connect either using MBIM or QMI.

mhi.ko
mhi_pci_generic.ko
mhi_net.ko
mhi_wwan_ctrl.ko
mhi_wwan_mbim.ko

I have these interfaces created:

mhi_hwip0 
mhi_swip0

Please note I am using PCIE mode not USB.

Can you please share the details of how you manged to connect? Thanks!

20

Those are exactly the symptoms I used to get before the patch, when the modem was being recognised as

mhi-pci-generic 0000:01:00.0: MHI PCI device found: qcom-sdx65m

instead of

mhi-pci-generic 0000:01:00.0: MHI PCI device found: quectel-rm5xx

I wonder if your has a different PCI id again from mine? What does lspci -v show on your router?

When it's working correctly, you'll get a wwan0 netdev but neither of the mhi_hwip0 or mhi_swip0 netdevs. In that case, you drive it via wwan0 and /dev/wwan0mbim0 in exactly the same way you would in USB mode.