How-To: Installing Vanilla OpenWRT on GL X3000

lspci -v
00:00.0 PCI bridge: MEDIATEK Corp. Device 1f32 (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 81
	Memory at 20100000 (64-bit, non-prefetchable) [size=32K]
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: 20000000-200fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Capabilities: [80] Express Root Port (Slot-), IntMsgNum 0
	Capabilities: [e0] MSI: Enable+ Count=1/32 Maskable+ 64bit+
	Capabilities: [f8] Power Management version 3
	Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
	Capabilities: [110] L1 PM Substates
	Capabilities: [200] Advanced Error Reporting
	Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -2

01:00.0 Unassigned class [ff00]: Qualcomm Technologies, Inc Device 0308
	Subsystem: Qualcomm Technologies, Inc Device 5201
	Flags: bus master, fast devsel, latency 0, IRQ 87
	Memory at 20000000 (64-bit, non-prefetchable) [size=4K]
	Memory at 20001000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable+ Count=4/32 Maskable+ 64bit+
	Capabilities: [70] Express Endpoint, IntMsgNum 0
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [148] Secondary PCI Express
	Capabilities: [168] Physical Layer 16.0 GT/s <?>
	Capabilities: [18c] Lane Margining at the Receiver
	Capabilities: [19c] Transaction Processing Hints
	Capabilities: [228] Latency Tolerance Reporting
	Capabilities: [230] L1 PM Substates
	Capabilities: [240] Data Link Feature <?>
	Kernel driver in use: mhi_q

I am getting different interfaces, why do you think ?

The lspci output is good news. I was worried that your modem might have yet another different PCI product/subproduct id, but it's the same as mine: you can see the product id 0x0308 and sub-id 0x5201 so that patch is correct for your modem too.

I think that patch isn't taking effect for some reason. To confirm: what does

dmesg | grep mhi-pci-generic

return for you? When the patch is correctly applied, you should see

mhi-pci-generic 0000:01:00.0: MHI PCI device found: quectel-rm5xx

in there, whereas if it isn't applied or doesn't quite match your kernel, you'll get

mhi-pci-generic 0000:01:00.0: MHI PCI device found: qcom-sdx65m

Assuming you see the latter, please could I take a look at your drivers/bus/mhi/host/pci_generic.c to see if there's some reason it hasn't applied right on the older kernel? You can email me as chris@arachsys.com if that's easier.

@SpitzAX3000 this guide is great, thank you for putting it together, my concern is how friendly is going to be with vanilla OpenWrt with the Modem side of the router, signal status, information, querying the modem, change from NSA to SA, block/unblock bands, etc., are there LuCI modules or tools to help with that? Or everything has to be done via the command line? I think this is probably the main advantage of using the GLi.Net firmware.

It would be nice to have an updated guide bundling all the work/discovery/changes you have done with @ChrisW, unfortunately to me, it has become too technical and risky to apply into my X3000, because it is the main gateway with TMHI, and if I brick it in the process, I will be isolated.

Please keep updating and maintaining this thread, it's invaluable for all of us who want to use plain vanilla OpenWrt. My cascading router behind the X3000 who acts primarily as a modem, is an MT6000 running 23.05.4.

I haven't explored LuCI tools to view such info. But, there should be some tools. Sometimes it's in your advantages to have to set up the connection using CLI as these GUI packages cause sometimes issues (as we have seen with some of the GL.iNet firmware). Not to mention that using one or two commands to start the 5G connection isn't that difficult nor does it require technical knowledge.

This GitHub repo has some tools to allow controlling the modem through LuCI but I have not tried it: GitHub - Siriling/5G-Modem-Support: 5G模块支持

My guide should be sufficient to get you a very stable vanilla OpenWRT. Chris and I have been discussing something else: modifying the Linux stock driver to recognize the modem module in PCIE mode. Once Linux drivers are patched as per Chris's fix, you won't even need the part of compiling the drivers that is mentioned in my guide.

If you are happy with GL.iNet firmware and having few or negligible issues, then just stick with it.

Thank you for the feedback

Man, how slow are we talking here? I've got a small NUC, J4125, that I'm trying to compile it in in an LXC container on proxmox, and its been like 6 hours now and its still pegged on make -j $(($(nproc)+1)) toolchain/install. I don't have any big x86 machines anymore to run this on natively.

Yes, it’s gonna be very slow. I compiled it in NUC with Linux installed and took about 40 minutes.

If you don’t want to install Linux on your NUC, you can try to boot Ubuntu/Mint from a fast usb. Using containers and virtual machines won’t utilize all of your cpu’s cores.

Oh man, protip, make sure the kernel doesn't change between when you request your image from OpenWRT, and when you build the kernel packages.

Agree! If you install a vanilla openwrt then few days later you compile the drivers, it won’t work as the kernel versions are rapidly being pumped up !

Do I have to run quectel-CM -4 -n 1 -s <apn> & in a shell every time the router reboots?

Same with

sed -i -e "/[[:blank:]]*option[[:blank:]]*name[[:blank:]]*'wan'.*/a\\\tlist device 'rmnet_mhi0.1'" /etc/config/firewall 
service firewall restart
echo 'nameserver 1.1.1.1' >  /tmp/resolv.conf.d/resolv.conf.auto
echo 'nameserver 1.1.1.1' >  /tmp/resolv.conf

Yes, and don’t forget to add also the -d option. I found adding -d to be better than using udhcpc. Although you need to run it on boot, you can automate it rather than manually invoking it.

For the second command you mentioned, no need. Just go to dnsmasq settings and choose any desired location for your resolvers.

You can always put it in /etc/rc.local so it will be activated on each boot.

Thanks, I got it working in PCIe mode over MHI using MBIM. But, the QMI connects without getting an IP. I tried udhcp but no luck.

Also, I should mention that the number of dropped packets when doing speedtest is huge.

I guess it needs more debugging,

I think MBIM is the supported interface over PCIe MHI, although QMI can be tunnelled within MBIM for commands that aren't supported directly. Allegedly uqmi has the --mbim flag to support this, although I've never tried it as I've not needed anything more complicated than basic connect.

I can't investigate the packet drops myself I'm afraid - my 100Mbps mobile signal here is nowhere near fast enough to be able to stress the device! It's a little puzzling though: there shouldn't be that much overhead and the mt7981 is nice and fast.

[Edit: the modem may not support in-band address discovery with dhcpv4/6. Not all of them do. You might need to use the addresses returned by a BASIC_CONNECT_IP_CONFIGURATION, but I think the openwrt umbim netif script will do that out of the box.]

I have now managed to get both to work: MBIM and QMI, although the qmicli tool cannot make a connection when the modem in QMI mode; instead I am using mbimcli to connect.

when I have time, I will try to find out the cause of the dropped packets. Initially it seems to me because of the datagrams and max size configured by the driver:

# qmicli  -d /dev/wwan0mbim0  --wda-get-data-format
[/dev/wwan0mbim0] Successfully got data format
                   QoS flow header: no
               Link layer protocol: 'raw-ip'
  Uplink data aggregation protocol: 'mbim'
Downlink data aggregation protocol: 'mbim'
                     NDP signature: '5460041'
Downlink data aggregation max datagrams: '32'
Downlink data aggregation max size: '32768'

I was able to change these values using qmicli. But when the connection starts, they are return to defaults. If I change it on the fly after establishing a connection, the Internet stops working.

I'm intrigued that you're seeing packet loss with both drivers. I wonder if that means it's something intrinsic to the modem itself, or its default configuration?

(You don't get any kind of packet loss using the same measurement methodology over the ethernet ports I assume? Sounds like your mobile connection is practically the same speed as gigabit ethernet (!) so this isn't a completely unfair comparison.)

I am using the 2.5 Gbps Ethernet port on the Spitz.

Today morning, I got some times to troubleshoot it - by switching the congestion control (kmod-tcp-bbr package), packet drops are no longer:

net.ipv4.tcp_congestion_control = bbr

Yet, the connection speed got degraded from about ~900 Mbps to ~700 Mbps.

Hi Chris, have you managed to find the root cause of these msgs? It has filled up my logs.

Yes, this is a minor bug in the modem itself - the sequence numbers are only used for debugging so it's harmless if they're broken, but the driver warns anyway.

It happens on both usb cdc-wdm and pcie-mhi, but in usb mode it's been downgraded to a debug message (presumably because it's so common) whereas in pcie mode it gets reported more loudly to dmesg.

The latest version of my patch has

diff --git a/drivers/net/wwan/mhi_wwan_mbim.c b/drivers/net/wwan/mhi_wwan_mbim.c
index f2aef84f..26cbb477 100644
--- a/drivers/net/wwan/mhi_wwan_mbim.c
+++ b/drivers/net/wwan/mhi_wwan_mbim.c
@@ -209,7 +209,7 @@ static int mbim_rx_verify_nth16(struct mhi_mbim_context *mbim, struct sk_buff *s
 	if (mbim->rx_seq + 1 != le16_to_cpu(nth16->wSequence) &&
 	    (mbim->rx_seq || le16_to_cpu(nth16->wSequence)) &&
 	    !(mbim->rx_seq == 0xffff && !le16_to_cpu(nth16->wSequence))) {
-		net_err_ratelimited("sequence number glitch prev=%d curr=%d\n",
+		net_dbg_ratelimited("sequence number glitch prev=%d curr=%d\n",
 				    mbim->rx_seq, le16_to_cpu(nth16->wSequence));
 	}
 	mbim->rx_seq = le16_to_cpu(nth16->wSequence);

to fix this.

Thanks! I commented out the entire checking block, and recompiled the driver. But definitely, your fix by adding the dbg is more elegant!