Hello,
I put the latest stable release OpenWrt 23.05.2 onto my E750. Many things work fine, but I have one strange problem.
If I boot the device with the wifi off, then I enable the 5GHz radio (either STA or AP mode), everything works fine. But upon boot, if the 5GHz radio is already enabled, the device crashes.
To summarize: 5GHz wireless works fine, unless it’s enabled at boot time, in which case the router crashes and stays dead. I had to use the magic reset button to wipe the config and start from scratch to recover.
After some more searching, I think this is the same problem mentioned in this thread.
It seems this has been a problem for two years at this point.
@luochongjun Can we get a reply on this? Are you using a special patched version of the ath10k_pci driver or something that is different than mainline?
Thanks for your reply! I think however that this would just be trying to recover from the crash, and the best solution would be to avoid crashing in the first place.
I don’t understand most of it, but it sounds like they have identified the root cause – some improper value being passed to the driver. I would like to understand what is different between the stock and the GL.iNet version that makes one work and not the other.
My new theory is that the interface is being brought up too early in stock OpenWrt. Before some other important process has completed that stages the ath10k calibration data where the driver expects to find it.
[ 42.604526] ath10k_pci 0000:00:00.0: unable to read from the device (-145)
[ 42.611652] ath10k_pci 0000:00:00.0: could not read board ext data addr (-145)
[ 42.619153] ath10k_pci 0000:00:00.0: could not push board ext data (-145)
[ 42.626183] ath10k_pci 0000:00:00.0: failed to download calibration data from EEPROM: -145
I am a customer reporting that YOUR product doesn’t work with OpenWrt as YOU advertise… I’ve spent hours and hours digging up supporting evidence of what is wrong… to fix a problem with YOUR product that has apparently been broken for at least two years. Only for you to say “oh, just simply apply this one-line patch” as if that’s trivial.
I’ve now spent the last twelve hours configuring an OpenWrt build environment to build the entire thing from scratch, then building it, applying your patch, and then trying the new firmware – and it made no difference at all.
It does not work for the stock OpenWrt build. I also tried stock OpenWrt 22, as that is what your firmware 4.3.8 is built on, with the same result.
From my perspective, if GL.iNet is going to claim to be a good partner with the open source OpenWrt community, then at a minimum, the stock OpenWrt should at least work. Currently, this is a wifi router that crashes if you try to use one of the wifi interfaces, which is not a good look to say the least.
As this code presumably works fine with all the other devices that use the ath10k driver except yours, I would say that this bug is something that Gl.iNet should address.
I’d have looked at other patches for your 4.3.8 firmware to assist more in finding the bug, but it appears that you have taken the code off of github, which is also quite frustrating.
Thank you for looking into this. I have found one way to consistently reproduce this. I am using the unmodified .config straight from openwrt github when I compile, so the simplest is to just use their precompiled version.
This will produce a file named “openwrt-23.05.2-ath79-nand-glinet_gl-e750-squashfs-sysupgrade.bin”
Connect to the Mudi via Ethernet. Ethernet is preferred because the default in OpenWrt is to have the wifi radios disabled, so after installing the stock image you will need to connect Ethernet.
Navigate to the firmware upgrade page in the GUI and flash this to the device. Keep no settings, this should be a completely clean install.
After reboot, the device will be on the OpenWrt default IP of 192.168.1.1 instead of the GL.iNet default IP of 192.168.8.1. The OLED screen will not function properly, this is normal and expected.
Consistent way to make router crash on boot:
Configure the router to use 5GHz wifi, any mode, doesn’t matter
opkg update; opkg install modemmanager
Reboot
The same problem does NOT seem to happen if you use 2.4GHz wifi, only 5GHz wifi causes the crash
Getting it to crash with no additional package installed is frustratingly inconsistent.
Sometimes after flashing a clean stock image, all I need to do is enable the 5GHz interface and reboot, then the device will crash on boot every time until I reflash it.
Other times, I will follow the exact same steps, and things will work fine.
So unfortunately I am unable to provide consistent reproducible crash result here – sometimes it crashes forever, sometimes it does not.
modemmanager is NOT the root cause of this, so a solution of “don’t use modemmanager” is not a real solution. This problem happens with other combinations of packages, or sometimes randomly as mentioned previously, even with NO additional packages installed. I cite modemmanager as installing that package seems to be a way to reliably reproduce the crash involving the 5GHz wifi interface.
Secondly, I have checked that this is not some malfunctioning hardware on my particular device. I have two Mudis I bought at different times. One is model “GL-E750C4” and one is model “GL-E750C6”. The issue occurs on both of them.
Unfortunately because of the unusual design of this device, it’s difficult for most people to troubleshoot. For instance, it’s standard for most devices that run OpenWrt to have a UART console… But on this device, the UART is consumed by the MCU that controls the power and OLED.
Also, this appears to be specific to this device. And GL.iNet are the maintainers for this device.
I’m very sorry
I tested the self-compiled firmware of openwrt. It does not have 5g wifi. Can you tell me how you turned it on?
Then I also tested the code to pull openwrt 23.05 and the compiled firmware did not reproduce.
If you have more details about the crash
Thu Apr 27 20:35:31 2023 daemon.err ModemManager[2576]: hotplug: parent device sysfspath not found
Thu Apr 27 20:35:31 2023 daemon.debug ModemManager[2576]: hotplug: cached event found: action=add, name=br-lan, subsystem=net, sysfspath=/sys/devices/virtual/net/br-lan
Thu Apr 27 20:35:31 2023 daemon.debug ModemManager[2576]: hotplug: event reported: action=add, name=br-lan, subsystem=net
Thu Apr 27 20:35:32 2023 daemon.err ModemManager[2576]: hotplug: parent device sysfspath not found
Thu Apr 27 20:35:32 2023 daemon.debug ModemManager[2576]: hotplug: cached event found: action=add, name=wlan0, subsystem=net, sysfspath=/sys/devices/pci0000:00/0000:00:00.0/net/wlan0
Thu Apr 27 20:35:32 2023 daemon.debug ModemManager[2576]: hotplug: event reported: action=add, name=wlan0, subsystem=net
Thu Apr 27 20:35:33 2023 kern.warn kernel: [ 101.336323] ath10k_pci 0000:00:00.0: unable to read from the device (-145)
Thu Apr 27 20:35:33 2023 kern.err kernel: [ 101.343507] ath10k_pci 0000:00:00.0: could not read board ext data addr (-145)
Thu Apr 27 20:35:33 2023 kern.err kernel: [ 101.350998] ath10k_pci 0000:00:00.0: could not push board ext data (-145)
Thu Apr 27 20:35:33 2023 kern.warn kernel: [ 101.358012] ath10k_pci 0000:00:00.0: failed to download calibration data from EEPROM: -145
Here is a way to try to get kernel errors as it’s crashing, without access to the serial console:
Enabling “netconsole”, which exports the kernel log via UDP to a remote host (how handy!): Enabling Netconsole | Community
HOST=192.168.1.2
uci set system.@system[0].log_ip=$HOST
uci set system.@system[0].log_port=‘514’
uci set system.@system[0].log_proto=‘udp’
uci commit system
reload_config
I connected my computer via ethernet with IP 192.168.1.2 and listened with wireshark to catch the log messages.
