Spitz x3000 - potential bug in modem.sh check_ip()

makkie · March 10, 2024, 12:55pm

Hey guys,

I am still struggling to keep the x3000 router connected to the 4g/5g network. I have been sniffing around in the firmware a bit and this is what I found up to now. Maybe there is a solution!

The router seems to loose internet connection exactly at 6 hour intervals. I have the impression that the 4g/5g provider is changing the module-ipadress every 6 hours which then results in a mismatch with the router interface-ipadress and hence a break in the internet connection:
- module-ip via gl_modem -B 0001:01:00.0 AT AT+CGPADDR
- interface-ip via ifconfig rmnet_mhi0
See also https://forum.gl-inet.com/t/gl-x3000-cellular-connection-issue/30221/53
I have seen that there is a /lib/functions/modem.sh file that seems to have a function called “check_ip” that is exactly checking that these two ip-adressess are the same. If they are not the same the interface is restarted:
- modem-ip via
  module_ip = $(gl_modem -B $bus AT AT+CGPADDR | grep "+CGPADDR: $apn_route" | grep -v '0.0.0.0')
- router-ip via
  interface_ip = $(ubus call network.interface.${modem_iface}_4 status | jsonfilter -e '@["ipv4-address"][0].address')
- check if they are both “not empty”
  if [ -n "$module_ip" -a -n "$interface_ip" ]; then
The problem is that the AT+CGPADDR command returns an ipv4 and an ipv6 adress. If this ipv6 adress somewhere contains 0.0.0.0 the $module_ip variable will be empty and hence the test will not continue and also the mismatch between the two ip-adresses will not be noticed and repaired. E.g. the output of gl_modem -B 0001:01:00.0 AT AT+CGPADDR in my case clearly contains 0.0.0.0 in the ipv6 adress:

image700×402 10.7 KB
Maybe a solution could be to remove the grep -v '0.0.0.0' when trying to find the ‘module_ip’:
- module_ip = $(gl_modem -B $bus AT AT+CGPADDR | grep "+CGPADDR: $apn_route")
I do not know how often this check_ip function in modem.sh is called and if this is the source of prolongued (and regular) internet disconnections …

I really hope some gl inet staff can look at this.

Marc

SpitzAX3000 · March 10, 2024, 3:37pm

btw, how did you find that the connection goes down?

SpitzAX3000 · March 10, 2024, 3:51pm

In my case, there is no IPv6; you can see my result below:

gl_modem -B 0001:01:00.0 AT AT+CGPADDR 

AT+CGPADDR
+CGPADDR: 1,"10.82.137.251","42.2.203.128.64.139.80.210.23.186.249.234.18.4.186.27"
+CGPADDR: 2,"0.0.0.0","0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0"
+CGPADDR: 3,"0.0.0.0","0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0"
OK

The reason I am not getting that empty IPv6 string is that I have disabled IPv6 on ALL interfaces: LuCi → network → Interfaces → Devices

I have managed to get my x3000 connected to the Internet without interrupt for over 2 weeks.

Once the modem is successfully connected, please share the result of the following command:
ps | grep -v '\['

makkie · March 10, 2024, 10:24pm

@SpitzAX3000 I think your modem has an ipv6 adress (see yellow highlight below). You are lucky that there is no 0.0.0.0 in that adress. So the check_ip script runs normally for your case. This might be the reason for your long up-time.

makkie · March 10, 2024, 10:27pm

root@GL-X3000:~# ps | grep -v '\['
  PID USER       VSZ STAT COMMAND
    1 root      1656 S    /sbin/procd
  483 root       856 S    sleep 10
  516 root       860 S    sleep 5
  633 root       856 S    sleep 5
  767 root       856 S    sleep 5
  777 root       856 S    sleep 1
  778 root      1232 R    ps
 1121 ubus      1256 S    /sbin/ubusd
 1246 root       944 S    /sbin/urngd
 2201 root      1036 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 2335 root      1240 S    -ash
 2570 root      1012 S    /usr/bin/fcgiwrap -c 4 -s unix:/var/run/fcgiwrap.socket
 2612 root       840 S    fwdd -d1 -e eth1 5G
 2615 root      1036 S    /usr/bin/fcgiwrap -c 4 -s unix:/var/run/fcgiwrap.socket
 2616 root      1036 S    /usr/bin/fcgiwrap -c 4 -s unix:/var/run/fcgiwrap.socket
 2617 root      1036 S    /usr/bin/fcgiwrap -c 4 -s unix:/var/run/fcgiwrap.socket
 2618 root      1036 S    /usr/bin/fcgiwrap -c 4 -s unix:/var/run/fcgiwrap.socket
 2653 root      3892 S    {gl-ngx-session} /usr/bin/lua /usr/sbin/gl-ngx-session
 3106 logd      1196 S    /sbin/logd -S 64
 3210 root      2224 S    /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 3524 root      2112 S    {mtk-wifi-config} /usr/bin/lua /usr/sbin/mtk-wifi-configurator
 3568 root      3400 S    {repeater} /usr/bin/lua /usr/sbin/repeater
 3829 root      1012 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 3956 root      6548 S    /usr/bin/modem_AT -B 0001:01:00.0
 4095 rpc       1356 S    /usr/sbin/rpcbind -f -w
 4165 root      1860 S    /sbin/netifd
 4306 root      1428 S    /usr/sbin/odhcpd
 4425 root      1236 S    /usr/sbin/crond -f -c /etc/crontabs -l 10
 4727 root      1236 S    /usr/sbin/crond -f -c /tmp/gl_crontabs -l 10
 4935 root      1664 S    /usr/bin/dbus-daemon --system
 5182 root     16148 S    /usr/sbin/ModemManager
 5335 root      8236 S    nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf -g daemon off;
 5470 dnsmasq   2940 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 5494 root     13820 S    nginx: worker process
 5495 root     12352 S    nginx: worker process
 7186 root      6060 S    /usr/bin/sms_manager
 7724 root      6556 S    /usr/bin/gl_b2r_daemon
 8436 root      2936 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 8509 root      1732 S    /bin/sh /usr/bin/modem_signal
 8527 root      1748 S    /bin/sh /usr/bin/gl_led
 8755 root      2604 S    lua /usr/bin/internet-detector service internet
 8880 root      1236 S<   /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.op
 8968 root       944 S    /usr/bin/carrier-monitor
 9094 root     10840 S    /usr/bin/eco /usr/bin/gl-cloud
 9135 root      9384 S    /usr/bin/lua /usr/bin/gl_clients_update
 9212 root       820 S    /usr/bin/gl_fan -T /sys/devices/virtual/thermal/thermal_zone0/temp -D 1000 -t 75
 9440 root      7720 S    eco /usr/bin/modem_manager.lua
 9724 root      1184 S    /usr/sbin/rpc.statd -p 32778 -o 32779 -F
 9725 root      1692 S    /usr/sbin/rpc.mountd -p 32780 -F
 9974 root      1580 S    /bin/sh /usr/sbin/mwan3rtmon
10766 root      1036 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
10906 root      1240 S    -ash
11950 root      5144 S    /usr/sbin/usbmuxd --systemd
14531 root      6788 S    /usr/sbin/rtty -I xxxxxxx -h rtty-ssh-cloud-eu.goodcloud.xyz -p 59127 -v -t xxxxxxxxxxxxxxxxxxxxxxxxx
19880 root      1108 S    qcm -4 -6 -n 1 -s telenetwap.be
20322 root      1452 S    /bin/sh /usr/sbin/mwan3track modem_0001 rmnet_mhi0 online 0.0.0.0 1.1.1.1 8.8.8.8 208.67.222.222 208.67.22
20566 root      1232 S    udhcpc -p /var/run/udhcpc-rmnet_mhi0.pid -s /lib/netifd/dhcp.script -f -t 0 -i rmnet_mhi0 -x hostname:GL-X
23949 root      714m S    /usr/sbin/tailscaled --port 41641 --state /etc/tailscale/tailscaled.state
31212 root      1448 S    /bin/ash -c sleep 30;. /lib/functions/modem.sh;check_ip
31577 root      1232 S    /bin/login

makkie · March 10, 2024, 10:35pm

I installed internet-detector. Each time internet is detected to be down, the interface is restarted and a small script is run that triggers a heartbeat (via a dedicated wget URL) here at home. See below. So in a sense I am able to get around the disconnection issue, but I want the solution to come integrated in the FW of the X3000. Not something that I had to fabricate myself!

SpitzAX3000 · March 11, 2024, 4:38pm

Is it? It does not look like IPv6 format! I will double check it - thanks!

kill the following processes using: kill -9 $pid

/bin/sh /usr/bin/modem_signal …
eco /usr/bin/modem_manager.lua …
/bin/sh /usr/sbin/mwan3track modem_0001 …
/usr/sbin/crond -f -c /etc/crontabs …

Next, SUSPEND the following processes using: kill -STOP $pid

udhcpc -p /var/run/udhcpc-rmnet_mhi0.pid -s /lib/netifd/dhcp.scrip …
/usr/sbin/crond -f -c /tmp/gl_crontabs -l 10
/bin/ash -c sleep 30;. /lib/functions/modem.sh;check_ip
sleep 5

To verify that you suspended the related processes:

# ps | grep -v '\[' | grep ' T'
10873 root      1240 S    grep  T
12112 root      1232 T    udhcpc -p /var/run/udhcpc-rmnet_mhi0.pid -s /lib/netifd/dhcp.script -f -t 0
16355 root      1236 T    /usr/sbin/crond -f -c /tmp/gl_crontabs -l 5
18214 root      1232 T    /bin/ash -c sleep 30;. /lib/functions/modem.sh;check_ip
21402 root       856 T    sleep 5

After that see if the connection goes down again. It’s a process of trial and error. In my modem, the above steps gave me a stable connection.

SpitzAX3000 · March 11, 2024, 4:51pm

Also you can disable the services that you do not use. For example, on my modem I have disabled the following services:

service | grep disabled
/etc/init.d/adguardhome       	  disabled	   stopped
/etc/init.d/avahi-daemon      	  disabled	   stopped
/etc/init.d/cron              	  disabled	   stopped
/etc/init.d/ddns              	  disabled	   stopped
/etc/init.d/dnscrypt-proxy    	  disabled	   stopped
/etc/init.d/edgerouter        	  disabled	   stopped
/etc/init.d/gl-cloud          	  disabled	   stopped
/etc/init.d/gl-tertf          	  disabled	   stopped
/etc/init.d/gl_b2r            	  disabled	   stopped
/etc/init.d/gl_clients        	  disabled	   stopped
/etc/init.d/gl_eqos           	  disabled	   stopped
/etc/init.d/gl_ipv6           	  disabled	   stopped
/etc/init.d/gl_led            	  disabled	   stopped
/etc/init.d/gl_nas_diskmanager	  disabled	   stopped
/etc/init.d/gl_nas_sys        	  disabled	   stopped
/etc/init.d/gl_nas_sys_dl     	  disabled	   stopped
/etc/init.d/gl_nas_sys_up     	  disabled	   stopped
/etc/init.d/gl_s2s            	  disabled	   stopped
/etc/init.d/gl_tethering      	  disabled	   stopped
/etc/init.d/led               	  disabled	   stopped
/etc/init.d/minidlna          	  disabled	   stopped
/etc/init.d/nfsd              	  disabled	   stopped
/etc/init.d/parental_control  	  disabled	   stopped
/etc/init.d/relayd            	  disabled	   stopped
/etc/init.d/repeater          	  disabled	   stopped
/etc/init.d/rpcbind           	  disabled	   stopped
/etc/init.d/rtty              	  disabled	   stopped
/etc/init.d/samba4            	  disabled	   stopped
/etc/init.d/sms_manager       	  disabled	   stopped
/etc/init.d/smstools3         	  disabled	   stopped
/etc/init.d/sysntpd           	  disabled	   stopped
/etc/init.d/tailscale         	  disabled	   stopped
/etc/init.d/umount            	  disabled	   stopped
/etc/init.d/usbmuxd           	  disabled	   stopped
/etc/init.d/vsftpd            	  disabled	   stopped
/etc/init.d/webdav_ser        	  disabled	   stopped
/etc/init.d/zerotier          	  disabled	   stopped

Doing so, not only make the connection more stable, but also increase the CPU/RAM performance.

lorenzoma · March 11, 2024, 6:25pm

That’s interesting, how did you create the entity in home assistant? thanks

lizh · March 12, 2024, 3:51am

Hi makkie:

Thank you very much for your feeedback.
It may be that the monitoring of IP updates fails.

makkie · March 12, 2024, 6:46pm

Hey. That is a good suggestion. But I will only try something so “drastic” as this when i am close to the router. Now I am a few 100s of km away so I can not risk of getting myself permanently locked out. That it why I would love to see a function implemented in the firwmware that allows me to restart/reset the router upon receiving a certain sms command (like the Teltonika routers). Like this I can always regain control …

makkie · March 12, 2024, 6:53pm

Hey. To get a webhook you have to create an “automation” and for the “trigger” choose “other … webhook”:

ywp · March 13, 2024, 3:19am

hello, could you put a double quotation mark here and test it for me

makkie · March 13, 2024, 8:48am

I have implemented the suggested change and disabled the “internet-detector” plug-in. I restarted the router. Now I have to wait at least 6h to see if it is solved … “fingers-crossed” …

Thanks!

makkie · March 13, 2024, 3:19pm

I think we have a succes! Exactly 6h after the reboot the ip adress change was detected and the appropriate action was taken! Nice … thanks … this should go in the official firmware .

SpitzAX3000 · March 13, 2024, 4:36pm

Great. However in my case the interface never go down. The ip is still maintained for over two weeks because I disabled the udhcp process.