HomeKit devices discovery via mDNS on VPN connection

[350D]

Hello! I have GL-SFT1200 (Opal) here. My setup looks like this - I have free/open wifi network (WIFI1) in remote location where I have my HomeKit device located (camera). GL-SFT1200 in repeater mode connected to this open wifi. Custom Wifi network (WIFI2) created on GL-SFT1200 for my HomeKit device. HomeKit device connected to WIFI2. VPN connection from GL-SFT1200 to my home router (Unifi router with WireGuard server) works fine.

I can ping external (connected via vpn) HomeKit device from my home internal network and I can use camera vendor setup app to control the camera (looks like to works somehow differently than HomeKit device discovery).

The problem is that HomeKit can't discover/setup new device due to mDNS issues I guess. I've tried to install Avahi service on GL-SFT1200 and enabled reflector for 4 interfaces (allow-interfaces=br-lan,wgclient,wlan0,wlan-sta0) but mDNS discovery doesn't work. 5353 port enabled in firewall.
Please advice how to properly setup IoT device discovery in my case if I want to add second camera/device and be able to manage both cameras via HomeKit.
Thank you!

[bruce]

Please create a network topology diagram, to make it clearer to understand how to check IoT discovery.

I have tried to
Did you install Avahi on the SFT1200?

Edit the Avahi configuration file to include the interfaces (the home kit) where mDNS packets should be reflected.

I just learned from the Google, cannot try in the local as the environment.

[350D]

@350D

GL-SFT1200

Avahi installed on SFT1200, reflector enabled, interfaces enabled.
I can ping Camera from my local network but can't add it to HomeKit.

[bruce]

mDNS with the HomeKit traffic probably only work in the layer 2 or same one sub-net.

Probably can try the Zerotier layer 2 network, or the OpenVPN-TAP mode.
Try to capture the packet from the VPN Client device to VPN Server interface, to check if the mDNS traffic can transit via your VPN tunnel.

[350D]

I've tried to setup OpenVPN server on Unifi router and Client on SFT1200 but VPN connection works fine only with dev tun option in config and constantly restarting with dev tap option.

Tue Sep 10 23:15:52 2024 daemon.notice ovpnclient[7893]: net_iface_up: set ovpnclient up
Tue Sep 10 23:15:52 2024 daemon.notice ovpnclient[7893]: net_addr_v4_add: 192.168.5.4/24 dev ovpnclient
Tue Sep 10 23:15:52 2024 daemon.notice ovpnclient[7893]: /etc/openvpn/scripts/ovpnclient-up ovpnclient 0 ovpnclient 1500 1555 192.168.5.4 255.255.255.0 init
Tue Sep 10 23:15:52 2024 user.notice ovpnclient-up: env value:route_vpn_gateway=192.168.5.1 daemon_log_redirect=0 script_type=up proto_1=tcp-client daemon=0 SHLVL=1 foreign_option_1=dhcp-option DNS 192.168.5.1 dev_type=tun route_network_1=192.168.1.0 remote_1=******** dev=ovpnclient route_network_2=172.16.1.0 route_network_3=192.168.3.0 X509_0_CN=UniFi_OpenVPN_Server X509_0_C=US remote_port_1=1194 X509_1_CN=UniFi_OpenVPN_CA X509_1_C=US ifconfig_netmask=255.255.255.0 tls_digest_sha256_0=************ daemon_start_time=1725999347 script_context=init ifconfig_local=192.168.5.4 common_name=UniFi_OpenVPN_Server tls_digest_sha256_1=************ X509_0_L=New York verb=3 X509_1_L=New York link_mtu=1555 X509_0_O=Ubiquiti Inc. route_gateway_1=192.168.5.1 trusted_ip=************ tls_serial_hex_0=************ X509_1_O=Ubiquiti Inc. tun_mtu=1500 route_gatewa
Tue Sep 10 23:15:53 2024 daemon.notice netifd: ovpnclient (7893): sh: 1: unknown operand
Tue Sep 10 23:15:54 2024 kern.info kernel: [  807.069653] IPv6: ADDRCONF(NETDEV_UP): ovpnclient: link is not ready
Tue Sep 10 23:15:54 2024 daemon.notice netifd: Interface 'ovpnclient' is now up
Tue Sep 10 23:15:54 2024 daemon.notice netifd: Network device 'ovpnclient' link is up
Tue Sep 10 23:15:55 2024 daemon.notice netifd: ovpnclient (7893): uci: Entry not found
Tue Sep 10 23:15:59 2024 daemon.notice netifd: ovpnclient (9547): Cannot find device "ovpnclient"
Tue Sep 10 23:15:59 2024 daemon.notice netifd: Interface 'ovpnclient' is now down
Tue Sep 10 23:15:59 2024 daemon.notice netifd: Interface 'ovpnclient' is setting up now
Tue Sep 10 23:16:05 2024 daemon.notice netifd: Interface 'ovpnclient' is setting up now
Tue Sep 10 23:16:06 2024 daemon.warn ovpnclient[10810]: WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
Tue Sep 10 23:16:06 2024 daemon.warn ovpnclient[10810]: DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-256-CBC' to --data-ciphers or change --cipher 'AES-256-CBC' to --data-ciphers-fallback 'AES-256-CBC' to silence this warning.
Tue Sep 10 23:16:06 2024 daemon.notice ovpnclient[10810]: OpenVPN 2.5.7 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Tue Sep 10 23:16:06 2024 daemon.notice ovpnclient[10810]: library versions: OpenSSL 1.1.1i  8 Dec 2020, LZO 2.10
Tue Sep 10 23:16:06 2024 daemon.warn ovpnclient[10810]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Tue Sep 10 23:16:06 2024 daemon.notice ovpnclient[10810]: Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Tue Sep 10 23:16:06 2024 daemon.notice ovpnclient[10810]: Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication

Also all wifi clients are disconnected after this TAP setup and can't reconnect anymore and I need to reset the firmware to setup everything back again.

Im on latest firmware v4.3.19

[bruce]

I'm sorry, just confirm from the R&D team, OpenVPN-TAP mode works in the 4.5 later, seems the SFT1200 does not support.

May try other ways to achieve the layer2, like Zerotier Layer2.

[350D]

I don't have Zerotier app on my SFT1200 unfortunately. What model I need to buy to have 4.5 and Zerotier, GL-A1300, GL-MT3000 ?

[bruce]

A1300 and MT3000 both meets which the OpenVPN-TAP and Zerotier.

But the Zerotier can be manually installed in the SFT1200 (Not recommended, as the performance probably not enough to work well in the SFT1200). Check the PM.

[350D]

Hello! Now I have A1300 and I'm trying to setup ZeroTier. I have Pi3 on my local network with Homebridge on it. I've installed ZeroTier server on it with default configuration. I've joined Pi to my ZeroTier network. A1300 joined to the same network. Local network 192.168.1.0/24. I can ping my IoT camera ip no problem. HomeKit mdns device discovery doesn't work at this moment. What I need to update/change to get this setup working?
I've updated AVAHI conf on Pi with ztre4vbtvd interface enabled. Avahi daemon installed on A1300 and reflector enabled.

Zerotier network clients:

[bruce]

Did you configure the layer2 network? bridge the br-lan and Zerotier interface.

[Share] Create Zerotier Virtual Network with Layer2 in the GL Router and Other Clients

Requirement: Remote Zerotier runs on Layer 2, means realizing the bridge of the interface Zerotier and LAN. Background and Topology: The Terminal Device does not support the Zerotier application, and it only works in the Layer2. [image] Key Words: Layer 2, Bridge, No Public IP, Remote Virtual Network. Configuration: Create the Zerotier ID group, bring the routers join the Zerotier network. Enable Allow Ethernet Bridging [image] Interface to bridge, Luci -> Network -> Interface…

[350D]

Hello! Yes, I've configured Bridge on my test Pi following docs and using my ip adresses (192.168.1.xx instead of 192.168.192.xx) but nothing works - I can't even ping my device connected to GL router from my local network.
I have tried to set static ips everywhere like 192.168.1.89 for my local Pi with Zerotier Bridge, 192.168.3.90 for remote GL router, camera connected to this router with 192.168.1.90, both ips added to Managed Routes like this:

192.168.1.0/24 via 192.168.1.89 (this is how its should be following official tutorial for RPi)
192.168.1.0/24 via 192.168.3.90 (this is how its provided in GL interface)

I've tried to setup remote router with 192.168.3.90 ip/subnet - same thing.
Allow remote access to LAN - enabled on router.

Im loosing my hope here....

Also I've tried to setup this Pi as OpenVPN TAP server and GL router accepted my config file and display it as TAP-S2S type. But if I try to connect this VPN - router timeouts and I need to reset network config every time. Will it work only with another GL router with TAP-S2S server on it or it should work with my OpenVPN setup too?
My TAP config file looks like this:

client
dev tap
proto udp
remote ****IP**** **PORT**
resolv-retry infinite
nobind
key-direction 1
remote-cert-tls server
tls-version-min 1.2
verify-x509-name openvpn_********** name
cipher AES-256-CBC
auth SHA256
auth-nocache
verb 3
<ca>.........

[xize11]

Hmm im not sure if it helps but i do know it helped with wireguard (even though wireguard doesn't support L2), can you try adding route 224.0.0.0/8 to the vpn and then restart avahi?

its also worth to allow more ports than just 5353, theoretical 5353 is only for avahis discovery, but then those devices also use ports to communicate.

Avahi communicates like:

zone iot -> device (this device/router self as destination)

But the device communication needs allowance like:

iot -> lan (8007-8009, 10100, 10001, 190, 8443)

^ Once they discovered each other and communicate.

My suspicion is that the multicast route is missing or the traffic rules fail.

here is how i have them defined:

If it still doesn't work, you can forward the firewall zones to it, and set forward to accept for the zone, this way you can check if avahi works or the vpn is at issue.

[350D]

Thank you for advice. Can you help me with this settings? I guess im doing it in a wrong way:

Static Route:

Traffic Routes:

[xize11]

Hi,

As for the static route, change type to multicast, and if you click on the tab advanced make sure the table main or default is selected.

You can verify if the route is correct when you use cli command ip route in ssh, if you don't see it, then something is not good, i often expect this to be happening on the table selection

As for the firewall rules:

For first rule you must specify 224.0.0.0/8 as destination otherwise for some odd reason the firewall won't do anything.

I think you want wgclient as source, destination to lan for the second rule, then the device on lan is automaticly allowed to talk back on the same line.

^ since you use avahi this makes it easier because avahi talks through 0.0.0.0 for each segmented interface i.e vpn gateway ip, but also the lan interface gateway ip this way it gets advertised so a handshake can be made so discovery will work.

You only need to make sure reflector is set to yes in /etc/avahi/avahi-deamon.conf

But after the discovery... then you have ports 8007-8008 etc, it either starts talking from lan to wg or the other way around, if you are not sure you can always set forwarding to accept for firewall zone lan, or wgclient.

You can check alot of these data with tcpdump

tcpdump -i br-lan -v multicast host <ip of host>

aslong one side is allowed to talk, the other side is allowed to talk back on the same line, this means you don't need reverse rules.

It's recommend to restart the router or avahi after the changes, avahi is very vulnerable to changes i have been doing alot of chromecast testing but it can suddenly stop working

[350D]

Well, I've updates the route first and and its visible in ip route:

root@GL-A1300:~# ip route
default via 192.168.6.1 dev wlan-sta0 proto static src 192.168.6.221 metric 1 
192.168.6.0/24 dev wlan-sta0 proto static scope link metric 1 
192.168.7.0/24 dev br-lan proto kernel scope link src 192.168.7.3 
multicast 224.0.0.0/8 dev wgclient proto static scope link 

Both rules was updated too and avahi restarted but I still can't add my device to home kit. I can ping it from my home network (192.168.1.) over vpn connected from my parking IoT network same time (192.168.7.)

tcpdump not available on GL router and I can't find it in plugins section to provide more info. Device connected over wifi and not by lan cable if it make sense to use LAN in my Rule #2.

[xize11]

Can you try setting both firewall zones to forwarding: accept? (As temporary solution just to see if multicast works)

You also need a traffic rule from the other side to the router: i.e:

name: allow-avahi-wgclient
Src: wgclient
dest: this device
Dest port 5353
dest ip: 224.0.0.0/8


name: allow-avahi-lan
Src: lan
dest: this device
Dest port 5353
dest ip: 224.0.0.0/8

name: allow-mdns
Src: wgclient
dest: lan
Dest port 8007-8009 8443 10100 10001 190
dest ip: 224.0.0.0/8 edit: please don't use this destination address, this are for direct connections not avahis, after the discovery advertisement.

And you are sure the wifi is not part of the guest network?

For tcpdump you first need to update the package list

[350D]

My Zones (defaults):

Package update results (but I can ping 1.1.1.1 on ssh session):

Update packages works when VPN disabled! And doesn't when connected, hmmm.

[xize11]

Under wgclient zone please set forward to accept as temporary test.

As for update manager it looks there is no internet or package list cannot be fetched, maybe needs to be reported as bug.

Edit:

It is still possible you may need to enable multicast on the wgclient interface now that this gl-inets version of wireguard i don't know if it works + if opal is using the newer DSA switch architecture.

you can try adding option multicast '1' on the wireguard interface, but no promise if it works

and if it then still not works then maybe you need to look into fragmented packets which is why tcpdump is really usefull because if avahi sents empty packet lengths the mtu on wgclient needs higher.

[350D]

multicast added and also ive added these three rules to my Unifi router at home network where my VPN client have ip 192.168.7.2:

tcpdump while camera app launched and I check for camera video feed in native app:

root@GL-A1300:~# tcpdump -i br-lan -v multicast and host 192.168.7.51
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
12:58:10.234371 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    Indoorcam.lan.25353 > 224.0.0.249.25353: UDP, length 41
12:58:10.237308 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 504)
    Indoorcam.lan.25353 > 224.0.0.249.25353: UDP, length 476
12:58:10.938119 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan (Broadcast) tell console.gl-inet.com, length 28
12:58:11.650371 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    Indoorcam.lan.25355 > 239.0.1.100.25355: UDP, length 41
12:58:11.652924 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 496)
    Indoorcam.lan.25355 > 239.0.1.100.25355: UDP, length 468
12:58:13.261676 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan tell Indoorcam.lan, length 46
12:58:21.177938 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan (Broadcast) tell console.gl-inet.com, length 28
12:58:23.262121 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan tell Indoorcam.lan, length 46
12:58:30.287620 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    Indoorcam.lan.25353 > 224.0.0.249.25353: UDP, length 41
12:58:30.287621 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 504)
    Indoorcam.lan.25353 > 224.0.0.249.25353: UDP, length 476
12:58:31.418009 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan (Broadcast) tell console.gl-inet.com, length 28
12:58:31.673359 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    Indoorcam.lan.25355 > 239.0.1.100.25355: UDP, length 41
12:58:31.673360 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 496)
    Indoorcam.lan.25355 > 239.0.1.100.25355: UDP, length 468

Dump while im trying to add this camera to HomeKit:

root@GL-A1300:~# tcpdump -i br-lan -v multicast and host 192.168.7.51
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
13:00:30.392767 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    Indoorcam.lan.25353 > 224.0.0.249.25353: UDP, length 41
13:00:30.392773 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 504)
    Indoorcam.lan.25353 > 224.0.0.249.25353: UDP, length 476
13:00:31.792443 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    Indoorcam.lan.25355 > 239.0.1.100.25355: UDP, length 41
13:00:31.793217 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 496)
    Indoorcam.lan.25355 > 239.0.1.100.25355: UDP, length 468
13:00:33.287817 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan tell Indoorcam.lan, length 46
13:00:34.298110 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan (Broadcast) tell console.gl-inet.com, length 28
13:00:43.287854 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan tell Indoorcam.lan, length 46
13:00:44.537426 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has Indoorcam.lan (Broadcast) tell console.gl-inet.com, length 28

...same loop over and over till I get ERROR in HomeKit interface about Device not found...

I've tried to add these two ports to both firewall rules on GL - 25353, 25355
and on all rules on my home router - same problem, can't add device to HomeKit.

[350D]

@xize11

dest ip: 224.0.0.0/8 edit: please don't use this destination address, this are for direct connections not avahis, after the discovery advertisement.

Can I use ANY option in this rule?