[beryl-ax] TLS/SSL timeout on some domains

Hello,

I have some strange issues establishing TLS connections to a few servers when I use Tailscale through my beryl-ax router.

One of the affected urls is this here: BundID

I use the following command to get reproducable tests:

$ curl -v https://int.id.bund.de/idp/profile/SAML2/POST/SSO

Some observations:

  • Using the tailscale client function of beryl-ax I can connect normally to the internet, incl. most of the https/tls/ssl connections. The connection is stable and reliable. One of the affected urls is the one posted above.
  • I have multiple tailscale exit nodes running and when I connect directly using the tailscale client (without beryl-ax) it works on all exit nodes. It means Tailscale works and my exit nodes work properly.
  • When I use beryl-ax as a Wireguard client it works also.
  • When I use beryl-ax as tailscale client I can establish the tls connection from it’s shell (beryl-ax). β†’ Beryl-ax can establish the connection using tailscale.
  • The issue may lay in the implementation the routing of tailscale inside beryl-ax.
  • I reproduced the behavior after a complete factory reset of the beryl-ax.

I highly appreciate every idea or solution, thank you in advance!

Steps to reproduce

  • factory-reset the beryl-ax through the web gui
  • enable tailscale in the web gui
  • log in using ssh, then tailscale up --authkey tskey-auth-xxxxx-xxxxx --accept-routes --reset
  • select an exit node and connect to it
  • enable subnet routing in the tailscale panel for all advertised routes
  • the clients on LAN (β€œNotebook”) are successfully connected through the tailscale tunnel and I can’t establish the connection to the url mentioned above.

My setup

     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                 
     β”‚ Notebook β”‚                 
     β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜                 
          β”‚                       
          β–Ό                       
         LAN                      
      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”                  
      β”‚beryl-axβ”‚(tailscale client)
      β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  
         WAN                      
          β”‚                       
          β–Ό                       
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”               
    β”‚ home router β”‚               
    β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜               
          β”‚internet               
          β–Ό                       
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”               
    β”‚ home router β”‚               
    β””β”€β”€β”¬β”€β”€β”€β”€β–²β”€β”€β”€β”€β”€β”˜               
       β”‚    β”‚                     
       β–Ό    β”‚                     
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”           
β”‚     orangepi        β”‚           
β”‚(tailscale exit node)β”‚           
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜           

Debugging and Packet capturing

I captured the traffic simultaneoulsy on the notebook (wireshark) and on the beryl-ax router (tcpdump). The output seems to be exactly the same. The pattern I’ve found is, that after Client Hello and another ACK, there is a TCP Previous segment not captured that has always the same sequence no. 4097.

Domains that cause problems:

I used the following command to trigger the handshake:

$ curl -v https://int.id.bund.de/idp/profile/SAML2/POST/SSO
*   Trying 80.245.156.58:443...
* Connected to int.id.bund.de (80.245.156.58) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL connection timeout
* Closing connection 0
curl: (28) SSL connection timeout

Capture of the unsuccessful TLS handshake:

From the Notebook:

I captured this on my beryl-ax using tcpdump on eth1


19:18:50.429114 IP (tos 0x0, ttl 64, id 5948, offset 0, flags [DF], proto TCP (6), length 60)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [S], cksum 0xc7ee (correct), seq 1127279642, win 64240, options [mss 1460,sackOK,TS val 1599965908 ecr 0,nop,wscale 7], length 0
19:18:50.698823 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [S.], cksum 0x8020 (correct), seq 2100488049, ack 1127279643, win 65160, options [mss 1420,sackOK,TS val 1591705801 ecr 1599965908,nop,wscale 7], length 0
19:18:50.699896 IP (tos 0x0, ttl 64, id 5949, offset 0, flags [DF], proto TCP (6), length 52)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xaa48 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 1599966179 ecr 1591705801], length 0
19:18:50.730504 IP (tos 0x0, ttl 64, id 5950, offset 0, flags [DF], proto TCP (6), length 569)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [P.], cksum 0xae24 (correct), seq 1:518, ack 1, win 502, options [nop,nop,TS val 1599966208 ecr 1591705801], length 517
19:18:50.982090 IP (tos 0x0, ttl 54, id 14039, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xa703 (correct), seq 1, ack 518, win 506, options [nop,nop,TS val 1591706088 ecr 1599966208], length 0
19:18:50.992094 IP (tos 0x0, ttl 54, id 14043, offset 0, flags [DF], proto TCP (6), length 711)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [P.], cksum 0xa261 (correct), seq 4097:4756, ack 518, win 506, options [nop,nop,TS val 1591706093 ecr 1599966208], length 659
19:18:50.992897 IP (tos 0x0, ttl 64, id 5951, offset 0, flags [DF], proto TCP (6), length 64)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0x840a (correct), seq 518, ack 1, win 502, options [nop,nop,TS val 1599966472 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:19:51.198188 IP (tos 0x0, ttl 64, id 5952, offset 0, flags [DF], proto TCP (6), length 64)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0x98d9 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600026681 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:19:51.466297 IP (tos 0x0, ttl 54, id 14051, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xa721 (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591766574 ecr 1599966472], length 0
19:20:52.633787 IP (tos 0x0, ttl 64, id 5953, offset 0, flags [DF], proto TCP (6), length 64)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xa8d8 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600088121 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:20:52.886862 IP (tos 0x0, ttl 54, id 14053, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xb72f (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591827999 ecr 1599966472], length 0
19:21:54.069822 IP (tos 0x0, ttl 64, id 5954, offset 0, flags [DF], proto TCP (6), length 64)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xb8d7 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600149561 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:21:54.330800 IP (tos 0x0, ttl 54, id 14054, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xc726 (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591889447 ecr 1599966472], length 0
19:22:55.506619 IP (tos 0x0, ttl 64, id 5955, offset 0, flags [DF], proto TCP (6), length 64)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xc8d6 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600211001 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:22:55.760588 IP (tos 0x0, ttl 54, id 14056, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xd72a (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591950882 ecr 1599966472], length 0
19:23:51.033023 IP (tos 0x0, ttl 64, id 5956, offset 0, flags [DF], proto TCP (6), length 64)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [F.], cksum 0xefe9 (correct), seq 518, ack 1, win 502, options [nop,nop,TS val 1600266531 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:23:51.085346 IP (tos 0x0, ttl 54, id 14057, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [F.], cksum 0xff1a (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1592006192 ecr 1599966472], length 0
19:23:51.086264 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [R], cksum 0x12ec (correct), seq 1127280160, win 0, length 0
19:23:51.294501 IP (tos 0x0, ttl 54, id 14058, offset 0, flags [DF], proto TCP (6), length 52)
    80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0x6a19 (correct), seq 4757, ack 519, win 506, options [nop,nop,TS val 1592006417 ecr 1600266531], length 0
19:23:51.295290 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    tuxedo.lan.46142 > 80.245.156.58.443: Flags [R], cksum 0x12eb (correct), seq 1127280161, win 0, length 0

And a capture of a successful TLS handshake:

 $ curl -v https://int.id.bund.de/idp/profile/SAML2/POST/SSO
*   Trying 80.245.156.58:443...
* Connected to int.id.bund.de (80.245.156.58) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=DE; ST=Nordrhein-Westfalen; O=Informationstechnikzentrum Bund; CN=int.id.bund.de
*  start date: Jan 10 00:00:00 2024 GMT
*  expire date: Jan  9 23:59:59 2025 GMT
*  subjectAltName: host "int.id.bund.de" matched cert's "int.id.bund.de"
*  issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV RSA CA 4
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /idp/profile/SAML2/POST/SSO HTTP/1.1
> Host: int.id.bund.de
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Thu, 28 Mar 2024 18:15:36 GMT
< Server: Apache
< Set-Cookie: INGRESSCOOKIE=5832166eb30b7d0aa56c661961dd03ca|864f5e10da27136412d8b1c00ef7b324; Path=/idp; Max-Age=3600; Secure; HttpOnly; SameSite=None
< Set-Cookie: JSESSIONID=69917BC01A7E8823814ACA00F1D16819; Path=/idp; Secure; HttpOnly; SameSite=None
< Set-Cookie: AL_SESS-S=ARU_nWWoMUG65LCRbcKihD8RPFBDCfwMOO6k4HstEOsCORNsApTbt6UO5T4rdDz9_cNP; Path=/; Secure; HttpOnly
< Content-Length: 0
< Cache-Control: no-store, no-cache, max-age=0
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src 'self'; img-src 'self' data:; style-src 'unsafe-inline' 'self'; connect-src 'self' http://127.0.0.1:24727; upgrade-insecure-requests; frame-ancestors 'none';  object-src 'none'
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains; preload
< Location: https://int.id.bund.de/idpError?idp.code=UnableToDecode&url
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html;charset=utf-8
< 
* Connection #0 to host int.id.bund.de left intact

Other specs

version
beryl-ax Admin Panel 4.5.0
beryl-ax tailscale 1.32.2-dev
exit node OS armbian, raspbian, ubuntu
exit node tailscale 1.56.1, 1.60.1, 1.58.2 …
notebook ubuntu 22.04
2 Likes

Even if I can’t really help you right now I would like to say that I highly appreciate this form of bug report. Must be one of the prettiest and professional ones I saw in a while. :clap:t2:

4 Likes

I dont have much knowledge about tailscale, but i wonder could it be that one of the ip you connect to on their site blocks your endpoint ip?

Especially the timeout thingy makes me wonder if that is true, it could be some type of hop routing to a difference ip which blocks it after a few packets passed within or after handshaking?

With mullvad/wireguard im aware of bigger cdn being the culprit, akamai, but also amazon aws can randomly block such ip without even informing the company peer, also on steam people found a way to make their acc inaccessible by just using a black listed word :grimacing:, i noticed when something of that happens the routing also differs if you do a tracert vs block and non block.

If its not that i wonder if MTU is at play, tbh i can’t really help but that comes in my mind.

1 Like

Hi xize11,
thank you so much for your reply!!
It is indeed the MTU. The interface was configured with 1500 and I figured out that it works with 1280.

This solved the issue:

sudo ip link set <interface> mtu 1280

I am wodnering why the other domains work normally…

Have a great day :sunflower:

1 Like