For me that doesn't look very surprised.
often the stability of a connection will lag if you upload when concurently downloading.
I noticed this myself when configurating sqm and decide to lower my upload to criminal low levels, I was able to higher the lat for downloading and still keep a A+ score.
But it can also be a issue by isp or equipment where upload becomes more than download.
is it wired or wireless?, try wired.
is a vpn involved?, try without it.
And there is MTU.
Often the max what most isp providers allow is 1500 mtu or maybe a bit more to allow vlans (1508).
The higher the mtu the faster the speed, but your network need to be able to support it, and the isp.
now the issue is, maybe something is putting you over this max mtu range, then this is likely why download becomes slower.
This is a pity in all gl firmware btw, the firmware doesn't respect the automatic calculations for mtu in OpenWrt, but lan already uses 1500 mtu, then comes ipv6, and you have 1540 mtu minimum and 1520 for ipv4, and then wireguard boom 60-80 on top of that, you may also use pppoe that also adds extra to the count.
^ i don't know the exact sizes, but this is one reason why to leave these mtu fields alone on the DSA configuration, it is very likely you are exceeding and this is how it looks like when the ISP is blocking this high mtu.
You could check what it shows when editing in advanced options -> network -> interfaces -> click devices tab -> edit br-lan, change mtu to empty and see with what it comes (backup before doing this).
My guess is to look into lowering the vpn mtu first.