Did you have to install the dependencies for dnsmasq-full before you were able to install it in Luci? I was having the same error as you when I ssh and install it that way. I went to system > software in Luci and I see that dnsmasq-full doesn't have the install button, but upgrade.
Edit: I am trying to use luci-app-vpnbypass and it requires dnsmasq-full. I know it's obsolete but flint 2 doesn't have luci-app-pbr and trying to see if I can set a rule for my network to bypass certain domains and certain devices while the rest use vpn. gl.inet tech support responded to me and said they don't support using both VPN Policy Based on the Target Domain or IP and VPN Policy Based on Client Device at the same time. Although he did say they are trying to do that for 4.8 firmware update.