Docker and nginx web server on router?

jdub · 2022-11-30T03:16:29.737Z

That’s the truth!

I get the sense that there are a few people around here who like to tear down and completely rebuild their network every couple of months. No judgment! I was one of those people at one point. I moved over to a pfSense+WAP setup about 10 years ago for my main network and haven’t looked back. With both of us WFH with COVID I moved over to a high-availability (two router) setup with dual WAN failover. As long as we have power, we’ll have internet!

IMO a HA setup between two inexpensive routers (e.g. Brume 2’s) would be much more useful for most people than dual-WAN failovers. But I’m not the developer

I’ll try to get docker running when it shows up. FYI, I noticed tonight that there is a squid package available, and you can certainly run nginx, as it is what Gl.iNet are running for their web interface. You’d have to alter the configs and possibly ports, but as a relatively lightweight proxy I would think the config should be doable. We run nginx in production in front of our webservers, but use haproxy to load balance. I’m not sure if that’s available, but it would be another possibility to segment things out from nginx.

Best of luck! Let me know what direction you’re leaning. I’ll try to get a slim Docker install when the Brume 2 comes in.

Vinhdizzo · 2022-11-30T03:25:17.092Z

@alzhao sorry, I couldn’t follow the translation of the post referenced exactly. I see luci-app-dockerman as a package that could be installed. I don’t see dockerd. How does one go about installing that? Assuming a repository needs to be added, what is the URL?

Thank you.

alzhao · 2022-11-30T07:43:35.636Z

Asking developers to add dockerd to the repo. Pls hold on.

Vinhdizzo · 2022-12-02T21:19:18.269Z

Any updates @alzhao ? Thanks.

jdub · 2022-12-02T23:33:27.676Z

If you’re not docker insistent I did get LXC working on a Brume 2 today

Vinhdizzo · 2022-12-03T02:50:49.865Z

@jdub I’ve never used LXC before. I’ve only gotten started with docker in the past 2 years, and finally got a handle on it via docker compose, and images provided by linuxserver. Hoping to just run nginx as a reverse proxy for my apps…

jdub · 2022-12-03T03:33:46.596Z

Gl-iNet’s web interface actually runs on nginx natively … you might be better off just modifying their config file to reverse proxy what you need to if that’s what you’re looking for. Though you’ll probably lose it on every firmware upgrade (use the more stock OpenWRT build!!! )

My guess is that LXC would end up being too heavy on the flint/slate ax anyway. Docker may be, if it’s even possible.

alzhao · 2022-12-03T04:43:40.028Z

Dockerd is now available in the repo.

But AX1800 does not have enough storage to install at all.
You need to use external storage.

Vinhdizzo · 2022-12-03T04:59:44.431Z

@alzhao thank you, I see both docker and dockerd now.

I can plug in a USB drive, that’s not a problem. Do I have to specify any configuration for openwrt to know to install these packages on the USB drive? Any thoughts on performance / stability of running docker apps on Flint?

Thanks again.

alzhao · 2022-12-03T05:26:32.264Z

You can try exroot to move openwrt to USB drive

Sorry I cannot answer the performance quetions of docker on the router.

Vinhdizzo · 2022-12-03T22:39:00.347Z

Thanks @alzhao, I was able to install extroot, mount a USB drive, and successfully install dockerd and luci-app-dockerman. However, I couldn’t get docker-compose installed. Error message:

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency python3-cryptography for python3-paramiko
 * pkg_hash_fetch_best_installation_candidate: Packages for python3-paramiko found, but incompatible with the architectures configured
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for docker-compose:
 * 	python3-cryptography
 * opkg_install_cmd: Cannot install package docker-compose.

Any chance this could be made available as that is how I configure docker apps. Thank you.

Vinhdizzo · 2022-12-03T22:49:33.227Z

I must’ve missed this post of yours. I know this may be off topic, but what how exactly should one use a Brume 2 device with my current setup? I do run wireguard on my the Flint so I can vpn home when I need to. Care to elaborate, @jdub? Thanks.

jdub · 2022-12-03T23:08:32.132Z

If you’ve already got a flint, a Brume is probably redundant, though I could give some reasons. It has more memory and storage, which would more easily let you do things like you’re considering (docker), though you can do them with extroot too. They’re cheap enough that you could get two and run a high availability setup, although that’s some work with OpenWRT.

Personally I prefer to have my router and access points separate so I can upgrade either of them independently. I’ve been running the same pfSense install (upgraded hardware and versions, obviously) for the last decade, and it’s been rock solid. I’ve gone from a single OpenWRT access point to a 5 AP setup in that time, all while keeping the same basic routing configuration. You could do something similar with OpenWRT, though it’s a little more complicated with the variety of packages involved.

At the end of the day I standardized on a different platform that works well for me, but OpenWRT is the king of low power devices (I even run it on an embedded armv5 board in production at work!).

If you were actually going to go that route you’d probably be better off getting a cheap x86 board or something like a nanoPi 6S and virtualizing your setup (OpenWRT or one of the BSD alternatives). That way you can easily move setups between hardware setups. Can describe more if interested.

Vinhdizzo · 2022-12-03T23:34:19.495Z

Thanks for the explanation. I do prefer simple setups over complicated setups, and lower energy footprint over higher energy :). Hoping flint can handle an nginx reverse proxy app via docker without issue. If not, I might explore a raspberry pi with openwrt and ax6 access point plugged in

jdub · 2022-12-03T23:36:47.327Z

I’d probably consider something like a Brume 2 with stock ish OpenWRT over a rpi actually. Cheaper and more available, and two wired ports built in. But you can run what you’ve got for a while, I’d think.

xize11 · 2022-12-04T00:49:41.902Z

Problem with docker packages itself is they are really big.

As example I wanted to do this on a mochabin totally different brand… the amount on the partition was 104 mb as extroot, not really much.

but then even if you gonna use for example a usb stick to add storage to a gl-inet router its still difficult you need to know alot of commands.

Tl;tr: for those who want the difficult technical approach:

click to expand

Basicly what you want for package storage:

Mount one partition into /overlay, and /overlay is for docker itself readonly because its only ment for installing huge packages im not sure why its made like that, but pulling images on /overlay will fail.

Then you want to mount /opt/ with a second partition and this is for the use of docker, otherwise you will get vague errors and docker will not pull images, this hunted me in the past

Tools you need:
fdisk e2fsprogs kmod-fs-ext4 kmod-usb-storage blkid block-mount

First you login via ssh and check if the disk exists with ls /dev/ often its callled sd, or sda or even usb, use blkid to make sure, you can also try lsusb.

To format you use:
mkfs.ext4 /dev/<usb device> this will turn it into a ext4 device, you can also use other file systems but I believe ext4 is stablest.

to partition:
use fdisk /dev/sda its important you dont select the child partition but basicly the master, then press p to see any partitions if you see any type d this is to delete it, and if you want to add one you press n then it asks you the partition number and then it gives you two options to create a primary disk or extend its always the first one so 1.

Then it asks you the begin size, just press enter, and then it asks for the end size in where you add: +1G which means you create from 0 to 1gb a partition, this is enough for your packages, you may also can use +512m it depends what suits you.

Congrats you made partition 1, the same wizard is for partition 2 you only have to specify number 2 then when it asks for the drive, and also you don’t have to add anything like +1G as you gonna use the full end.

Edit totally forgot: when done, you write ‘w’ and press enter to write the partition layout and exit fdisk.

Now we need to mount them to docker and package manager…

first we need to make a hold on the uuid table which we can read from blkid, your disk did not list?, then its likely you have to reformat the child partitions to ext4 with mkfs.ext4 /dev/sd.

Now import our disk table into block-mount by using:
block detect | uci import fstab

Then open fstab here with vi or nano: /etc/config/fstab

if correct you will see both of your partitions along with their uuid.

great!, now the only thing you have to do is:

add a section like this:

     config mount 'overlay'
         target '/overlay'
         uuid '<paste uuid from the sda1 partition>'

     config mount 'opt'
        target '/opt'
        uuid 'uuid from partition 2'

Then reboot, this should give you 1gb for packages from your usb device, and all the rest in /opt for docker, as for persistence im afraid you have to redo this everytime with a upgrade or maybe you can re-mount /overlay im not sure about this…

Sources:
[OpenWrt Wiki] Using storage devices

[OpenWrt Wiki] Fstab Configuration

[OpenWrt Wiki] Mounting Block Devices

Vinhdizzo · 2022-12-04T01:16:06.195Z

The link I referenced above for extroot shows a step by step guide with all commands needed to get the USB drive up and running. Not as complicated as one would think.

Thanks.

Vinhdizzo · 2022-12-06T18:00:13.250Z

@alzhao Hi, wanted to follow up on getting docker-compose installed on the router. Would this be possible? Thank you.

alzhao · 2022-12-14T08:57:01.902Z

Sorry, too many depends for docker-compose and I gave up.

alzhao · 2022-12-14T09:06:32.846Z

Docker-compose and the depends has been added to the repo. It installs.

However didn’t test how it works.