Docker and nginx web server on router?

jdub · 2022-12-02T23:33:27.676Z

If you’re not docker insistent I did get LXC working on a Brume 2 today

Vinhdizzo · 2022-12-03T02:50:49.865Z

@jdub I’ve never used LXC before. I’ve only gotten started with docker in the past 2 years, and finally got a handle on it via docker compose, and images provided by linuxserver. Hoping to just run nginx as a reverse proxy for my apps…

jdub · 2022-12-03T03:33:46.596Z

Gl-iNet’s web interface actually runs on nginx natively … you might be better off just modifying their config file to reverse proxy what you need to if that’s what you’re looking for. Though you’ll probably lose it on every firmware upgrade (use the more stock OpenWRT build!!! )

My guess is that LXC would end up being too heavy on the flint/slate ax anyway. Docker may be, if it’s even possible.

alzhao · 2022-12-03T04:43:40.028Z

Dockerd is now available in the repo.

But AX1800 does not have enough storage to install at all.
You need to use external storage.

Vinhdizzo · 2022-12-03T04:59:44.431Z

@alzhao thank you, I see both docker and dockerd now.

I can plug in a USB drive, that’s not a problem. Do I have to specify any configuration for openwrt to know to install these packages on the USB drive? Any thoughts on performance / stability of running docker apps on Flint?

Thanks again.

alzhao · 2022-12-03T05:26:32.264Z

You can try exroot to move openwrt to USB drive

Sorry I cannot answer the performance quetions of docker on the router.

Vinhdizzo · 2022-12-03T22:39:00.347Z

Thanks @alzhao, I was able to install extroot, mount a USB drive, and successfully install dockerd and luci-app-dockerman. However, I couldn’t get docker-compose installed. Error message:

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency python3-cryptography for python3-paramiko
 * pkg_hash_fetch_best_installation_candidate: Packages for python3-paramiko found, but incompatible with the architectures configured
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for docker-compose:
 * 	python3-cryptography
 * opkg_install_cmd: Cannot install package docker-compose.

Any chance this could be made available as that is how I configure docker apps. Thank you.

Vinhdizzo · 2022-12-03T22:49:33.227Z

I must’ve missed this post of yours. I know this may be off topic, but what how exactly should one use a Brume 2 device with my current setup? I do run wireguard on my the Flint so I can vpn home when I need to. Care to elaborate, @jdub? Thanks.

jdub · 2022-12-03T23:08:32.132Z

If you’ve already got a flint, a Brume is probably redundant, though I could give some reasons. It has more memory and storage, which would more easily let you do things like you’re considering (docker), though you can do them with extroot too. They’re cheap enough that you could get two and run a high availability setup, although that’s some work with OpenWRT.

Personally I prefer to have my router and access points separate so I can upgrade either of them independently. I’ve been running the same pfSense install (upgraded hardware and versions, obviously) for the last decade, and it’s been rock solid. I’ve gone from a single OpenWRT access point to a 5 AP setup in that time, all while keeping the same basic routing configuration. You could do something similar with OpenWRT, though it’s a little more complicated with the variety of packages involved.

At the end of the day I standardized on a different platform that works well for me, but OpenWRT is the king of low power devices (I even run it on an embedded armv5 board in production at work!).

If you were actually going to go that route you’d probably be better off getting a cheap x86 board or something like a nanoPi 6S and virtualizing your setup (OpenWRT or one of the BSD alternatives). That way you can easily move setups between hardware setups. Can describe more if interested.

Vinhdizzo · 2022-12-03T23:34:19.495Z

Thanks for the explanation. I do prefer simple setups over complicated setups, and lower energy footprint over higher energy :). Hoping flint can handle an nginx reverse proxy app via docker without issue. If not, I might explore a raspberry pi with openwrt and ax6 access point plugged in

jdub · 2022-12-03T23:36:47.327Z

I’d probably consider something like a Brume 2 with stock ish OpenWRT over a rpi actually. Cheaper and more available, and two wired ports built in. But you can run what you’ve got for a while, I’d think.

xize11 · 2022-12-04T00:49:41.902Z

Problem with docker packages itself is they are really big.

As example I wanted to do this on a mochabin totally different brand… the amount on the partition was 104 mb as extroot, not really much.

but then even if you gonna use for example a usb stick to add storage to a gl-inet router its still difficult you need to know alot of commands.

Tl;tr: for those who want the difficult technical approach:

click to expand

Basicly what you want for package storage:

Mount one partition into /overlay, and /overlay is for docker itself readonly because its only ment for installing huge packages im not sure why its made like that, but pulling images on /overlay will fail.

Then you want to mount /opt/ with a second partition and this is for the use of docker, otherwise you will get vague errors and docker will not pull images, this hunted me in the past

Tools you need:
fdisk e2fsprogs kmod-fs-ext4 kmod-usb-storage blkid block-mount

First you login via ssh and check if the disk exists with ls /dev/ often its callled sd, or sda or even usb, use blkid to make sure, you can also try lsusb.

To format you use:
mkfs.ext4 /dev/<usb device> this will turn it into a ext4 device, you can also use other file systems but I believe ext4 is stablest.

to partition:
use fdisk /dev/sda its important you dont select the child partition but basicly the master, then press p to see any partitions if you see any type d this is to delete it, and if you want to add one you press n then it asks you the partition number and then it gives you two options to create a primary disk or extend its always the first one so 1.

Then it asks you the begin size, just press enter, and then it asks for the end size in where you add: +1G which means you create from 0 to 1gb a partition, this is enough for your packages, you may also can use +512m it depends what suits you.

Congrats you made partition 1, the same wizard is for partition 2 you only have to specify number 2 then when it asks for the drive, and also you don’t have to add anything like +1G as you gonna use the full end.

Edit totally forgot: when done, you write ‘w’ and press enter to write the partition layout and exit fdisk.

Now we need to mount them to docker and package manager…

first we need to make a hold on the uuid table which we can read from blkid, your disk did not list?, then its likely you have to reformat the child partitions to ext4 with mkfs.ext4 /dev/sd.

Now import our disk table into block-mount by using:
block detect | uci import fstab

Then open fstab here with vi or nano: /etc/config/fstab

if correct you will see both of your partitions along with their uuid.

great!, now the only thing you have to do is:

add a section like this:

     config mount 'overlay'
         target '/overlay'
         uuid '<paste uuid from the sda1 partition>'

     config mount 'opt'
        target '/opt'
        uuid 'uuid from partition 2'

Then reboot, this should give you 1gb for packages from your usb device, and all the rest in /opt for docker, as for persistence im afraid you have to redo this everytime with a upgrade or maybe you can re-mount /overlay im not sure about this…

Sources:
[OpenWrt Wiki] Using storage devices

[OpenWrt Wiki] Fstab Configuration

[OpenWrt Wiki] Mounting Block Devices

Vinhdizzo · 2022-12-04T01:16:06.195Z

The link I referenced above for extroot shows a step by step guide with all commands needed to get the USB drive up and running. Not as complicated as one would think.

Thanks.

Vinhdizzo · 2022-12-06T18:00:13.250Z

@alzhao Hi, wanted to follow up on getting docker-compose installed on the router. Would this be possible? Thank you.

alzhao · 2022-12-14T08:57:01.902Z

Sorry, too many depends for docker-compose and I gave up.

alzhao · 2022-12-14T09:06:32.846Z

Docker-compose and the depends has been added to the repo. It installs.

However didn’t test how it works.

autodeck · 2023-05-01T08:58:09.755Z

Hi, did you ever get the docker daemon running? I get:

root@GL-AX1800:~# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I have done the following:

Updated to firmware 4.2.1
extroot a USB stick using the official guide
Installed the dockerd, docker & luci-app-dockerman packages
Ran service dockerd start

When running dockerd, I get:
failed to start daemon: Devices cgroup isn't mounted

Any help / thoughts appreciated.

Thanks

autodeck · 2023-05-05T10:46:57.128Z

@Vinhdizzo any ideas?
I only noticed the following error when installing (opkg install docker dockerd luci-app-dockerman):

Configuring btrfs-progsuci: Entry not found

The following error when running docker (dockerd):

failed to start daemon: Devices cgroup isn't mounted

See attached install.txt & dockerd.log for the full output

docker_install.zip (2.7 KB)

autodeck · 2023-05-06T10:23:23.147Z

@alzhao I see you added dockerd to the repo for the AX1800. Any ideas why it is not working? Please see logs, above

beniamin · 2023-05-07T16:43:53.049Z

Cgroups aren’t really built into openwrt.

OpenWrt Forum – 11 Jan 23

Dockerd: failed to start daemon: Devices cgroup isn't mounted

Installing and Using OpenWrt

I installed lxc today and docker stop working, the dockerman only have one 'configuration' submenu, EDIT: I try to try dockerd in cmdline, and got "failed to start daemon: Devices cgroup isn't mounted", seems like when I installed lxc cgroup got...

But you can try this:

cgroupfs-umount
cgroupfs-mount