joseph.soper

Write-up · BezaForge

How BezaForge is built, and why.

BezaForge is the private cloud I run at home for everything I build: Git hosting, project tracking, dashboards, local AI models and the business's accounting. I wanted infrastructure that keeps working when I don't remember how I set it up, so every machine, service and setting in it is code. This is how it's shaped, how a change reaches it, how it's backed up and watched, and what it still isn't.

November 2026

The shape

Two machines do the work. A Proxmox hypervisor runs the virtual machines and owns the storage. A mini PC runs every containerised service on bare metal, behind one Traefik reverse proxy.

The shape of BezaForge The internet reaches a gateway that splits the network into segments. Behind it, a bare-metal service host runs Traefik, DNS, monitoring and the everyday services, and a Proxmox hypervisor runs an inference VM with the GPU on its own segment, an ERP VM, and a ZFS mirror. The service host backs up to the mirror nightly, and the mirror copies to Google Cloud Storage daily, encrypted. Traefik proves certificate ownership through Cloudflare DNS. Internet Gateway Network segments, firewall rules, VPN in BARE METAL Service host Traefik, TLS for all DNS Monitoring, alerts Git, project tracking, dashboards, chat UI PROXMOX Hypervisor Inference VM GPU, its own segment ERP VM ZFS mirror nightly backups daily, encrypted Cloudflare public DNS, cert proof Google Cloud off-site backups
Violet is mine, cyan is an outside service, pink is where data is kept. Addresses and host names are left out on purpose.

The split is on purpose. DNS, the reverse proxy and the monitoring sit on the bare-metal host instead of in a VM, so I can reboot the hypervisor without taking every name and address down with it. The GPU is passed through to its own VM on its own network segment, where Ollama serves local models. No inference leaves the building.

The gateway splits the network into segments with firewall rules between them. Nothing is published to the internet, and every service still has a real certificate: Traefik asks Let's Encrypt for one per service and proves it owns the name with a DNS record at Cloudflare, so a service never has to be reachable from outside to get one. The only public name is the VPN endpoint I use when I'm away.

Everything is code

Terraform declares the virtual machines through one module, so a VM is a single block that lists only what differs from the defaults. The module can adopt a VM that was built by hand without recreating it, which is how my early VMs came under management without being rebuilt. Ansible configures every host and service from one playbook, and a second, read-only playbook checks the result. Secrets live in encrypted vault files. The repository is public, so it carries their names and nothing else.

How a change reaches the hosts

Every change goes through a branch and a pull request, including the ones Renovate opens when something upstream releases. CI lints the Ansible and YAML, formats and validates the Terraform, and checks the alert rules. Patch and minor updates merge themselves once CI passes. A major version waits for me, and a database major moves one service at a time, by dump and restore into a new data directory, so going back is a revert.

How a change reaches the hosts A change, mine or one Renovate opens, goes through a branch and a pull request, then CI checks, then merges to main; patch and minor updates merge themselves and majors wait for me. Nothing deploys itself. A drift alert fires if main is ahead of the hosts. I run the fleet update, which upgrades OS packages, applies the playbook, runs DNS and health checks and names what changed, and that reaches every host. Branch and pull request every change goes through one Renovate upstream CI checks lint, validate, alert rules Merge to main patch and minor updates merge themselves; majors wait for me Drift alert main ahead of the hosts nothing deploys itself Fleet update, run by me OS packages, then the playbook DNS and health checks ends by naming what changed Every host
CI without CD. A merge changes the repository; only the fleet update changes a machine.

Deploys aren't automatic, and that's a decision. A deploy needs the password to the encrypted secrets, and putting it in a CI runner would weaken the thing the encryption is for. I'm one operator with one environment, so automatic deploys would buy no coordination, and they would turn every merge into a change to the fleet with no staging in front of it to catch a bad one.

The cost is that merged isn't the same as deployed, and that has caught me out more than once. So the fleet update is one command: it pulls main, upgrades OS packages, applies the playbook, checks that both DNS resolvers agree, runs the health checks and ends by naming what changed. It never reboots anything. A drift alert says when main is ahead of what's running.

Every container image is pinned by its digest, not only its version tag. A version tag once moved in a registry, which would have quietly upgraded a service with a database behind it. A digest can't move.

Backups, by what they survive

Backups are in four layers, and each one is there for a different kind of loss.

Four backup layers Layer one, snapshots every fifteen minutes, undoes a mistake. Layer two, nightly app state, dumps each database and copies each service's data to the ZFS mirror, so one service can be restored. Layer three, nightly VM images, rebuilds a whole virtual machine. Layer four, daily and encrypted, copies to Google Cloud Storage, for losing the house. The layers run from closest and fastest to restore at the top to furthest and surviving the most at the bottom. CLOSEST, FASTEST TO RESTORE 1 Snapshots · every 15 minutes Undo a mistake: data as it was minutes ago. 2 App state · nightly Each database dumped, each service's data copied to the mirror. Restore one service. 3 VM images · nightly A whole virtual machine, ready to boot. 4 Off-site · daily, encrypted To Google Cloud Storage, for losing the house. FURTHEST, SURVIVES THE MOST
Layers one to three stay at home, on the hypervisor's disks. Layer four is the only one that leaves.

A backup I've never restored is a guess, so restores are drilled from a written runbook on a schedule. The off-site restore, pulling an encrypted backup back down from Google Cloud and checking it, has passed, and so has restoring a whole VM and booting it.

One thing I learned the hard way: the backup datasets themselves aren't snapshotted. Snapshots of backups that had already been pruned were holding on to more than a terabyte.

Who watches the watcher

Prometheus, Grafana and Loki collect metrics and logs, and alerts go to Discord. Two rules shape what they watch. A service is up when it answers, so every routed service is probed over HTTPS instead of being trusted because its host looks healthy. And a job that stops running is as bad as one that fails, so every nightly job sends a heartbeat and silence raises an alert.

Both rules came from something going quietly wrong. The nightly VM-image backup failed every night for seven weeks and nothing told me, because its failure emails went to a mailbox nothing reads. Proxmox errors now go to the same channel as every other alert.

Who watches the watcher The service host watches everything: metrics, logs, HTTPS probes and nightly-job heartbeats, and alerts to Discord. The hypervisor watches the service host on its own, with a ping and a DNS question every minute, and alerts to Discord directly when the service host stops answering. SERVICE HOST Watches everything metrics, logs, HTTPS probes, nightly-job heartbeats Discord alerts HYPERVISOR Watches the service host ping and DNS, every minute still there?
The monitor runs on the machine it would most need to report. So a second, much smaller watcher on the other machine checks it.

The gap was the monitor itself. Everything that watches the fleet runs on the service host, and the service host is also the primary DNS. When a kernel fault froze it for over an hour, nothing reported it. The first sign was two phones saying there was no internet. Now the hypervisor checks it every minute, with a ping and a DNS question, and alerts on its own when it stops answering. The kernel is pinned to a version that has run clean, and kernel changes go in one at a time so a fault can be traced to its cause.

A fault that looked like a disk

The hypervisor started flapping between up and down in the alerts many times a day while it was plainly running. Underneath were write timeouts on one drive of the ZFS mirror, and its link dropping to a slower speed, which looks like a failing disk. The drive's own health data was clean and it was barely used. It was the cable. A new one fixed it for twelve weeks. When the same signs came back on the same drive, I moved it to a different port instead of trying another cable, and that fixed it.

The stall also hung a nightly backup. The network share it writes to is a hard mount, which waits instead of failing, so the job sat there until systemd's timeout killed it.

What I keep from it: when a lightly used drive throws timeouts only on writes and drops its link speed while its health data is clean, look at the cable, the port and the power before the disk.

Where it falls short