Skip to content
CodeFloe

Architecture

This page shows every machine CodeFloe runs on, which cluster or provider it sits in, and the main paths between them.

CodeFloe production infrastructure: Bunny DNS sends visitors in Asia and Oceania to the edge in Singapore and the Americas to the one in Ashburn; the edges and European visitors reach the HAProxy ingress on the three Ceph hosts, which fronts forgejo-1 and the Pages nodes; Hetzner Cloud carries misc and a nine-node Talos Kubernetes cluster; the CI agents and the three-node database sit below; object storage, email and monitoring along the bottom.

Everything that is not a public entry point talks over a Headscale tailnet. The CI agents on the Proxmox clusters have no public address at all, which is why a runner reaches the Crow server and Forgejo the same way whether it sits in a datacenter or on a desk.

Most of CodeFloe now runs as virtual machines on two self-managed Proxmox VE clusters owned by devXY, both built from rented Hetzner bare metal servers. They are separate clusters with separate hardware, and the difference between them decides what is allowed to live where.

The Ceph cluster is formed out of three AX102 machines with Ceph replica-3 across them, so a VM’s disk does not belong to the host it happens to run on. That brings live migration and failover, which is what Forgejo and Pages need.

The local cluster is composed out of zeus, hera, apollo and poseidon with node-local RAID1 and no Ceph. No shared storage, direct use of the attached NVMEs. This suits the database and CI: both want local raw NVMe connectivity for max performance.

The hardware page lists the per-VM specifications and what each one costs.

Since 13 November 2026 codefloe.com is served by three Forgejo nodes at once, ceph-ax102-forgejo-ha-1 to -3, one on each Ceph host. They run Arbinel, our soft fork of Forgejo, which adds what running several nodes against one database needs.

CodeFloe's forge in high availability: users reach the edge HAProxy directly or through the regional edges in Singapore and Virginia; the edge balances across three Forgejo nodes running Arbinel, zoekt and HAProxy; all three share the repositories on CephFS, PostgreSQL, sessions and queues in Valkey and Typesense for issue search (both on Kubernetes), and S3 and Garage for files.

The nodes have no public address. The entry point in Germany is a shared HAProxy that runs on all three Ceph hosts behind a floating address, the same one that fronts Pages: it terminates TLS and HTTP/3, picks the backend from the name the visitor asked for, and reaches the nodes over the cluster’s private 10 Gb/s network. It hashes each client’s /16 (/64 for IPv6) onto one node, so that node’s rate limits see all of that client’s requests, and checks every node’s /api/healthz. A node that is about to stop answers that check with 503 first, so the edge takes it out of rotation while it finishes what it is serving. Visitors in Asia, Oceania and the Americas reach the same HAProxy through a regional edge instead.

Each node runs its own HAProxy behind the edge, which is where the defense rules live and where git over SSH is proxied to sshd; the nodes’ HAProxies share their stick tables as peers. The edge forwards the client’s address in the PROXY protocol, so the rate limits and bot rules key on the visitor rather than on the hypervisor. sshd asks Forgejo for the key behind each login, and every node presents the same host keys, so it does not matter which node a connection lands on.

What the nodes share:

  • Repositories live on the forgejo-prod CephFS subvolume, mounted on every node; Ceph keeps three copies of every file.
  • Sessions, cache and job queues are in Valkey with Sentinel on the k8s-ceph-fsn1 cluster: a primary, two replicas and three Sentinels, reached over TLS.
  • The database is the PostgreSQL cluster on the local cluster; each node reaches it through a pooler of its own.
  • Attachments, LFS objects, avatars and packages are in object storage, and repository archives in Garage on artemis.
  • Issue search is served by a Typesense cluster on Kubernetes, one index every node writes to and reads from. Forgejo’s built-in issue index is a set of files on one server, which several nodes cannot share.

What each node keeps to itself is the zoekt code-search index, about 80 GB. zoekt memory-maps its index files, which wants a local disk, so every node has its own copy and catches up with the database every minute: a push is indexed right away by the node that received it, and the others notice the new commit and index it too.

Scheduled jobs and the incoming mailbox run on one node only: the nodes elect a leader through a lease row in PostgreSQL, and another node takes over within seconds when the leader stops renewing it. Deploys and the weekly restarts go one node at a time, each node draining first; only a schema migration stops all three at once.

It also hands the landing pages to an nginx on misc, so the front page and the forge share one hostname and one certificate.

The CDN is not in front of any of that. Nothing the forge serves passes through a cache CodeFloe does not control. Two subdomains are delegated to it instead: static.www.codefloe.com for the landing page site’s assets, which it pulls from the site itself, and cdn.forum.codefloe.com for the forum’s, which it pulls from object storage.

Forgejo and the database stay in Germany. For a visitor far away, a page load is mostly waiting for the network: from Tokyo every round trip to Germany takes about 240 ms, and a first visit needs four of them before the page starts to arrive.

Since 2026-11-01, two small VMs of CodeFloe’s own shorten that path:

  • edge-sin1 in Singapore for Asia and Oceania
  • edge-ash1 in Ashburn, Virginia, for the Americas

An edge sets up the connection close to the visitor, serves the static assets from its cache, and passes everything else on to Germany over connections it keeps open. Pages themselves are never cached, and the forge’s rate limits still see the visitor’s real address. Git over SSH is passed through unchanged.

Bunny DNS sends each visitor to the nearest site; Europe, Africa and the Middle East still go straight to Germany. An edge that stops answering drops out of DNS on its own. Only codefloe.com, www.codefloe.com and git over SSH use the edges; Pages, CI and every other service do not.

An edge decrypts HTTPS, so while it passes a request on to Germany it handles the request in the clear, the same as the HAProxy in Germany does. It does not log visitor traffic, and the only thing it stores is its cache of the public static assets under /assets/. Git over SSH is not decrypted: the edge only forwards the encrypted connection, so it sees the addresses and the amount of data, not its content.

codefloe.com and www.codefloe.com resolve to one of these sets, depending on where you are:

Site Serves IPv4 IPv6
Germany (Falkenstein) Europe, Africa and Middle East 142.132.154.74 2a01:4f8:261:2a16::2
142.132.154.75 2a01:4f8:261:2a14::2
142.132.154.80 2a01:4f8:261:2a0b::2
edge-sin1 (Singapore) Asia and Oceania 5.223.62.53 2a01:4ff:2f0:1b33::1
edge-ash1 (Ashburn) the Americas 178.156.209.36 2a01:4ff:f0:c444::1

If you allowlist CodeFloe in a firewall, allow all of them: when an edge is down, DNS sends its visitors to Germany instead. We announce changes to these addresses on the forum before they happen.

To keep your traffic away from an edge, pin codefloe.com to the addresses in Germany. In /etc/hosts (on Windows C:\Windows\System32\drivers\etc\hosts):

142.132.154.74       codefloe.com www.codefloe.com
142.132.154.75       codefloe.com www.codefloe.com
142.132.154.80       codefloe.com www.codefloe.com
2a01:4f8:261:2a16::2 codefloe.com www.codefloe.com
2a01:4f8:261:2a14::2 codefloe.com www.codefloe.com
2a01:4f8:261:2a0b::2 codefloe.com www.codefloe.com

HTTPS and git over SSH from that machine then go straight to Germany and never reach an edge. curl -sv https://codefloe.com -o /dev/null 2>&1 | grep Connected shows which address you reached. This only applies to machines that have the entry, so add it to CI runners and other devices too.

How long https://codefloe.com/explore/repos takes until it is visible on a first visit, measured with WebPageTest in desktop Chrome, median of three runs:

Visitor in Served via Before (2026-10-29) After (2026-11-02)
Tokyo Singapore 2.1 s 0.8 s
Toronto Ashburn 1.0 s 0.7 s
Frankfurt Germany 0.5 s 0.5 s

From Tokyo the whole page now finishes loading in 1.4 s instead of 3.6 s. Repeat visits gain less, because the browser already has the assets and only the page itself still has to come from Germany.

Single requests from other cities (Globalping, 2026-11-01) got 47-65 % faster from Seoul, Singapore and Sydney, 46 % from New York and 22 % from São Paulo. Mumbai gains nothing: DNS sends it to Singapore, the nearer site, but its direct route to Germany is just as fast.

The same test against the repository overview of three other European forges:

Visitor in CodeFloe pushin.eu opencommit.eu Codeberg
Toronto 0.7 s 0.5 s 1.2 s 3.7 s
Tokyo 0.8 s 0.6 s 2.0 s 4.9 s
  • pushin.eu runs its own software behind a CDN and sends a much lighter page: 164 KB against CodeFloe’s 815 KB.
  • opencommit.eu runs Forgejo in Helsinki, with a page about the size of CodeFloe’s.
  • Codeberg runs Forgejo in Germany and took 2.2-3.4 s to start sending the page, so most of its time is spent on the server rather than the network.

Measured between 2026-11-02 and 2026-11-04, Codeberg on 2026-10-29.

Three VMs spread across three “Local” PVE hosts run PostgreSQL through autobase, with Patroni for failover and etcd for coordination. Each node has pgBouncer in front of it and HAProxy in front of that, so a client connects to a load balancer rather than to a node it has to track.

Splitting the database off the CI machines means a heavy build can no longer cost the forge database latency. See the database page for the topology and the measured throughput.

Both CI systems share one pool of agents. artemis is a dedicated AX42, ci-1 to ci-5 are VMs on the local cluster, and gaia, rhea and theia are arm64 Mac Minis.

The control planes are split: the Crow CI server runs on misc, while Forgejo Actions is part of Forgejo on forgejo-1. The amd64 agents register with both, while the Mac Minis are scheduled independently. The CI/CD hardware page has the per-runner capacity and how it is divided.

artemis additionally runs roostd, the daemon behind preview environments. It owns a Caddy instance and starts one container or static site per open pull request.

Three VMs on our Ceph PVE cluster, one per host, each holding a complete copy of every site. HAProxy on all three cluster hosts terminates the connection and picks the site from the hostname, so losing a node costs nothing.

The Pages technical details cover publishing, addresses and limits.

Nine Hetzner Cloud nodes running Talos Linux carry the services that are a poor fit for a single VM. Everything on the cluster is deployed by Argo CD from the codefloe/kubernetes repository rather than by Ansible, and reached through one floating IP, so every hostname below resolves to the same address.

  • auth.codefloe.com is Zitadel, the identity provider the platform’s own services sign in against.
  • openbao.codefloe.com is OpenBao, the source of truth for every operational credential. It is CodeFloe’s own instance, not a hosted one: the secret store the deployment automation depends on runs on CodeFloe’s own hardware.
  • cr.codefloe.com is the container proxy cache, a Harbor instance.
  • translate.codefloe.com is Weblate.
  • renovate.codefloe.com is the hosted Renovate operator, with webhook.renovate.codefloe.com taking the forge’s webhooks so a run starts on a push rather than on a timer.

Typesense runs here too, without a hostname: it is reached privately by the three Forgejo nodes, and it is the one thing on this cluster the forge cannot do without. Three replicas keep the index behind Raft, and it is rebuildable from Postgres, so losing it costs a reindex rather than data. The backup operator runs here as well.

The hardware page lists the node pools.

The following external services are used:

  • Object storage at Scaleway holds Forgejo’s packages and attachments, the forum’s uploads and the container registry, and receives the off-site backups in a separate project and region. See storage and backups.
  • Email runs on Scaleway: Transactional Email sends the notifications, and Scaleway Mailbox receives the replies. See email.
  • Grafana and ntfy collect metrics and deliver alerts. The Grafana is devXY’s, borrowed until CodeFloe has its own. See monitoring.

Secrets are deliberately not on that list. OpenBao runs on the Kubernetes cluster above, because a deploy that cannot reach its secret store cannot run at all, and that is not a dependency worth handing to someone else.

CodeFloe runs a second, smaller copy of production. It exists so that an Ansible change or a Forgejo upgrade is applied somewhere disposable first.

The development environment

Five Hetzner CAX11 instances, the cheapest arm64 machines available, mirror the shape of production without its capacity.

The development environment: git-dev and misc-dev both talk to a three-node Postgres cluster, all on Hetzner CAX11 instances, with three separate ceph nodes below.

The differences from production are deliberate:

  • There is no Proxmox cluster and no shared storage. Everything is a cloud instance, so a rebuild costs nothing.
  • misc-dev is both the Crow server and its only agent. Development has no separate CI fleet.
  • The three ceph nodes are a storage experiment alongside the five instances, not part of the environment a change is tested against.

Because dev runs on four vCPUs per node against production’s dozens, it is useful for checking that a change applies cleanly and useless for judging how it performs. A load test in dev says nothing about how production behaves under the same traffic.