Database

Alright, let’s talk about the “heart” of each service: the database !

To start off, let’s address the most prominent point: using a managed service or rolling our own. This question will likely be there until humanity exists and surely, we also thought about it when starting out. Outlining all arguments for each case would fill multiple pages, so we’ll focus on explaining what we’ve decided to do and why:

We went with Postgres deployed through autobase on individual VMs in a HA-setup.

Reasoning

Postgres is the de-facto standard for large-scale DB needs. It is FOSS, used by the majority of projects (likely) and has a strong ecosystem around extensions and tools (like autobase) which make it realistic to administrate a large DB at scale. In addition, maintaining a self-hosted instance likely saves the project from financial death. The costs for managed DBs (compared to self-hosted ones) are super high. And once the instance requirements grow, the costs grow with it exponentially.

We believe that it must be possible to run a production-grade Postgres service in 2025+ without a managed service.

And just to get something clear: we are not just starting out with Postgres 😉️ We have substantial experience in running databases, including Postgres, in production environments. Yet, “production” is always different and a project like CodeFloe has the potential to grow to another level we have not experienced before. This means both excitement and challenges ahead! But we are confident that we can handle it, together with the community and an open & transparent approach.

Technical Setup

The database is run in a three-node setup with automatic failover and replication being orchestrated through patroni. Backups are executed via pgbackrest to two distinct (S3) locations. autobase allows to add additional nodes to the cluster if needed. We can also perform seamless major upgrades by just executing an Ansible playbook.

Each node has a connection pooler (pgbouncer) in front. On top, each pooler has a load balancer (LB) (HAProxy) in front which serves as the central entrypoint for any traffic. The LB is able to contact the connection pooler instances in a “round-robin” fashion, ensuring it always finds a healthy one in case one of the poolers is down. In the same manner, any client can use the LBs as the primary connection point(s) for the database.

For example, Forgejo is able to make use of a so-called “EngineGroup” connection (starting with v12) which allows specifying multiple connections for primary and replica nodes, respectively. The connection pool subsequently contains the respective entrypoints of the HAProxy LBs sitting in front of the connection poolers.

Latency and IOPS

Both of these characteristics are known to be highly important for a database system.

To get some numbers, we ran the following pgbench benchmark:

sudo -u postgres createdb pgbench_test
sudo -u postgres pgbench -i -s 10 pgbench_test
sudo -u postgres pgbench -c 30 -j 4 -T 120 pgbench_test # write
sudo -u postgres pgbench -S -c 30 -j 4 -T 120 pgbench_test # read

which resulted in

Host	Write TPS	Read TPS	Read Latency (ms)	Write Latency (ms)
demeter	19971	160344	0.187	0.256

While the results for a pgbench benchmark are only comparable across the same test setup, the numbers give some indication of the overall performance characteristics of the database system.