The Docs Stop Where Operations Begin

Neon separates Postgres into stateless compute and S3-backed storage, and the whole engine is open source. What the repository does not contain is a self-hosting story: there is a development compose file where the operator documentation should be, and since July 2025 there is not even a versioned release to pin. We deployed it for real — auth, durability, recovery, rotation — and wrote down everything the build paid for. This is that report, and the two skills it left behind.

Neon self-hosting: the docs stop where operations begin. We built it, measured what the docs don't say, and left two skills — a Package Skill and a Context Skill — to get you started.
the compose file gets you a stack that runs; nothing tells you how far that is from a deployment you can trust
What we found

The distance between Neon's docker-compose/ and an operated deployment is real, measurable, and nowhere written down. Crossing it took fifteen converges against a live host, and the load-bearing facts came from the pinned source and live probes, not from prose.

  • The compose example is a development fixture — MinIO, trust auth, floating latest tags, fsync=off, a committed signing keypair.
  • There is nothing to pin. Upstream moved to untagged continuous deployment in July 2025; the last deliberate release pairing has to be excavated from git tags.
  • The compute spec is the auth system — undocumented, and readable only in the pinned source.
  • Single-node durability is not what you would guess: a fresh safekeeper cannot serve its offloaded WAL back. We measured the real recovery bound instead of assuming one.

1. What upstream actually gives you

Let's be precise, because "the documentation is zero" would be false. The repository carries an 83-line docs/docker.md, a working docker-compose/ example, and some twenty architecture documents on the pageserver's internals. The architecture docs are excellent — for contributors. The compose example works — as a development fixture. It runs MinIO as the object store, three safekeepers on one machine, cloud_admin with trust authentication, latest image tags, fsync=off in the compute spec, and a signing keypair committed to the repository. Its startup script even loses its own --dev flag to a missing line continuation, which tells you how it is tested: as a fixture, not as a product.

None of this is a criticism of Neon's priorities. Neon-the-company documents Neon-the-cloud-service, which is where its operational knowledge pays their bills; publishing the engine at all is a gift. But it leaves a specific, unglamorous gap: nothing in the repository tells you how to run this with credentials, how durable one node actually is, how to recover one, or how to upgrade one. The docs stop exactly where operations begin.

2. You cannot even pin it

The first operational question — which version? — already has no documented answer. Upstream cut its last versioned releases in July 2025 and moved to untagged continuous deployment. Docker Hub is stale; ghcr.io is canonical, and its latest resolves to a CI build, not to any release. The storage and compute images version on separate trains with separate numbering, so there is no shared release number to match. The last pairing upstream deliberately shipped — storage release-9129, compute release-compute-9073 — has to be excavated from git tags.

Two rules fall out, and both were learned the hard way. Pin both images by tag and digest, because the floating tags move under you. And read docker-compose/ at the pinned tag, never at main: a year of drift separates them, including the compute configuration layout. Our package's validation refuses an undigested image for exactly this reason.

3. The compute spec is the auth system

The dev fixture runs with trust authentication, and no document explains how to run with anything else. The answer lives in the compute spec — the JSON file compute_ctl boots Postgres from — and we got it by reading the pinned source: cluster.roles[].encrypted_password accepts a SCRAM-SHA-256 verifier string directly, cluster.databases[] creates your database, and compute_ctl renders pg_hba.conf itself — trust on the container's own loopback, password auth for everything else. Connections from the host arrive via the Docker bridge gateway, so they do hit the password rule. Generate the verifier once and store it: the salt is random, so regenerating it per converge makes the spec non-deterministic and needlessly recreates the database container every run.

Around that spec sit three traps whose error messages name nothing useful, so here they are verbatim:

What you seeWhat it means
Error: Permission denied (os error 13) — two lines of log, container restart-loopsThe compute container runs as uid 1000, not root. A root-owned 0600 spec is unreadable to it — and every writer of that file, including any rotation rollback path, has to preserve 1000:1000 ownership.
could not start the compute node: File exists (os error 17)A volume sits on the pgdata path. compute_ctl insists on creating pgdata itself; compute local state is a disposable projection of the storage tier, so mount nothing there.
FATAL: lock file "/tmp/.s.PGSQL.55433.lock" already exists after a reboot, foreverThe container's writable /tmp preserves the stale postgres lock across restarts. Mount a tmpfs on /tmp, and treat the compute container as recreate-only — which is also how spec changes apply, since roles are reconciled at startup only.

4. Durability, measured instead of assumed

Point the pageserver and safekeeper at Cloudflare R2 and the uploads genuinely work: layer files, initdb.tar.zst, and 16 MB WAL segments all land under your prefix, and this is the part of Neon's design that earns the enthusiasm. The natural conclusion — WAL is continuously archived to S3, so losing the machine costs seconds of data — is the one we went in with. It is wrong, and we found out by rehearsing recovery on the live deployment rather than writing the runbook from intuition.

What the rehearsals showed: wipe the pageserver's local state on a surviving host and everything comes back — it re-attaches from R2 at the next generation and the still-running safekeeper replays WAL to the latest byte. We deleted 38 MB of local state and read every row back. Delete the safekeeper's volume instead and it also rejoins cleanly. But a fresh safekeeper cannot serve its own offloaded WAL back — so on full host loss, what you recover is what the pageserver had uploaded, and layer uploads follow its checkpoint cadence. The honest single-node recovery point is the activity since the last checkpoint upload — minutes, not seconds. Write that in your runbook instead of the comforting version.

Two instrument readings to distrust while verifying any of this: remote_consistent_lsn reads 0/0 right after any restart or fresh attach even when R2 holds a complete copy, and the /checkpoint API that would force an upload answers Cannot run timeline checkpoint because pageserver was compiled without testing APIs on release images. The way to prove uploads are live is to close a WAL segment with pg_switch_wal() and require a new object beyond a pre-switch listing — an old object satisfying the check would let a broken uploader hide indefinitely.

5. The starting point

Everything above, plus the parts this article skips, now exists in two forms — which is what "a starting point" means concretely.

The neon Package Skill is the tested implementation: one Vultr instance carrying the storage broker, the pageserver, one safekeeper and a Postgres 17 compute, everything bound to loopback, the firewall open on port 22 alone, layers and WAL in R2 under the deployment's own prefix. Its acceptance asks the system what it has — a SQL round-trip, a wrong password refused, a passwordless connection refused, a new WAL segment in R2 — and then runs the same probe from the operator's workstation through the SSH tunnel that is the supported client path. It ships in all three colours, rendering byte-identical artifacts from one colors.yml.

The neon-single-node Context Skill is the knowledge, routed by symptom: the verbatim failure catalogue behind the table above, the release archaeology, the acceptance doctrine, and the pinned version set with its retest conditions. If your agent is staring at Permission denied (os error 13) with no filename, that string routes to the entry that explains it.

One honesty clause, because it is the point of the whole exercise: these claims are true at the pinned versions and unknown after them. The pins are the July-2025 pairing; the Context Skill's pins.md says exactly what to re-verify when they move. Neither skill is affiliated with Neon — and the strongest advertisement for Neon here is one of our own rehearsals: we destroyed the storage engine's entire local state on a live database and the architecture handed every row back.