The Converge Passed in 614 Milliseconds

n8n stores your workflows, your execution history, and every API credential your automations use. The usual advice is SQLite for a trial and a colocated Postgres for anything real. We tried a third thing: a colocated self-hosted Neon, so the durable copy of all of that lives in object storage rather than on the instance's disk.

The obvious objection is latency. Neon separates storage from compute — a pageserver, a safekeeper, and a stateless Postgres — and n8n writes an execution row per run. Storage/compute separation is supposed to be the wrong shape for that.

So we measured it rather than arguing about it.

soak: 7950 executions, 0 failed, 10 workflows
sql round-trip p95=75ms p99=80ms
host memory 12%  disk 9%

Five minutes, roughly twenty-six executions a second, zero failures, on one 8-core box that was also running the entire storage tier. Two independent runs produced identical percentiles. The interesting number is the gap between p95 and p99: five milliseconds. At this rate the split adds no visible tail latency at all.

That is the good news, and it is the least useful thing we learned.

1. The green run that proved nothing

Getting there took sixteen converges. The one worth writing about is the first one that fully passed.

>>> :n8n/acceptance (create)
<<< :n8n/acceptance (614ms)
EXIT=0

Every stage green. Acceptance green. And not one application gate had executed.

The smoke script was written, rendered, installed onto the host — and never called. We had inherited an acceptance step from the storage-tier package we build on, and assumed inheriting the step meant inheriting the gates. It doesn't. That step checks the operator's client path; the server-side gates live in the playbook, and a downstream playbook has to invoke its own.

What caught it was arithmetic, not insight. The gate suite does apg_switch_wal() and then waits twenty seconds for the segment to appear in object storage. A suite that finishes in 614 milliseconds is not running. The real one takes twenty-six seconds.

Time your acceptance stage, and disbelieve a fast pass.A test suite that completes faster than its own sleeps has told you nothing, and it has told you nothing in the most reassuring possible voice.

2. Three more gates that lied

Once the gates ran, they started failing. Six failures — and only two were the infrastructure's fault. The other four were defects in the checks themselves, and each one failed in a different direction.

The gate that accused a correct host

A gate reported password authentication is ENABLED on a host where sshd plainly had it disabled. The configuration was right; the gate called sshd -T once per assertion, and a transient empty result is indistinguishable from a wrong setting. A second gate checking the same file passed in the same run.

A gate that can accuse correct configuration is worse than no gate, because it teaches you to disbelieve the suite. Capture the probe once, and make emptiness its own distinct failure.

The gate that could never have passed

Our restore drill checked that a stored credential still decrypted by reading it back through the API and comparing. It reported that the encryption key had not survived the restore — the most alarming possible result in a recovery test.

It hadn't. n8n redacts credential values in API responses, returning a fixed-length sentinel: 54 characters whether the secret was 27 characters or 28. The comparison could never have succeeded, on any instance, with any key.

The honest test is to use the credential. n8n resolves and decrypts a node's credential at execution time and fails the node if it can't — so the drill now runs a workflow whose HTTP node carries the credential and asserts the execution reaches success.

The gate that certified an empty database

The same drill reported that the pinned image had booted successfully against the restored data. It had. The restored database wasempty — pg_restore had failed and we were swallowing its output — and n8n cheerfully ran its own migrations against the blank database and declared itself ready.

Health is not evidence of content. The drill now asserts the table count first, then the rows.

3. The stack was green and two containers had never started

A later converge finished with ok=45 changed=0, no failures, and the site down. Two containers sat in Docker statecreated: started never, by anything.

An earlier run had failed their health dependency and left them that way. Every run since had changed no file — so the handler that brings the stack up was never notified, and never fired.

A handler expresses "an input changed", not "the world should look like this." Desired-state convergence needs the enforcing command to run unconditionally. docker compose up -d is idempotent; on a converged host it costs nothing and it is the difference between a green playbook and a running service.

Only two gates caught it — the certificate check and the one that executes a workflow on the task runner. Every task in the playbook was green. That is the argument for gating end state rather than task outcomes, made about as concretely as it can be made.

4. What n8n 2.x actually does

A separate category of problem, and the one most likely to bite a reader directly: n8n 2.x moved things that every guide still describes the old way. All of these came from the deployed version's own environment-variable reference, and several contradict n8n's own 2.0 breaking-changes page.

The webhook one is the expensive mistake. With the deprecated key, n8n falls back to its own host and port and hands third parties a URL that never resolves — and webhook URLs given to third parties are effectively permanent.

Since pruning is already on, the execution table isn't what fills your host. These two are, and both default to the unsafe value:N8N_DEFAULT_BINARY_DATA_MODE defaults todefault, which holds binary payloads in memory, andN8N_CONCURRENCY_PRODUCTION_LIMIT defaults to-1, unbounded. A Code node duplicates its payload before and after processing, so peak memory is roughly three times the largest payload times the concurrency bound. That product sizes the machine, not the average workload.

5. The one that poisons the disk

n8n writes N8N_ENCRYPTION_KEY into its settings file onfirst boot and refuses to start ever after if the environment disagrees:

Error: Mismatching encryption keys. The encryption key in the settings file
/home/node/.n8n/config does not match the N8N_ENCRYPTION_KEY env var.

One bad first boot — a wrong key, or an unrendered template expression that reached the container as a literal string, which is exactly how ours happened — poisons the data directory permanently. And the error appears on the next boot, so by the time you see it the cause has scrolled off.

Our converge now detects the mismatch and refuses to repair it. Deleting the settings file is correct only when the database holds no encrypted credentials; when it does, that file is the only thing that can decrypt them, and removing it destroys them silently. Auto-repair here would be a data-loss bug wearing a helpful face.

6. Encode the trap, don't write it down

The most uncomfortable finding is about us.

Ansible splits a shell task's arguments before running anything, counting brace pairs across the whole block — comments included. Two consecutive opening braces anywhere and the playbook won't load. This trap is documented by name in the Context Skill from our previous build. We read that skill during planning. We copied the trap into the plan.

Then we walked into it while writing a guard whose entire purpose was to detect unrendered template braces — three times, including once where thecomment explaining the trap triggered it.

Prose gets read once. A gate runs every time.

So the package's validator now refuses the deprecatedWEBHOOK_URL spelling by name, refuses a task-runner image whose version differs from the main image, refuses binary data left in memory, refuses an unbounded concurrency limit, and refuses a Cloudflare-only firewall without a proxied DNS record — which would otherwise pass the converge and fail hours later with no certificate, because the ACME challenge arrives from an address the firewall drops.

7. Recovery, and the number nobody states

Neon uploads WAL to object storage continuously, which reads like continuous backup. It isn't. A rebuilt safekeeper does not recover its offloaded WAL — the walproposer bootstraps it from the compute basebackup instead.

So losing the whole host falls back to the logical backup set, which makes the backup interval the real recovery point objective. For a system holding live third-party credentials, nightly isn't defensible. Ours runs every six hours, and the dump is small.

The rehearsal restores both artifacts into an isolated scratch stack, boots the pinned image, and proves four things a checksum cannot: that an operator can actually log in, that the workflows are there, that a stored credential decrypts, and that a binary payload is readable. Whole-host recovery restores the database and the data directory but not the host's generated secrets — so a restore nobody can sign into is not a recovery.

What we left behind

Two skills, both public. Then8n Package Skill converges the whole thing from a single colors.yml — and notably contains no copy of the storage tier at all: it pins the Neon package and renders its templates directly, so there's nothing to drift.

The n8n Single-Node Context Skill carries the rest: fifteen failures with their verbatim error text, distilled from forty-two recorded during the build, plus the version table above, the pinned set with its retest conditions, and the acceptance doctrine — indexed by the symptom you'd actually paste into a search box.

The database question turned out to have a boring answer: Neon handles n8n's write pattern with an order of magnitude to spare. The gates were the interesting part. Four of ours were wrong, and the deployment they were guarding was fine the whole time.