Thu 10 Sep · 17:00 CEST · onlineCommunity Town Hall: the roadmap, the three colours, and a live provisioning demoAgenda and registration →
Colors

Field Report & Operations

Six Machines and a Contract Nobody Told the Gates About

Langfuse · Neon · ClickHouse · Vultr · Cloudflare R2September 3, 2026·11 min read

Langfuse's own guidance is one Docker Compose host for testing or Kubernetes for production. We built the shape in between — every component Langfuse says should be separate, on its own machine, reachable only by the peer that needs it — and wrote gates for all of it. The gates were written against Langfuse v3's contract. The deployment was Langfuse v4. Everything was healthy, the ingestion answered 207, and not one trace ever appeared.

Topology of the Langfuse deployment: one VPC holding the app host, a self-hosted Neon on 10.50.0.3:55433, Redis on 10.50.0.8:6379 and three ClickHouse replicas with Keeper voters, with Cloudflare R2 outside it. Blue edges mark the gates that passed by raw TCP (55433, 6379, 8123); a red dashed edge marks the app host refused on Keeper 9181. Beside it, the three contract facts the gates were wrong about: a fresh v4 answers the legacy ingestion endpoint with 207 and stores nothing, the Neon smoke table raised Prisma P3005, and a type=s3 backup disk wrote 128 objects with random names. Below, the two markers .colors-ready and .colors-recovery-verified, and the counts: six machines, eleven converges, three rehearsals, four seconds from POST to read-back.
six machines and eleven converges; the infrastructure was the easy half
What we found

Eleven converges against six Vultr machines, three rehearsal runs, and a four-round adversarial plan review before any of it. The infrastructure was the easy half. The expensive facts were contracts: a storage tier's health check in the wrong database, a backup disk type that hides its own files, and a write mode that rejects your data politely.

  • A fresh Langfuse v4 rejects the legacy ingestion endpoint while answering 207. The raw event still lands in S3, the legacy trace reads are 404 for everyone, and the traces table stays empty by design. Every gate had to be rewritten for OTLP in and the Observations API v2 out.
  • The Neon play's smoke table broke Prisma. P3005 The database schema is not empty on a database nobody had used — except the storage tier proving its own round-trip inside it.
  • A ClickHouse backup returned BACKUP_CREATED and wrote 128 objects with random names. A type=s3 disk keeps the path mapping in local metadata; s3_plain is the one that writes files at their paths.
  • Recovery is a boot, not a row count. The pinned image, in a second Compose project, against both restored databases, with the operator-held keys: the live API keys authenticate, the trace reads back, the encrypted LLM connection decrypts.

Langfuse v4 wants six things: a web container, a worker, Postgres, ClickHouse, Redis, and S3. The Compose file puts all of them on one host and says so: "testing and low-scale deployments". The production answer is Kubernetes. We wanted the third option — six ordinary machines in one VPC, each sized at the documented minimum, with the data tiers reachable only from the app host — and we wanted every one of those separation claims to be something a converge proves rather than something a README asserts.

The Postgres is a self-hosted Neon, rendered from the neon Package Skill's templates off a SHA pin, so there is no second copy of that tier to drift. The ClickHouse is three replicas with their own Keeper voters, derived from the clickhouse package and owned here. Cloudflare R2 holds Neon's layers and WAL, Langfuse's raw events and media, and the backups.

1. Two firewalls, and a denial in the gates

The first design had one Vultr firewall group for all six machines and a subnet-wide allow for the database ports. The adversarial review pointed out what that means: a compromised Redis host can reach Keeper. We had also learned, on the AutoMQ build, that a Vultr firewall group filters the private interface too — and selectively, passing ICMP while dropping TCP, so every node pings every other node while the cluster never forms.

So each role has its own group, every east-west rule names the peer's/32, ufw mirrors the same rules on every host, and the smoke gate proves a refusal as well as the allows: the app host must reach Neon, Redis and the ClickHouse client ports by raw TCP and must be refused on Keeper. An allow-only gate cannot tell a wide firewall from a right one.

  ok    N1 neon 10.50.0.3:55433 reachable by raw TCP
  ok    N1 redis 10.50.0.8:6379 reachable by raw TCP
  ok    N1 clickhouse-0 10.50.0.7:8123 reachable by raw TCP
  ok    N2 app is refused on Keeper 10.50.0.7:9181

One detail the addresses forced: OpenTofu cannot for_eachover instance addresses that do not exist until apply. The per-peer rules are count over static lengths, indexed into the instance list.

2. Three converges spent on ClickHouse not starting

The ClickHouse tier failed to start on all three nodes twice, and the two failures had nothing in common except how badly systemd described them. The first was status=232/ADDRESS_FAMILIES with an empty err.log: the server dies before logging is configured, and only journalctl carries the real exception — SAXParseException: Invalid token ... line 62 column 48. Line 62 was a comment. XML forbids -- inside comments, and we had written "stay -- v4 reads". Python'sxml.dom.minidom now parses every rendered XML file in the golden gate; it costs a second.

The second was Failed with result 'protocol' — aType=notify service that never notified — with the message in clickhouse-server.log:

Code: 36. DB::Exception: If 'engine' is specified for system table,
PARTITION BY parameters should be specified directly inside 'engine'
and 'partition_by' setting doesn't make sense. (BAD_ARGUMENTS)

Langfuse's scaling docs offer an <engine> override to put TTLs on the system log tables. The packagedconfig.xml already carries partition_by for those tables, and a config.d override merges beside it rather than replacing it. The <ttl> setting does the same job without the collision. We kept query_log— v4 reads system.query_log* — and removed the six tables Langfuse never reads.

3. The storage tier left a table in the application database

With ClickHouse and Redis converged and their gates green, the app tier came up and langfuse-web restart-looped:

Error: P3005
The database schema is not empty. Read more about how to baseline an
existing production database: https://pris.ly/d/migrate-baseline
Applying database migrations failed. Common causes:
  1. The database is unavailable or unreachable.
  2. DATABASE_URL / DIRECT_URL credentials contain special characters

Neither cause. The database was brand new. It was also not empty: the imported Neon play proves its SQL round-trip by creating acolors_smoke table — inside the application database, because that is the database it provisions. Prisma refuses to apply 436 migrations to a schema that has tables but no_prisma_migrations. Any Neon-provisioned database handed to a Prisma application meets this, and the fix is exactly one drop, guarded on the migrations table not existing yet; every later converge recreates the smoke table and Prisma no longer cares.

Two more from the same converge, smaller: the compute reportsSHOW timezone as GMT because Neon's spec sets no TimeZone, and Langfuse wants the literalUTC on both databases (ALTER DATABASE ... SET timezone, by the owning role); and the documented ClickHouse grant list is exactly what Langfuse needs and not one column more, so the package's own gates — clusterAllReplicas oversystem.one — needed their own grants, labelled as ours.

4. Healthy, 207, and no traces

Migrations ran on both databases. Health with the database flag: 200. Worker health with the stuck-queue flag: 200. The ingestion batch: 207. A new raw-event object in R2. And the trace never became readable.

warn  Rejected 2 event(s) from the legacy /api/public/ingestion endpoint
      for project langfuse-vultr because this Langfuse v4 deployment runs
      in events_only mode. These events were not stored.

Every gate had been written from v3 knowledge, and the deployment was v4.27.0. A fresh v4 runs the events_only write mode: the legacy batch endpoint answers 207 while rejecting every event type except scores, GET /api/public/traces/:id is 404 —for everyone, so the negative gate expecting 401 on that route failed too — and the data model is events_full, withtraces and observations migrated and empty. The raw event landed in S3 because Langfuse writes the payload before deciding.

The rewrite was the honest one: OTLP/HTTP JSON to/api/public/otel/v1/traces withx-langfuse-ingestion-version: 4 (without the header, data appears up to ten minutes late), reads throughGET /api/public/v2/observations?traceId=, scores throughPOST /api/public/scores, ClickHouse evidence inevents_full on node 0 and on the last replica, negatives on a v2 route. The probe that settled the contract took four seconds from POST to read-back. Two smaller lies in the same run:POST /api/public/media answers 201 for a flow that works end to end, and a Redis NOAUTH reply is 33 bytes, which ahead -c 64 waits on until the timeout kills it and the bytes are lost.

  ok    A6 v4 write mode confirmed: legacy batch 207, legacy trace read 404
  ok    I2 trace fe2c8ea5... readable through the Observations API v2
  ok    I7 event rows replicated to 10.50.0.4
  ok    P1 200 traces queryable within 120s

5. The backup that wrote 128 objects with no names

The rehearsal's first step is a fresh ClickHouse backup:BACKUP DATABASE default TO Disk('backups', '<stamp>/')to an S3 disk on the backup bucket, so the credential lives in the disk configuration and never in SQL or query_log. It returned BACKUP_CREATED, 270 files. The bucket held 128 objects named agp/jjppeptxopeoqjzmlhpvtwrwopqkb and nothing under the stamp.

A type=s3 disk stores objects under random keys and keeps the path mapping in local metadata under/var/lib/clickhouse/disks/. The backup was real and only that node could ever read it. s3_plain writes every file at its own path — .backup, data/,metadata/ — which is what makes a set listable by a script and restorable from anywhere. Switching types on a live node means stopping the server and removing the old metadata directory.

6. Restore is a boot

The review had rejected our first restore rehearsal as "a dump-import test". It was right. The one that shipped restores both stores and then boots the pinned web image in a second Compose project on loopback, against the restored databases, with the operator-heldENCRYPTION_KEY and SALT and a freshNEXTAUTH_SECRET, and asks through the API: do the live project keys authenticate (hashed keys, salt, Postgres)? Does the smoke trace read back with its generation (ClickHouse restore usable through the app)? Does the seeded LLM connection return its display secret (the row decrypts)?

The first run of that answered the second question with a 500: thelangfuse user's grants are ON default.*, and the restored database is restore_check. One more grant. The third run passed, then stopped a replica under ingestion, restarted Redis with a job queued, and wrote a second marker beside the first:

.colors-ready                langfuse-vultr
.colors-recovery-verified    langfuse-vultr clickhouse=20260903T071332Z postgres=20260903T071340Z

Two markers because "the service answers" and "the service can be recovered" are different claims, and automation should be able to tell them apart.

7. The pairing rule, and the one the review caught

Postgres dumps every six hours; ClickHouse backs up nightly. We do not quiesce the two for a coordinated snapshot — pausing ingestion on an observability system every six hours is the wrong trade — so a restore pairs a ClickHouse set with a Postgres dump, and the direction matters. Our first draft recorded, at backup time, the newest dumpbefore the ClickHouse set. Codex pointed out that pairs a newer ClickHouse snapshot with an older Postgres one, and a project created between the two would be referenced by traces and absent from the database. The rule that shipped is decided at restore time: the oldest dump completed after the ClickHouse set, so Postgres is always the newer snapshot, and a set with no later dump is refused rather than paired backwards. The rehearsal printed it:

clickhouse-restore-check: restored 20260903T071332Z into restore_check,
  paired with postgres set 20260903T071340Z

What we left behind

Two skills, both public. The Langfuse Package Skill converges all six machines from one colors.yml, in green, and carriesrehearse and describe beside the lifecycle verbs. The Langfuse Multi-Node Context Skill carries the rest: fifteen failures with their verbatim text, the pinned set with its retest conditions, the acceptance doctrine, and seven evals shaped as the support message you would actually send.

Splitting Langfuse across machines turned out to be the easy half. The contract of the thing being deployed was the expensive half, and it was expensive precisely because everything was healthy while we were wrong about it.