Both Agents Reported Success. Only One Checked.

1. Why a 200 proves almost nothing

PostHog's ingestion path is four processes: a Rust capture service writes to Kafka, a Node plugin server consumes that topic and produces another, and ClickHouse's Kafka engine tables read that. The HTTP request touches only the first one. Capture answers 200 the moment it has handed the event to Kafka, and every hop after that can fail without the client ever learning.

This is not a quirk of one release. It is the defining property of self-hosted PostHog, and it repeats at every layer: a web container that stays alive with nothing listening, a plugin server that logs Shutting down completed. Exiting... at info level and restarts forever, a Celery worker that never starts because a required async migration is pending — and the migration can normally only be run by Celery. A deployment in any of these states serves a dashboard and looks fine.

So "did the deploy succeed?" is not answerable by the deploy. It is answerable only by reading an event back out of the database it was supposed to land in.

2. Method

Two subagents ran concurrently under Google Antigravity on Gemini 3.7 Flash in high-reasoning mode, with isolated memory, separate filesystems, and no cross-visibility. Neither had read access to the workspace containing the original PostHog package. Both targeted DigitalOcean fra1 on an s-8vcpu-16gb Ubuntu 24.04 droplet, with Cloudflare DNS, Let's Encrypt TLS via Caddy, and OpenTofu state in Cloudflare R2.

Both received create-package-skill, the workflow skill that defines the Clojure DAG, OpenTofu, Ansible and Compose structure. The single variable was a second skill given only to Arm B: posthog-single-node, a knowledge package containing the container topology, the ClickHouse Keeper and named-collection configuration PostHog's migrations require, a catalogue of roughly thirty failures written as symptom → cause → fix, and a set of acceptance criteria.

3. The process cost of not knowing

The wall-clock column is the honest one to dwell on, because it is the least impressive. The skill removed almost none of the elapsed time: converging PostHog is dominated by applying migrations, and migrations take as long as they take. What it removed was work — 492 tool calls of it.

The 49 polling timers are the clearest signature of that work. The baseline built its own convergence loop out of scheduled wake-ups: start something, wait, check whether it came up, retry after the next fix. The skill's playbook encodes the ordering directly — start the datastores alone, flush the ClickHouse system logs, migrate with the application down, then bring the application up — so there is nothing to poll. Zero timers is not an efficiency gain at the margin; it is a different shape of work.

4. What each agent actually verified

We read both agents' acceptance code rather than trusting their self-reports. The baseline's contains health checks and nothing else — no ClickHouse client invocation, no event query, no worker check.

To be precise about what this does and does not show: we are not claiming the baseline's deployment was broken. We are claiming nobody knows, including the agent that built it and reported it complete. Given that the failure mode this stack produces most often is an accepted event that is never stored, "we did not check" and "it works" are not close together.

The skill's acceptance step was written around this. It verifies TLS without -k, resolves the events table from system.tables rather than hardcoding a database name the migrations own, and — because ingestion is asynchronous — polls for the row instead of sampling once. It distinguishes ingested from dropped (2xx, no stored row) from rejected, so a failure names which hop broke.

5. The limitation: the arms did not deploy the same software

This is the part a benchmark write-up is tempted to leave out, and it changes how the architecture comparison should be read.

Arm A pinned a PostHog release from 2022. On that release the architecture in its column is not a mistake — capture really was part of Django, the plugin server really did ship inside the image, and ./bin/plugin-server really was the right command. Its fixes were correct for the software it chose.

So this is not a clean architecture comparison, and we are not going to present it as one. Arm A did not select a legacy topology on the merits; it pinned an old release and inherited one. That was a rational move without domain knowledge — 1.43.0 is roughly where the documented single-server story stops — and it is exactly the kind of decision a curated skill forecloses, since Arm B was told which commit to use and why the application and plugin server must share it.

The consequence for the numbers: the process metrics in section 3 remain a like-for-like comparison of effort on the same task, because both agents were building the same thing to the same acceptance bar. The architecture table is not a quality judgement between two designs. It is a record of where each agent's version choice led.

One more correction while we are here. An earlier draft of the internal report recorded Arm B on ClickHouse 24.x. Its colors.yml pins 26.6.2.158, which matters because PostHog's schema puts TTLs on DateTime64 columns and 24.8 rejects them outright. Had Arm B actually run 24.x, its migrations would have failed. We caught this by reading the deployment rather than the summary — which is the same discipline the article is about.

6. What transfers

The reusable finding is not "skills make agents faster." On this task the skill barely moved the clock. It is that a curated failure catalogue changes what an agent believes it has to prove.

The eight fix commits in Arm A are all real knowledge, discovered the expensive way and then stranded in one agent's transcript. The same eight were already written down for Arm B, which is why it spent its commits on features instead. But the sharper difference is that Arm B's skill told it what a working deployment looks like — a row in ClickHouse, a live Celery worker, a backup object that exists — and so it went and checked. The baseline had no such definition, so it used the one available by default: the deploy command exited zero.

If you take one practice from this: write down the acceptance criteria before the deployment steps. Configuration files tell an agent what to build. The failure catalogue and the acceptance criteria tell it when to stop believing it is done.

The skill used here is open source, as is the earlier analytics benchmark that first made this failure mode visible to us.