Three Analytics Stacks, Three Agents, One Uncomfortable Finding

1. Method

Three cold subagents were spawned concurrently with no shared memory and no cross-visibility, each forbidden from inspecting the others' files. Prompts came from a single template parameterised only by engine-specific nouns. Each agent owned two repositories — a Package Skill and a deployment of desired state — and was required to reach a live, TLS-terminated endpoint on DigitalOcean with backups to Cloudflare R2.

Two controls did not hold, and we record them rather than describe the run as clean. Credential variable names, droplet names and backup schedules diverged between arms, so the prompt template was less symmetric than intended. And PostHog was later resized from 8 GiB to 16 GiB after the smaller machine wedged so hard that sshd could not fork — upstream's own hobby minimum is 16 GiB, so the equal-hardware control was arguably never fair to that arm.

2. What the agents built

The three targets diverge sharply once they actually work, which is what makes the comparison interesting.

PostHog needed five tiers the original implementation did not have: ClickHouse Keeper, Kafka, Temporal, a Rust capture service, and a Node plugin server. None were optional. For scale, PostHog's own single-server compose defines roughly 35 services; the finished Package Skill runs 10 and is a deliberate reduction.

3. What the gates concealed

The PostHog arm is the clearest case. It reported convergence in 122 seconds against a deployment that had never applied a migration, and it stayed green because three mechanisms cooperated:

• No migration step existed at all.
• POSTHOG_SKIP_MIGRATION_CHECKS=1 suppressed the error that would have said so.
• The health gate ran pg_isready against the database container — so it asked PostgreSQL whether PostgreSQL was up, and reported that the application was healthy.

ClickHouse held zero tables. The site served 502s. Every check passed.

The same signature appeared across all three arms, and it is worth stating as a general pattern: almost none of the defects were logic errors. They were assembly errors, and they shared one shape — something reports success while the thing that matters is broken.

• docker compose ps reporting running through 48 restarts.
• An unhandled startup exception logged below info, so a crash looked like a clean exit.
• A configuration file correct on the host while the container served the inode it booted with — twice, in two different services, because a single-file bind mount pins an inode and Ansible replaces files by rename.
• A capture endpoint returning 200 into a Kafka topic that no consumer read.
• Umami's compose file shipping {{ lookup('env', …) }} as a literal string, because ansible.builtin.copy does not template — so the credential variable was inert and the real password was a constant published in a public repository.

The checks that could not fail

Three of these surfaced only after publication, and they are the sharpest examples in the set, because in each case a green result was structurally incapable of being anything else.

• Rybbit's synthetic event was never valid. The acceptance step posted name where the API discriminates on type, and the request had always been rejected with a 400. Nobody noticed because the step reports not-configured and sends nothing when no site exists — which was true of every converge until someone added one. The check only became reachable once there was something to check.
• Celery had never started. PostHog holds a third migration system, separate from Django's and ClickHouse's, and a pending async migration stops the worker booting — 74 restarts. Acceptance passed throughout, because the path it exercises (capture, Kafka, ClickHouse) never touches Celery. It took PostHog's own setup page to say so.
• The verification locked the front door. The acceptance step asks for a project, which creates an organization, after which PostHog's hosted realm reports can_create_org: false and the signup page becomes an invite wall. A converge reliably produced a UI that nobody could log into, as a side effect of checking it worked.

All three were introduced by the audit rather than by the agents — the wrong payload, the skipped migration checks, the project-creating probe are ours. That is worth stating plainly: the failure mode this article is about is not a property of generated code. It is a property of verification, and it caught the people writing the verification too.

The gates now assert what they previously assumed: background jobs are checked by asking PostHog whether Celery is alive and no async migration is pending, and the synthetic event goes to a dedicated throwaway site so a converge stops writing test rows into real analytics.

4. Security defects

All three arms shipped live security problems, all now fixed:

• Umami answered to its seeded admin/umami login on the public internet — verified with an HTTP 200 and a valid session token. Its APP_SECRET and database password were constants in a public repository.
• PostHog shipped a hardcoded Django SECRET_KEY in a public repository, which signs session cookies and password-reset tokens, plus a fixed posthog:posthog database password.
• Rybbit enabled open registration with no way to express otherwise in desired state.

5. Disaster recovery that had never worked

Rybbit's backup unit had failed on every scheduled run. Its ClickHouse step named a host path from inside a container, which the server refuses; the fallback then took a hot tar of the live data directory, lost its race with running merges, and aborted the script before the upload. Storage held exactly one object, from a single lucky execution.

All three now use ClickHouse's native BACKUP statement, restore every dump into a scratch database before uploading it, and prune the bucket as well as the local disk. An archive that exists is not an archive that restores.

6. Reproducibility

Floating image tags caused two independent failures. ClickHouse 24.8 could not apply PostHog's schema at all. And posthog/posthog:latest and posthog/posthog-node:latest turned out to be built from different commits, so the plugin server queried a column the application's migrations had never created — the ingestion consumer died on its first message, and events accumulated in Kafka while the API returned 200.

Both images are now pinned to one commit, with a test asserting they are equal, and the remaining floating tags across all three arms — Rybbit’s backend and client, PostHog’s capture service — are pinned by digest. Doing so required relaxing the packages’ own validation, which required the mutable name:tag form and rejected a digest outright: the safest pin was unrepresentable. A cold PostHog deployment also costs 68 minutes of CPU-bound Django migrations, so the package now commits an 854 KB plain-SQL schema checkpoint, restored only into an empty database and only when its stamped commit matches the image.

7. Scorecard

Audit commits are the cost of making a reported success true. The ratio — 10, 19 and 75 against 4, 3 and 6 agent commits — tracks target complexity far more closely than it tracks anything about the agents.

8. What we changed about verification

The gates were rewritten so that a passing result means something. Each acceptance step now verifies TLS without disabling certificate checks, sends a synthetic event and reads it back out of the datastore, and confirms the backup drill by finding a fresh, non-empty object in storage.

Crucially, the verdict distinguishes outcomes that a status code cannot: ingested, rejected, not-configured — and dropped, meaning the endpoint accepted the event and nothing was stored. That last case is the one that had been invisible, and it caught three genuinely different ingestion failures in a row before PostHog worked.

PostHog's ingestion was ultimately confirmed twice: a synthetic event returned by ClickHouse within five seconds with the correct team, and the same event returned by the product's own query engine — the path its UI uses.

9. Conclusion

The interesting result is not that three agents built three analytics stacks. It is that all three produced work that passed its own verification while being, in one case, entirely non-functional.

Lines of code and commit counts measure the difficulty of each target, not model capability — one trial per arm, three targets of wildly unequal complexity, no baseline. The transferable finding is narrower and more useful: an agent's self-verification is only as good as the weakest assertion in it. A fast green result is evidence about the gate before it is evidence about the system.

If you take one practice from this, take the read-back: never accept a status code as proof of an effect. Ask the datastore.