The Cluster Was Fine

1. Replication factor 1, on purpose

AutoMQ speaks Apache Kafka's protocol — 3.9.1, at the version we pinned — but replaces the replicated local disks with object storage. A produce is acknowledged once the record is in the bucket. That single change moves durability out of the broker, and the visible consequence is a configuration that looks like somebody made a mistake: every topic, including __consumer_offsets, is replication factor 1. Upstream's own server.properties ships it that way.

It is not a mistake, and the reflex to "fix" it is expensive: replicas would multiply write amplification and cost without adding durability, because the bytes are already in the object store. What three nodes actually buy is a KRaft controller quorum, partition failover, and throughput.

The honest half of that sentence matters more than the marketing half. Durability and availability are different claims. Losing a broker loses no bytes — we killed one and read every pre-failure record back afterwards. But a partition whose leader dies is unwritable until it is reassigned, and since __consumer_offsets is RF=1 too, a consumer group's committed position lives on exactly one broker. Both of those windows are real. We measure them and report the numbers rather than asserting the comfortable version.

2. Ping says the network is fine

The longest failure of the build presented like this, and the order matters because only the last line is the cause:

A Vultr node sits behind two firewalls. The provider's firewall group is the one you think about, and it does filter the private interface — but opening it changed nothing, which is what sent us looking further. The second is ufw, which the Vultr Ubuntu image ships enabled, with exactly one rule:

Status: active
Default: deny (incoming), allow (outgoing), deny (routed)
22/tcp                     ALLOW IN    Anywhere

That is the entire allowlist. It passes ICMP — hence the healthy-looking pings — and drops every inter-node TCP connection, so the controllers never exchange a vote. It would also have silently blocked the public Kafka port, which we would have discovered later and blamed on something else.

The generalisable part is a diagnostic order. ping proves layer 3 and nothing above it. The real test is a raw TCP connection to a listener you stand up yourself — we used a five-line Python listener on a peer — and when that fails, check ufw status on the host before the provider's console, because the host drops the packet after the provider's rules have already let it through.

3. A marker is not evidence

The most expensive failure produced an error that pointed nowhere near its cause. Every SASL principal stopped authenticating at once — including ones nothing had touched:

org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed
during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512

The obvious hypotheses were all wrong, and ruling them out was worth the time. The secret bundle was byte-identical on all three nodes. The salted-password derivation was correct — we proved it by formatting two scratch directories, one with an explicit saltedpassword and one letting Kafka derive it from the plaintext and the same salt, then decoding both metadata checkpoints and comparing: identical storedKey, identical serverKey.

What was actually true is that the cluster's metadata log contained no credential records at all:

kafka-dump-log.sh --cluster-metadata-decoder \
  --files /var/lib/automq/metadata/bootstrap.checkpoint | grep -c USER_SCRAM_CREDENTIAL_RECORD
# 0 → this cluster can never authenticate anyone

An earlier converge had claimed a "genesis" marker and then failed during the format that was supposed to follow it. Every later run asked "has genesis been claimed?", got yes, and formatted its nodes without bootstrap credentials. The cluster was permanently unable to hold a credential, and re-running could never fix it — the one repair path you would reach for is closed:

org.apache.kafka.common.errors.UnsupportedEndpointTypeException: This Admin API is
not yet supported when communicating directly with the controller quorum.

The rule we took out of it is short enough to apply everywhere: a marker may record that something happened; only evidence that it happened may decide what to do next. "Has this cluster been initialized?" is now answered by whether any node carries a format-complete record — an artifact written after the work, not a flag written before it. The same principle makes that per-node record two-phase, intent then complete: without the split, a converge killed mid-format is indistinguishable from a lost disk on the next run, and those two demand opposite responses.

4. Three gates, three ways to be wrong

Once the infrastructure settled, the failures kept coming — and they were ours. Three separate defects in the acceptance gates, each a different species.

The gate that passes once. It created a topic, produced 500 records, consumed 500 from the beginning, and compared. Perfect on a fresh cluster. On the second converge it read the previous run's records and failed. Gates run on every converge against a cluster that keeps its data, so anything counting absolute totals is a first-run demonstration rather than a gate — and the first run is the one you watch, which is exactly why the defect survives review.

The gate that assumes placement. The failover test killed "node 2". After an earlier failover, leadership had drifted and node 2 led none of the six partitions, so a completely healthy cluster produced no partition of colors-failover is led by node 2. The gate now reads the leadership map and derives its victim from it, which is also a better test.

The gate that cries catastrophe. This one is worth dwelling on. It reported:

FAIL — only 0 of 100 pre-failure records survived

That is the most alarming sentence this deployment can produce: it claims an object-store-backed system lost acknowledged writes. It was false. Checked directly, seconds later, the partition held all 101 records — the hundred written before the kill plus the one written during the outage. The gate had read the partition once, immediately after its leader was killed, while it was still being reassigned; the fetch failed, stderr was suppressed, grep -c counted zero, and transient unavailability was reported as permanent loss.

Which yields the rule we would most like other people to steal: a gate asserting a catastrophic outcome needs more evidence than one asserting success, not less. "Not readable yet" and "gone" are indistinguishable from a single read taken at the wrong moment, and only one of them is an emergency. The survival check now retries and keeps the best observation, and the setup step — did the produce actually succeed? — is verified rather than assumed, because an unverified produce would make the same gate claim data loss for records that were never written.

5. What a cold reader found in working code

Before any code existed, the plan went through five rounds of adversarial review by a different model. That caught two Kafka-level errors that would have failed the first converge, and corrected us on the controller listener — where our own reasoning had been wrong twice in opposite directions.

The more surprising pass was the second one. After the cluster was converged and its gates were passing, a fresh session of that model read the finished code cold and found fourteen more defects. Three could not have been caught by any amount of converging, because they only manifest on a future event:

• Rebuilding one node would have replaced the cluster's credentials. The secret bundle was create-once per host, so a rebuilt node 0 would mint fresh passwords and salts and push them over working ones — while the metadata log kept the old. Silent, and unrepairable by the route described in section 3.
• Configuration changes converged and did nothing. docker compose up -d does not recreate a container because the contents of a bind-mounted file changed; it compares its own service definition. Listener, ACL, storage and retention edits reported a clean converge while the JVM kept running the old configuration.
• Two nodes could hold the same restart lease. An expired lease was replaced unconditionally, so both contenders could read it, both write, and both believe they held it — on combined broker-and-controller nodes, that is a simultaneous restart and a lost majority.

None of these are exotic. They are the ordinary consequences of writing code that works, on the first path through it. The lesson is not "use two models"; it is that a passing converge is evidence about one execution, and quite a lot of what you want to know is about executions you have not run yet.

6. The starting point

The final converge ran all five stages green: infrastructure, ssh-config, DNS, convergence, acceptance. Three vc2-4c-8gb nodes in ams, one VPC carrying the quorum and inter-broker traffic, a public SASL_SSL endpoint on 9092 with SCRAM and an ACL authorizer, and two Cloudflare R2 buckets as the storage tier.

The automq Package Skill is the tested implementation, and the automq-vultr Context Skill is everything above routed by symptom — seventeen catalogue entries with verbatim error text, so an agent staring at we still don't know the high water mark lands on the firewall entry rather than on Kafka's own troubleshooting.

One honesty clause, as always. These claims hold at the pinned versions and are unknown after them; pins.md lists what to re-verify when they move. Four things are named there as untested rather than implied: transactional workloads, sustained throughput, timer-driven certificate renewal, and recovery from real disk loss. Neither skill is affiliated with AutoMQ — and the best thing we can say about the system is that across twenty-three converge runs, nearly every failure was ours rather than its.