The 60-Second Coincidence

A courier swipes to accept a delivery. The spinner holds. The request ends in HTTP 504.

The timing lined up with a new shared Kafka library across the API fleet, so Kafka was the first place to look. That is the right instinct — and also why isolation matters. Correlation is not cause. This post is a walkthrough of how we isolated producer vs consumer vs broker, what the 60-second timeout actually identified, and a second persistence pattern on the same hot path that is worth fixing whenever you see it.

Isolate the broker with a real produce and consume

Accepting a delivery only publishes. No consumer sits on that HTTP path. Still, before changing application code, confirm the cluster can accept a write on the topic the producer claims to use:

# is the topic available? leader elected, replicas in sync, ready to take a write
kafka-topics.sh --bootstrap-server <broker>:9092 \
  --describe --topic delivery.events

# deposit a probe message
echo 'probe-key:{"type":"AGENT_PROBE","note":"broker liveness"}' \
  | kafka-console-producer.sh --bootstrap-server <broker>:9092 \
      --topic delivery.events \
      --property parse.key=true --property key.separator=:

# read it back
kafka-console-consumer.sh --bootstrap-server <broker>:9092 \
  --topic delivery.events --from-beginning --max-messages 5

A liveness check that only answers "is the process up?" is weaker than one that answers "can this topic take a write?" Historical records on the same topic also date the last successful produce. If the envelope format already existed, the cluster and the schema are not the regression — the client path is.

Producer, consumer, or broker?

With the broker cleared, the request path is a producer question. Application logs showed the client never reached the cluster:

WARN  o.a.k.clients.NetworkClient : [Producer clientId=driver-api-producer-1]
      Connection to node -1 (localhost/127.0.0.1:9092) could not be established

ERROR TimeoutException: Topic delivery.events not present in metadata after 60000 ms

localhost:9092 is Spring Boot's local default when KAFKA_BOOTSTRAP_SERVERS is unset:

spring.kafka.bootstrap-servers=${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}

Correct for a laptop. In ECS it means the task is talking to itself.

Compare publishers, not guesses

Several services in the fleet produce to Kafka. Mapping runtime env against what the repo's task definition actually ships isolates the gap immediately:

Services that stayed healthy had the bootstrap address in the file the pipeline registers. Services that fell back to localhost did not. GitOps is binary here: whatever the task-definition JSON contains is what the next deploy installs.

Finding 1: configuration a deploy renders, a deploy owns

This fleet deploys by registering .aws/task-definition.json as a new ECS revision. A definition with no environment block will not carry Kafka bootstrap — even if a previous revision had it:

{
  "name": "driver-api",
  "image": "…/driver-api:latest",
  "essential": true,
  "logConfiguration": { "…": "…" }
}

A later, unrelated deploy then publishes a clean revision from that file. The env var is not "changed" in the console; it is replaced by the artifact. That is why "it worked last week" and "nobody touched Kafka" can both be true.

Finding 2: 60 seconds is a fingerprint, not a slow request

The client felt "about thirty seconds." The producer said after 60000 ms. The load balancer idle timeout was 60 seconds.

That alignment is Kafka's default max.block.ms (60000) meeting the ALB default idle timeout (also 60 seconds):

Two production rules follow.

KafkaTemplate.send() is not fully non-blocking. It returns a CompletableFuture, which looks async. Before that future exists, the client must resolve topic metadata. If the bootstrap host is unreachable, the calling thread waits for max.block.ms. An "async" publish can occupy an HTTP worker for a full minute.

The business write can succeed while the HTTP response fails. DynamoDB already had the assignment (BUSY, incremented current deliveries) while the app received 504. That split view — store committed, client told failure — is exactly what timeout budgets on a publish path are meant to prevent.

Layer timeouts strictly: producer block < API timeout < ALB idle. If any inner default equals the outer one, the edge will cut the request first and the client will never see the application's error.

Finding 3: keep publish failures on the future, not on the stack

The shared messaging helper wrapped send like this:

CompletableFuture<SendResult<String, String>> future = kafkaTemplate.send(record);
future.whenComplete((result, ex) -> { /* metrics + logging */ });
return future;

whenComplete covers asynchronous failures. It does not run when send() throws synchronously (no metadata, exhausted buffer). That exception leaves the helper and enters the business method that called it.

Fire-and-forget callers ignore the returned future — a reasonable pattern for a notification. It is only safe if failures arrive through the future, not by unwinding the request thread after DynamoDB has already committed.

Blast radius: direct publish vs outbox

The same missing env var does not have the same user-facing effect on every service. The difference is whether Kafka sits on the HTTP thread:

With outbox-enabled=true, the request writes the business row and an outbox row in the same store, then returns. A poller drains Kafka. If bootstrap is wrong, user-facing latency does not move — events queue and retry.

A service that publishes on the request thread couples Kafka availability to the accept UX. Same cluster, same missing env, different blast radius. The outbox is not only a durability pattern; it keeps infrastructure faults off the courier's spinner.

Persistence: putItem plus a stripped DTO

On the same accept path, a second pattern is easy to miss: DynamoDB putItem is a full-item overwrite. If a read path nulls a field so it never reaches a response body, and a later mutation saves that same entity, the null is persisted.

// repository
public Driver save(Driver driver) {
    table.putItem(driver);   // full-item overwrite
    return driver;
}

// service — strips the hash so it never reaches a response body
public Driver getDriver(String driverId) {
    Driver driver = driverRepository.findById(driverId).orElseThrow();
    driver.setPassword(null);
    return driver;
}

// these run on every accept
public void recordOfferAccepted(String driverId) {
    Driver driver = getDriver(driverId);   // password already null here
    driver.setOffersAccepted(n + 1);
    driverRepository.save(driver);          // putItem writes the null away
}

Clearing the hash before serialization is good API hygiene. putItem is idiomatic DynamoDB. Combined on one bean that is both persistence model and response DTO, accepting a delivery can drop the stored credential. Recurring "reset the demo password" work is a signal to inspect save paths, not a password-policy problem.

Guard it at the repository so every call site is covered:

/**
 * putItem is a full-item overwrite, and several read paths strip the password hash
 * before handing the entity out. Re-hydrate it instead of writing an item without one.
 */
public Driver save(Driver driver) {
    if (driver.getPassword() == null && driver.getDriverId() != null) {
        findById(driver.getDriverId()).map(Driver::getPassword).ifPresent(driver::setPassword);
    }
    table.putItem(driver);
    return driver;
}

updateItem with ignoreNulls(true) is the more idiomatic DynamoDB shape. It also changes global semantics: an intentional null-to-clear stops working. Prefer the change whose blast radius you can name — here, re-hydrate on save.

Resolutions

Four layered changes. No single one is the whole design.

1. Ship bootstrap in the task definition so a deploy restores it instead of dropping it. Apply it to every publisher that reads KAFKA_BOOTSTRAP_SERVERS, not only the service on the hot path.

"environment": [
  { "name": "KAFKA_BOOTSTRAP_SERVERS", "value": "<broker>:9092" },
  { "name": "AWS_REGION", "value": "us-east-1" }
]

2. Bound max.block.ms under the ALB idle timeout so a publish cannot outlive the request it is attached to.

# Metadata lookup blocks the calling thread for up to max.block.ms.
# Keep this well under the ALB idle timeout.
spring.kafka.producer.properties.max.block.ms=5000

3. Surface synchronous send failures on the future so a broker fault degrades event delivery without failing a committed assignment.

CompletableFuture<SendResult<String, String>> future;
try {
    future = kafkaTemplate.send(record);
} catch (RuntimeException e) {
    meterRegistry.counter("kafka.produce.failure", "topic", topic, "reason", "send").increment();
    log.error("Failed to publish Kafka event topic={} eventType={}", topic, eventType, e);
    return CompletableFuture.failedFuture(e);
}

4. Re-hydrate credentials on putItem, as above.

If the broker is down after this, accept returns in milliseconds, the delivery is recorded, a failure counter increments, and operators have a metric. The failure is bounded, visible, and non-destructive. It is not silent.

Pre-fix and post-fix probe

A config change is not verified by reading the diff. Re-run the same class of request and watch both ends: edge timing, and the broker's own ack.

Attach a consumer before the request so the event is captured as it lands:

{
  "eventId": "…",
  "eventType": "DELIVERY_STATUS_UPDATED",
  "schemaVersion": "1",
  "payload": { "orderId": "…", "driverId": "…", "status": "…" }
}

Two verification details worth keeping. Success logs should include partition and offset — that is the difference between "we called send()" and "the broker accepted the record." And confirm the live task definition's environment came from the rendered file, not a one-off console edit, or the next deploy will drop it again.

Takeaways

Isolate with a real produce/consume first. Confirm topic, ISR, and a round-trip write before treating the broker as the fault.

Round timeouts are fingerprints. 60s here was max.block.ms sitting on the ALB idle timeout. Every blocking client default should sit strictly inside the layer above it.

A method that returns a future can still block before it returns one. Verify send() under a missing bootstrap host.

An event publish should not fail a committed transaction. Bound it, or take it off the request path with an outbox. Both is better.

Do not use the same bean as persistence entity and response DTO with putItem. A read path that nulls a field will eventually persist that null.

A fix you have to re-apply is incomplete. If a credential has to be re-seeded after normal traffic, inspect the save path.

One follow-up stays intentional: this service still publishes directly rather than through the outbox, so a broker outage now drops the event quickly instead of hanging the request. Enabling the outbox would make delivery events durable like orders and payments, at a few seconds of poller lag. That is a behaviour change on a real-time path and deserves its own verification window.

For retries, dead-letter topics, idempotency, and the outbox itself, see Production Kafka for Order and Payment Events.