Event-driven architecture is the right call for decoupled, scalable systems. It is also where debugging goes to die — unless you design for understandability from the start.
Name events with intent, not CRUD
OrderShipped tells every consumer what happened. OrderChanged tells them nothing — they have to diff the payload to figure out what matters. Name events after business intent. The payload carries only what downstream consumers need to react. If a consumer needs more, include a reference ID to fetch it.
The test: the producer should not need to know who consumes the event, and the consumer should not need to call back to the producer. If either condition fails, redraw the boundary.
Schemas are contracts, not suggestions
Events cross team boundaries. Without schema governance, a single breaking change cascades into runtime errors or silent data corruption.
- Schema registry as a first-class citizen. Enforce backward compatibility checks at produce time, not at runtime.
- Additive-only evolution. Add optional fields. Never rename or remove. Include a
schemaVersionfield, but resist creating new event types for every change — that leads to topic sprawl. - AsyncAPI + EventCatalog. Turn your event specs into browsable documentation with ownership, versioning, and relationship metadata. If you would not deploy a REST API without OpenAPI docs, do not publish events without a catalogue.
Make every event traceable
Traditional request tracing breaks when a user action triggers a cascade of events processed by different services over hours.
- Correlation IDs are non-negotiable. Every event carries a
correlationIdthat links every log line from every service that touched a transaction. - OpenTelemetry context propagation. Inject W3C Trace Context headers into Kafka message headers. For loosely coupled systems, use span links instead of parent-child relationships.
- Structured logging. Every log line includes
eventType,eventId,correlationId,source, andtimestamp. This turns grep from useless to powerful.
The five metrics that matter: consumer lag, throughput, consumer error rate, DLQ size, and end-to-end latency. Build dashboards around these. Budget for observability from day one or pay for it in incident response time.
Plan for failure with dead letter queues
Failed messages are inevitable. Without a DLQ strategy, poison pills block entire consumer groups.
Classify errors before retrying. Transient errors (network timeouts) are worth retrying with exponential backoff. Non-transient errors (deserialization failures, schema mismatches) will never succeed — route them to the DLQ immediately. Capture rich metadata: original topic, partition, offset, error, stack trace, and retry count.
A DLQ is not a trash can — it is an ICU for messages. If you are not monitoring, alerting on, and regularly draining your DLQ, you are silently losing data.
The anti-pattern checklist
Pin this to your team wiki:
- Fat events that dump entire entities → intent-focused payloads
- No schema registry → enforce compatibility at produce time
- DB write + event publish without atomicity → Outbox Pattern
- Immediate retries → exponential backoff with jitter
- No correlation ID → mandatory in every event
- Shared “God event” across contexts → bounded-context-scoped events
- Ignoring the DLQ → monitor, alert, build replay tooling
The bottom line
Event-driven systems do not become unmanageable because of events. They become unmanageable because of missing contracts, invisible flows, and unplanned failure modes. Name events with intent, enforce schemas as contracts, trace everything, and treat your DLQ as critical infrastructure. The architecture stays understandable because you designed it to be.