Schema Registry: contracts that survive over time 🗄️
The event design guide tells you how to think about contracts. This one tells you which tools to use to enforce them before a renamed field knocks out three consumers at 2 a.m.
The problem it solves (and which remains unsolved without a registry) 🧠
Without a schema registry:
- each team manages event versions with Word docs, wikis, or “we know it”;
- consumers discover breaking changes in production;
- there is no single place to see all schema versions;
- compatibility becomes a personal responsibility, which in practice means “nobody’s responsibility”.
A schema registry is a centralized service that:
- collects event schemas (Avro, Protobuf, JSON Schema);
- validates messages before they reach the broker;
- checks compatibility between schema versions;
- exposes APIs for producers and consumers.
How it works in practice ⚙️
flowchart LR
P[Producer] -->|validate schema| SR[(Schema Registry)]
SR -->|schema ID| P
P -->|message with schema ID| B[(Broker / Kafka)]
B --> C[Consumer]
C -->|retrieve schema by ID| SR
C -->|deserialize payload| C
The producer does not include the full schema in the message: it includes only a schema ID (a few bytes). The consumer retrieves the schema from the registry via the ID and deserializes the payload. Result: smaller payloads and a centralized source of truth.
Compatibility modes 🔄
Compatibility is configured per subject (typically per topic) or globally. The main modes are:
| Mode | Description | Typical use |
|---|---|---|
BACKWARD |
New consumers can read messages produced with the previous version | Most common, ordinary evolution |
FORWARD |
Old consumers can read messages produced with the new version | When you cannot update all consumers at once |
FULL |
Compatibility in both directions | External/public events, stable contracts |
NONE |
No compatibility checks | Only for local development |
Practical rule: start with BACKWARD, then move to FULL for external events. Do not use NONE in staging or production.
Supported schema formats 🧱
Apache Avro
Binary schema, compact, with native support for nullable fields and evolution. It is the most common format with Kafka.
|
|
Adding the channel field as nullable with a default null is a backward-compatible change. Removing amount would be a breaking change.
Protocol Buffers (Protobuf)
Binary schema, cross-language, very efficient. Preferred in polyglot environments or where payload compactness matters. Requires compilation of stubs (.proto → code in target languages).
JSON Schema
Readable and familiar, but with less automatic evolution support than Avro. Useful if the team does not want Avro/Protobuf tooling or mostly works with REST + JSON.
The most common registries 🗂️
Confluent Schema Registry
The de facto standard for Kafka. Distributed under the Community license (open source for self-hosted use), with enterprise tiers.
- native integration with Kafka producers/consumers via serializers (Avro, Protobuf, JSON Schema);
- REST APIs to manage schemas, versions, and compatibility settings;
- included in Confluent Platform and Confluent Cloud;
- supports all three formats.
Apicurio Registry
Open-source registry (Apache 2.0) developed by Red Hat. Supports Avro, Protobuf, JSON Schema, and AsyncAPI, making it a good fit if you want to integrate catalog and registry in one tool.
- compatible with the Confluent Schema Registry API (drop-in replacement for migrations);
- available standalone or integrated with Red Hat OpenShift / AMQ Streams;
- supports artifact lifecycle states:
ENABLED,DEPRECATED,DISABLED.
AWS Glue Schema Registry
Managed registry on AWS, natively integrated with MSK (Managed Kafka), Kinesis Data Streams, and AWS Lambda.
- zero infrastructure to manage;
- integration via AWS SDK;
- supports Avro and Protobuf;
- IAM for native access control.
Ideal if you are already on AWS and want to avoid another service to maintain.
Schema evolution in practice: what you can do and what you cannot 🧬
With Avro + Confluent Registry in BACKWARD mode:
| Operation | Compatible? |
|---|---|
| Add an optional field (nullable with default) | ✅ Yes |
| Add a required field (without default) | ❌ No |
| Remove a field with default | ✅ Yes (BACKWARD) |
| Remove a required field | ❌ No |
| Rename a field | ❌ No (equivalent to remove + add) |
| Change a field type | ❌ No (except safe promotions such as int → long) |
When you need an inevitable breaking change, create an explicit v2 version and maintain an overlap period where both versions coexist.
CI/CD integration 🔁
Having a registry in production is not enough: it must be part of the deployment pipeline.
flowchart LR
DEV[Development] -->|defines schema| SR_DEV[Registry DEV]
CI[CI Pipeline] -->|check compatibility — fail fast| SR_STG[Registry STG]
DEPLOY[Deploy] -->|promotes schema| SR_PROD[Registry PROD]
Recommended practices:
- Schema-first: define the schema before implementing producers and consumers—not after;
- Gate in CI: compatibility checks fail the pipeline, they are not just warnings;
- Promotion per environment: DEV → STG → PROD, do not register directly in production.
Schema Registry adoption checklist ✅
- Have you chosen a schema format (Avro / Protobuf / JSON Schema) consistently for all events?
- Have you configured the compatibility mode for each subject/topic?
- Is compatibility checking part of the CI/CD process?
- Does the registry have defined SLOs? (Is it a critical dependency for producers and consumers?)
- Do you have a migration plan for legacy events not yet in the registry?
- Do teams know how to update a schema, promote a version, and manage a breaking change?
Next steps 🚀
- To design the event schema from the domain perspective: see the event design guide.
- To document and catalog events with concrete tools: see the tool and ecosystem guide.
- For schema testing in CI: see the testing guide.