Skip to content

    Architecture

    How Confluent Kafka Works: Architecture, Delivery Guarantees and Publishing from Spring Boot on AWS

    How Confluent Kafka works from producers and partitions to consumer groups and delivery guarantees, plus onboarding Confluent Cloud on AWS with Spring Boot.

    By Published 33 min read

    Share
    How Confluent Kafka Works: Architecture, Delivery Guarantees and Publishing from Spring Boot on AWS

    Sign up for a free account to read this article in full.

    How Confluent Kafka works: infographic covering the big idea, key terms, end-to-end architecture, the produce and consume paths, the cluster internals, reliability, retention, the Confluent ecosystem, security, an e-commerce order flow, myths versus reality and key takeaways

    Download the full-size infographic

    What you'll learn: how a record travels through Confluent Kafka, from the producer to the partition log to a consumer group; what actually protects your data (acks, replication, min.insync.replicas, idempotence); where exactly-once stops being true; which Confluent services belong to Confluent Cloud and which to self-managed Confluent Platform; and how to onboard Confluent Cloud on AWS and publish a first message from Spring Boot.

    This guide turns two infographics into a full walkthrough: how Confluent Kafka works end to end, and how to onboard Confluent Cloud on AWS and publish a message. Everything in the pictures is also in the text and tables below, so it is searchable and screen-reader friendly. Where an infographic was loose, the article follows the official documentation. For the tuning side, read Kafka Scalability Patterns afterwards.

    1. The big idea

    Kafka is a distributed, durable event log. Producers append events to named topics, Kafka stores them, and any number of consumers read them independently. Per the Confluent documentation, topics are append-only and events are immutable once written. Three properties explain most of the design:

    • Events are stored in topics (append-only logs). Nothing is updated in place.
    • Consumers read independently, at their own pace. The only per-consumer state is the offset in each partition.
    • Events are not removed because one consumer read them. Unlike a classic queue, a read deletes nothing. Data leaves by retention (age or size) or compaction.

    Confluent is the company around Apache Kafka. Confluent Cloud is the fully managed service, built on what Confluent calls the Kora engine. Confluent Platform is software you download and run yourself. Both speak the Apache Kafka protocol, so a standard Kafka client works against either.

    2. Why it matters and the key terms

    Why Kafka matters What it means in practice
    Real-time, scalable data pipeline Events flow as they happen, and you scale by adding partitions and consumers
    Decouples producers and consumers The order service does not know who reads its events
    Multiple use cases from the same data Inventory, payments, analytics and search can all read the same topic
    Durable, fault-tolerant, scalable Data is replicated across brokers and survives broker failures

    Quick recap of the key terms:

    Term Meaning
    Topic A named stream of events
    Partition An ordered, append-only log within a topic
    Offset The position of a record in a partition
    Broker A Kafka server that stores data and serves clients
    Consumer group A group of consumers that share a topic's partitions
    ISR In-sync replicas: the replicas that are caught up with the leader

    The producer writes events and the consumer reads them. For background, see event streaming and CQRS and event sourcing.

    3. End-to-end architecture

    Event sources (business services, databases through change data capture, IoT devices, logs and files) feed Kafka through producers or Kafka Connect source connectors. Brokers hold the topics. Consumer groups and stream-processing jobs read them, and downstream systems (databases, search, analytics, notifications) receive the results.

    Figure 1. End-to-end flow from event sources to downstream systems.

    End-to-end Confluent Kafka architecture: sources, producers, Kafka cluster, consumer groups, stream processing and downstream systemsEvent sources to Producers; Producers to Kafka cluster (produce); Kafka Connect to Kafka cluster (CDC, files); Kafka cluster to Consumers (poll); Kafka cluster to Stream processing (poll); Consumers to Downstream systems; Stream processing to Downstream systemsCONFLUENT KAFKA (EVENT STREAMING LAYER)Event sources: services, IoT, logsEvent sourcesservices, IoT,logsKafka Connect: source connectorsKafka Connectsource connectorsProducers: send eventsProducerssend eventsKafka cluster: brokers, topicsKafka clusterbrokers, topicsConsumers: consumer groupsConsumersconsumer groupsStream processing: Streams, FlinkStreamprocessingStreams, FlinkDownstream systems: DBs, search, BIDownstreamsystemsDBs, search, BIproduceCDC, filespollpoll

    How to read this: packets travel along request and response arrows (producers writing, consumers polling). Dashes show event flow into downstream systems.

    The infographic's example cluster has three brokers and a topic orders with three partitions and a replication factor of two. That is an illustration: Confluent's introduction notes that a common production setting is a replication factor of 3.

    Around the core sits an optional layer of Confluent services (Schema Registry, a monitoring UI, ksqlDB, Cluster Linking, a REST interface). Section 9 says which are Confluent Cloud features and which are Confluent Platform components.

    4. The produce path: from app to topic

    Step What happens
    1. Create event The app builds a record: key, value, timestamp and optional headers
    2. Serialize The producer converts key and value to bytes (Avro, Protobuf, JSON or plain strings)
    3. Schema (optional) With Schema Registry, the serializer registers or looks up the schema and checks compatibility
    4. Get metadata The producer asks a broker which brokers lead which partitions
    5. Choose partition The partitioner picks a partition. With the default partitioner, records with the same key go to the same partition, as long as the topic keeps the same number of partitions
    6. Batch and compress Records are buffered into batches, optionally compressed (gzip, snappy, lz4 or zstd)
    7. Send to leader The batch goes to the leader broker of the target partition
    8. Append and replicate The leader appends to the partition log, assigns offsets, and followers replicate it
    9. Acknowledge The producer gets an ack according to acks

    Figure 2. Sequence of one produced record with acks=all.

    sequenceDiagram accTitle: Produce path with acks=all accDescr: The application hands a record to the producer, which serializes it, fetches metadata, batches it and sends it to the partition leader. The leader appends it and a follower replicates it. Once the in-sync replicas have it, the leader acknowledges the producer. participant App as Application participant P as Producer client participant L as Leader broker participant F as Follower broker App->>P: send(topic, key, value) P->>P: serialize, pick partition by key P->>L: metadata request (via bootstrap broker) L-->>P: partition leaders P->>P: batch and compress P->>L: produce batch L->>L: append to partition log F->>L: fetch new records L-->>F: records L-->>P: ack (acks=all, after ISR has the write) P-->>App: send completed

    Two details are easy to miss. First, the producer fetches metadata from brokers (any broker in the bootstrap list can answer), not by calling a special controller endpoint. Second, the partitioner only guarantees that equal keys land together. Records without a key are spread across partitions and have no cross-partition order.

    The acknowledgment levels, from the Apache Kafka producer documentation:

    Setting Meaning Trade-off
    acks=0 Do not wait for any acknowledgment. Retries do not take effect Fastest, can lose data silently
    acks=1 Wait for the leader only Data can be lost if the leader fails before followers copy it
    acks=all Wait for the leader and the in-sync replicas Strongest guarantee, slightly higher latency

    In the Apache Kafka 4.1 documentation the default is acks=all and enable.idempotence=true. An idempotent producer gets a producer ID from the broker, and the broker deduplicates retried batches by sequence number, so a retry inside the same producer session does not write the record twice. It does not stop your own application from sending the same event again, or a restarted producer from repeating work. Idempotence needs acks=all, retries above zero and max.in.flight.requests.per.connection of 5 or less. Check the defaults of the client version you actually use, because older clients differ.

    acks=all plus a suitable min.insync.replicas improves durability, but does not replace monitoring and capacity planning.

    5. Inside the cluster: partitions, leaders, followers, ISR and KRaft

    A topic is split into one or more partitions. Each partition is an ordered, append-only log, and every record in it has an offset. Order is guaranteed within a partition, not across the whole topic. (With idempotence off and more than one in-flight request per connection, a retry can reorder records within a partition.)

    Replication. The unit of replication is the topic partition. Each partition has one leader and zero or more followers (a replication factor of 1 has none). Writes go to the leader and followers pull from it.

    In-sync replicas. A replica drops out of the ISR if it loses its session or falls too far behind (replica.lag.time.max.ms). Only ISR members can become leader, unless unclean leader election is enabled. It is false by default because it can lose data.

    When a write counts as committed. A write is not committed until all in-sync replicas have it, and consumers cannot read it before that. With min.insync.replicas (default 1 at broker level) you set how many replicas, including the leader, must be in sync for a write to succeed.

    Replication factor min.insync.replicas acks=all write succeeds while Broker failures tolerated for writes
    3 2 at least 2 replicas are in sync 1
    3 1 the leader is alive 2, but data can be lost if the leader then fails
    2 2 both replicas are in sync 0 (writes stop if either broker fails)

    Control plane: KRaft. Modern Kafka manages cluster metadata with a built-in Raft quorum, so no ZooKeeper. A server's process.roles can be broker, controller or both; the Kafka docs recommend 3 or 5 controllers (a majority must be alive) and advise against combined mode in critical deployments. The last bridge release for migrating from ZooKeeper is Kafka 3.9. In Confluent Cloud you do not run controllers, so KRaft matters when you self-manage. The data plane is the brokers: in the infographic, each of the three orders partitions is led by a different broker with a follower elsewhere.

    6. The consume path: groups, offsets, rebalancing and replay

    Step What happens
    1. Subscribe The consumer subscribes to topics with a group.id
    2. Group management The group coordinator (a broker) manages membership as consumers join and leave. Partition assignment is computed by the group leader (one member) with the classic protocol, and by the coordinator with the newer consumer protocol
    3. Partition rule Within a group, a partition is assigned to at most one consumer at a time
    4. Pull and read The consumer polls the partition leaders from its current offsets
    5. Process Your code handles the records: update a database, call a service
    6. Commit offset Offsets are committed automatically or manually to record progress
    7. Replay (optional) A consumer can reread by resetting offsets, as long as the data is still retained

    Figure 3. Joining a group, reading and committing.

    sequenceDiagram accTitle: Consumer group join, poll and offset commit accDescr: A consumer subscribes with a group id. The group coordinator assigns partitions. The consumer polls the partition leaders, processes records, and commits its offset to the internal offsets topic. If a member leaves, the group rebalances and the new owner resumes from the last committed offset. participant C as Consumer participant G as Group coordinator participant L as Partition leader participant O as Offsets topic C->>G: join group (group.id) G-->>C: assigned partitions C->>L: poll from committed offset L-->>C: records C->>C: process records C->>O: commit offset Note over G,C: a member joins or leaves, so partitions are reassigned G-->>C: new assignment C->>O: read last committed offset

    Some facts worth keeping straight:

    • Each partition is consumed by exactly one consumer within each consumer group at any given time. More consumers than partitions means the extras sit idle. Partition count is therefore the ceiling on parallelism for one group.
    • Different groups are independent. Each gets its own logical read of the whole stream.
    • The committed offset is the position of the next record the consumer intends to read, stored in the internal __consumer_offsets topic. If a partition moves to another member, it resumes from the last committed offset, so records after it may be processed again.
    • Rebalancing is triggered by membership or metadata changes. With the classic protocol, pauses can affect every consumer. Kafka 4.0 added a consumer protocol (group.protocol=consumer) that reassigns incrementally, but Confluent's pages differ on its availability, so check your environment. group.instance.id makes a consumer a static member, avoiding rebalances on brief restarts.
    • Defaults in the 4.1 documentation: enable.auto.commit is true, auto.offset.reset is latest, max.poll.interval.ms is 300000.

    Replay lets a new service start from the earliest offset, or a fixed bug be reprocessed by resetting offsets. Data that retention has deleted cannot be replayed.

    7. Reliability and delivery semantics

    Area What Kafka gives you What you still must do
    Replication and failover Followers replicate from the leader. If the leader fails, a follower in the ISR becomes leader Choose a replication factor of 3 for important topics, and set min.insync.replicas deliberately
    Producer reliability Retries with idempotence avoid duplicate writes Use acks=all, and handle send failures in code
    Delivery semantics At-least-once by default Make consumers idempotent, or use transactions where they apply
    Operations Monitoring surfaces problems Configure timeouts and retries, a dead-letter topic for poison records, and alert on consumer lag

    Delivery semantics, precisely. The Apache Kafka design documentation defines three levels:

    • At most once: messages may be lost but are never redelivered. You get this by disabling producer retries and committing offsets before processing.
    • At least once: messages are never lost but may be redelivered. The documentation states Kafka guarantees this by default. Duplicates appear when a consumer processes a record and then fails before committing.
    • Exactly once: each message is processed once and only once. Kafka supports this within Kafka itself: Kafka Streams, and transactional producers combined with read_committed consumers for Kafka-to-Kafka pipelines. A consume-transform-produce application also turns off auto commit and commits its consumer offsets as part of the producer transaction.

    The real limit of exactly-once. It has scope: the guarantee covers reads, processing and writes that stay inside Kafka. When your code writes to PostgreSQL, calls a payment API or sends an email, Kafka cannot undo or deduplicate that side effect. External systems must cooperate: use idempotent writes (a unique key or upsert) or store the offset with the result in one transaction. Treat exactly-once as effectively once, inside Kafka.

    Poison records. One malformed event can block a partition if the consumer retries it forever. Route it to a dead-letter topic after bounded retries. Kafka Scalability Patterns shows that flow.

    8. Data lifecycle: retention, compaction and tiered storage

    Mechanism What it does Use it for
    Retention by time or size Deletes old segments after a time or size limit Operational events, replay windows
    Log compaction Keeps at least the latest value for each key State: user profile, product catalogue
    Tiered or infinite storage Moves older data to cheaper object storage Long retention without huge broker disks
    Replay Consumers reset offsets and reread retained data Reprocessing, new consumers, bug fixes

    Retention. Broker defaults: log.retention.hours is 168 (7 days), log.retention.bytes is -1 (no size limit), cleanup policy delete. The infographic's "7 days" and "100 GB" illustrate the two kinds of limit.

    Compaction. With cleanup.policy=compact Kafka always retains at least the last known value for each key. Offsets never change and records are never reordered. A key with a null value is a tombstone (a delete marker), itself removed after a retention period. Older values remain readable until compaction runs, so it is a storage policy, not an instant "latest only" view.

    Tiered storage. Apache Kafka defines a tiered storage feature but ships no remote-storage implementation of its own. Confluent Platform lists Tiered Storage as a feature that automatically moves data to cost-effective object storage (the infographic mentions S3 and GCS as examples). In Confluent Cloud the documented way to keep data indefinitely is Infinite Storage: setting retention.ms to -1. Use the term for your product.

    9. The Confluent ecosystem

    The infographic marks these as optional, value-add services. The important detail is where each one lives.

    Service What it does Confluent Cloud Confluent Platform (self-managed)
    Schema Registry Stores and validates Avro, Protobuf and JSON Schema schemas, and enforces compatibility Yes (part of the Cloud overview, with Stream Governance) Yes
    Kafka Connect Ready-made connectors between Kafka and external systems (CDC, files, databases, cloud storage) Yes, as fully managed connectors Yes, you run the workers
    ksqlDB SQL over streams (the infographic adds "where supported") Available, but fully managed ksqlDB is not available on Enterprise clusters. Confluent describes Flink as the recommended stream processing engine on Confluent Cloud Yes
    Kafka Streams A Java library for stateful stream processing inside your own application It is a client library, so it runs in your app against either Same
    Flink Stateful, low-latency stream processing Confluent Cloud for Apache Flink, a fully managed service Confluent Platform for Apache Flink
    Control Center Web UI for monitoring and managing Kafka Not listed as a Cloud feature. Cloud has the Confluent Cloud Console, and self-managed Control Center can connect to Cloud Yes
    Cluster Linking Mirrors topics from one cluster to another, across regions or environments Yes Yes
    REST Proxy HTTP service for interacting with a Kafka cluster Cloud documents a Kafka REST API for producing and administering instead Yes, as REST Proxy
    Tiered Storage Moves older data to object storage Cloud uses Infinite Storage Yes

    Control Center and REST Proxy are Confluent Platform components. In Confluent Cloud the matching tools are the Console and the Kafka REST API. Schema Registry runs next to Kafka, not inside the broker.

    Stream processing choices. Kafka Streams is a library embedded in a Java service, ksqlDB expresses a stream as SQL, and Flink is a separate engine. Pick based on who owns the job and their language. For LLM workloads, see event-driven AI with Kafka Streams.

    10. Security and operations

    Area What to do
    Encryption in transit Use TLS. Confluent Cloud clients must support TLS 1.2
    Authentication Confluent Cloud clients use SASL with an API key (PLAIN) or OAUTHBEARER. Confluent Cloud also supports mutual TLS (mTLS) with your own certificate authority. Self-managed Kafka supports SASL variants and mTLS too
    Authorization Role-based access control (RBAC) and/or ACLs. Confluent documents RBAC as preferred and says ACLs work together with role bindings
    Quotas Limit producer and consumer usage (bandwidth, request rates) so one client cannot starve others
    Schema compatibility Define and enforce evolution rules (backward, forward) in Schema Registry
    Capacity planning Choose partition count, replication factor and disk for the workload
    Key monitoring Under-replicated partitions, ISR changes, request latency, throughput, consumer lag, broker disk and network
    Backup and disaster recovery Plan and test recovery: cluster linking or replication, multi-region design, snapshots of configuration

    The infographic lists "SASL or mTLS" generically. For Confluent Cloud the client documentation names SASL PLAIN (API key) and OAUTHBEARER. Confluent Cloud also supports mTLS, where you upload your own certificate authority (see the Confluent Cloud mTLS documentation in the references).

    11. A worked example: e-commerce order flow (illustrative)

    This is an educational reference design, not a real company's system. An Order Service publishes an OrderCreated event keyed by orderId to the orders topic, which has three partitions. Three consumer groups read it independently:

    • Inventory group: reserves stock.
    • Payment group: processes the payment.
    • Shipping group: creates the shipment.

    Each group publishes the result to an order-status topic, and an analytics sink reads that topic for reporting.

    Figure 4. One event, three independent consumer groups.

    Illustrative e-commerce order flow: Order Service publishes OrderCreated to the orders topic, three consumer groups react, and results go to an order-status topic and an analytics sinkOrder Service to orders topic (event); orders topic to Inventory group; orders topic to Payment group; orders topic to Shipping group; Inventory group to order-status topic; Payment group to order-status topic; Shipping group to order-status topic; order-status topic to Analytics sinkOrder Service: producerOrder Serviceproducerorders topic: 3 partitionsorders topic3 partitionsInventory group: reserve stockInventorygroupreserve stockPayment group: process paymentPayment groupprocess paymentShipping group: create shipmentShipping groupcreate shipmentorder-status topic: updated eventsorder-statustopicupdated eventsAnalytics sink: reporting, BIAnalytics sinkreporting, BIevent

    How to read this: dashes show events flowing. Each group reads the whole orders topic independently.

    Why it works: keying by orderId keeps one order's events on one partition, so each group sees them in order. A new group (fraud checks, say) can be added without touching the Order Service, and if shipping is slow only its lag grows.

    12. Myths versus reality

    Myth Reality
    Kafka is a queue that deletes on read Kafka is an append-only log. Events stay until retention or compaction removes them
    Ordering is global across a topic Ordering is per partition. Use a key to keep related events together
    More partitions is always better More partitions allow more parallelism, but add overhead and metadata. Right-size them
    Replication is a backup Replication gives availability and fault tolerance. It does not protect against deleted data or bad writes, and it is not a backup
    Exactly-once is always guaranteed Exactly-once has scope. It holds within Kafka (transactions, Streams), not for external side effects
    Schema Registry is part of Kafka core Schema Registry is a Confluent service that runs alongside Kafka brokers

    13. Onboarding Confluent Cloud on AWS: prerequisites and flow

    The second infographic covers the practical side: stand up Confluent Cloud on AWS, connect an application privately and publish a message.

    Confluent Kafka SaaS on AWS: infographic covering prerequisites, the onboarding flow, the PrivateLink target architecture, the seven steps to publish a message, a Spring Boot producer, the Kafka CLI alternative, verification, security, a production checklist and common pitfalls

    Download the full-size infographic

    Prerequisites

    • An AWS account, with a VPC and the permissions to create networking resources.
    • A Confluent Cloud account (sign up at confluent.cloud).
    • A network choice. PrivateLink is the infographic's recommendation, with VPC peering as the alternative. Cluster type decides which are available (see below).
    • Client tools: a Java application (Spring Boot) or the Kafka CLI.
    • A basic understanding of topics, API keys and ACLs.

    Which cluster types support which network option. The infographic leaves this out. Per the Confluent documentation:

    Cluster type Intended use (per Confluent) Private networking on AWS
    Basic Development, testing and basic use cases Not described. Public networking
    Standard Production workloads that use public networking Public networking
    Enterprise Production workloads that require private networking PrivateLink (and Private Network Interface on AWS)
    Dedicated High-throughput or private-networking workloads with dedicated capacity PrivateLink, VPC peering and Transit Gateway
    Freight Available in select AWS regions Private Network Interface (documented for Freight and Enterprise)

    So "PrivateLink or VPC peering" is only a free choice on Dedicated clusters. Enterprise clusters offer PrivateLink but not peering, and Basic and Standard clusters use public endpoints. Options change, so check the cluster types and networking overview pages before choosing.

    Overall flow: create the Confluent organization, set up AWS networking, provision the Kafka cluster, configure security (API key and ACL or role), create the topic, publish a message, then verify and monitor.

    Figure 5. Applications in your VPC reach a Confluent Cloud cluster over AWS PrivateLink.

    Confluent Cloud on AWS with PrivateLink: producer and consumer apps in your VPC reach the Kafka cluster in a Confluent environment through a VPC endpointProducer app to VPC endpoint (SASL_SSL); Consumer app to VPC endpoint (SASL_SSL); VPC endpoint to Kafka cluster (PrivateLink); Private DNS to VPC endpoint (resolves)YOUR AWS ACCOUNT: VPCCONFLUENT CLOUD: ENVIRONMENTProducer app: Spring BootProducer appSpring BootConsumer app: Spring BootConsumer appSpring BootPrivate DNS: Route 53 zonePrivate DNSRoute 53 zoneVPC endpoint: PrivateLinkVPC endpointPrivateLinkKafka cluster: managed by ConfluentKafka clustermanaged byConfluentSASL_SSLSASL_SSLPrivateLinkresolves

    How to read this: client traffic stays on the AWS network from your VPC to Confluent Cloud. Private DNS resolves the cluster's hostname to the endpoint.

    What the Confluent PrivateLink documentation requires (this is the Dedicated cluster guide, and Enterprise clusters use a separate guide):

    • A Confluent Cloud network of type PrivateLink for AWS, then a PrivateLink Access that registers your AWS account ID. Wait for its status to move from Pending to Ready.
    • DNS resolution and DNS hostnames enabled on your VPC.
    • A security group allowing inbound TCP on 443 and 9092 (port 80 is optional) from your VPC CIDR. The infographic's "allow egress to VPCe" wording points the wrong way: the rules in the documentation are inbound rules on the endpoint's security group.
    • An interface VPC endpoint built from the Confluent VPC endpoint service name, with a subnet in each Confluent zone, and Enable DNS name unchecked.
    • A private hosted zone in Route 53 for the Confluent DNS domain, associated with your VPC, with a wildcard CNAME for the bootstrap host and one zonal record per availability zone. Broker names in the metadata are not static, so do not hard-code them.
    • Limits: all AWS availability zones are supported except use1-az3 in us-east-1. Cross-region PrivateLink connections are not supported. Missing a zone can cause connectivity problems (check the Confluent PrivateLink guide for the current wording). The DNS resolution type is fixed when the network is created.

    The endpoint cannot be created until the Confluent network and PrivateLink Access exist, so the AWS-side work follows the Confluent-side work.

    15. The seven onboarding steps

    The infographic shows seven stages in its banner and nine in its detailed list. The detailed list adds "Plan and prepare" (the prerequisites above) and splits "Integrate application" from "Publish message". Here they are as one sequence.

    Step Action Detail
    1. Organization and environment Sign up, create an organization and an environment (for example prod) Environments group clusters and keep access separate
    2. Networking Create the Confluent network and connect it: PrivateLink (recommended), or VPC peering where the cluster type supports it See section 14. The network steps follow the Dedicated-cluster guide; Enterprise clusters use a separate PrivateLink guide
    3. Cluster Create the Kafka cluster with a type, cloud (AWS), region and, where offered, availability Provisioning is asynchronous: wait until the cluster shows as ready. Match the type to the networking choice
    4. Service account and API key Create a service account for the application, grant least privilege, create an API key and secret The secret is shown once, so store it in a secrets manager
    5. Topic Create the topic in the Console or CLI Auto-creation of topics is disabled by default in Confluent Cloud
    6. Publish Configure the client (bootstrap server, SASL_SSL, API key) and send a message Spring Boot or CLI
    7. Verify and operate Consume the message and check it in the Console Then monitor

    Step 4 in detail: least privilege. The infographic suggests granting roles such as CloudClusterAdmin, DeveloperWrite (produce) and DeveloperRead (consume). DeveloperWrite and DeveloperRead are real predefined roles, but CloudClusterAdmin is not least privilege: it is a broad administrative role for a cluster (check the predefined roles page for its exact permissions). An application that only produces needs nothing like that.

    Task Roles in the Confluent documentation
    Produce to orders DeveloperWrite on the topic
    Consume from orders DeveloperRead on the topic, plus read permission on the consumer group (the docs say a consumer group role is needed to subscribe)
    See topics and messages in the Console At least DeveloperRead
    Create or delete topics from an app DeveloperManage, which cannot change their configuration

    Confluent documents RBAC as preferred over ACLs, and says ACLs work in conjunction with role bindings. Permissions belong to the service account, not to the key. Use one service account per application, avoid user-account keys in production, and rotate keys, which stay valid until rotated or deleted. Find the exact role binding or ACL command for your CLI version in the predefined roles and ACL pages.

    Step 5 in detail. The infographic creates orders with 3 partitions and replication 3: three partitions let up to three consumers in one group work in parallel, and 3 is the common production replication factor. Whether the Console form exposes replication depends on the cluster type, so check the topic settings after creation.

    16. Publish a message from Spring Boot

    This section builds a small order service end to end: a REST endpoint that triggers a producer, a consumer that reads the same topic, JSON payloads, and a dead letter topic for records that cannot be processed. Every code block below is copied from the project examples/spring-kafka-orders in the repository behind this site, and the file path is shown above each block. Shell commands are shown as you would type them; they are not files.

    NOTE

    What was and was not verified. The Java code, the YAML profiles and the tests were compiled and run on 2026-11-16 with Spring Boot 4.1.2 (Spring for Apache Kafka 4.1.2, Kafka clients 4.2.2) against an embedded in-process Kafka broker started by the tests. Nothing here was run against Confluent Cloud: the Confluent-specific settings (SASL_SSL, PLAIN, the API key in the JAAS line, client.dns.lookup) come from the Confluent documentation, and the shell and Confluent CLI commands were not executed. Check them against your own cluster before relying on them.

    Step 1: Prerequisites and project creation

    You need a JDK (the example targets Java 25, and Spring Boot 4.1.2 supports Java 17 and later), the Confluent Cloud cluster, topic and service-account API key from section 15, and the bootstrap server of your cluster.

    Create the project at start.spring.io: choose Gradle or Maven, language Java, Spring Boot 4.1.2, Java 25, and add the two dependencies Spring Web and Spring for Apache Kafka. Generated on 2026-11-16, the Initializr's Gradle build had these dependencies:

    File: examples/spring-kafka-orders/build.gradle.kts (the dependencies block; the project's own file, which follows the generated one):

    dependencies {
        implementation("org.springframework.boot:spring-boot-starter-kafka")
        implementation("org.springframework.boot:spring-boot-starter-webmvc")
        testImplementation("org.springframework.boot:spring-boot-starter-kafka-test")
        testImplementation("org.springframework.boot:spring-boot-starter-webmvc-test")
        testRuntimeOnly("org.junit.platform:junit-platform-launcher")
    }

    The Maven equivalent below is the dependency list the Initializr generated for the same choices (it is not part of the example project, which uses Gradle):

    <dependencies>
    	<dependency>
    		<groupId>org.springframework.boot</groupId>
    		<artifactId>spring-boot-starter-kafka</artifactId>
    	</dependency>
    	<dependency>
    		<groupId>org.springframework.boot</groupId>
    		<artifactId>spring-boot-starter-webmvc</artifactId>
    	</dependency>
    
    	<dependency>
    		<groupId>org.springframework.boot</groupId>
    		<artifactId>spring-boot-starter-kafka-test</artifactId>
    		<scope>test</scope>
    	</dependency>
    	<dependency>
    		<groupId>org.springframework.boot</groupId>
    		<artifactId>spring-boot-starter-webmvc-test</artifactId>
    		<scope>test</scope>
    	</dependency>
    </dependencies>

    In Spring Boot 4 the starters are spring-boot-starter-kafka (which brings in spring-kafka, resolved here as org.springframework.kafka:spring-kafka:4.1.2) and spring-boot-starter-webmvc for the REST controller. Older Boot 3 projects use org.springframework.kafka:spring-kafka and spring-boot-starter-web instead, and some class names differ (see step 7). The application class is the usual one:

    File: src/main/java/com/example/orders/OrdersApplication.java

    package com.example.orders;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    
    @SpringBootApplication
    public class OrdersApplication {
    
        public static void main(String[] args) {
            SpringApplication.run(OrdersApplication.class, args);
        }
    }

    Step 2: Confluent Cloud settings

    Put credentials in environment variables. Never commit an API key or secret. The bootstrap server, API key and secret below are placeholders that you replace from the Console (Cluster settings, then API keys):

    File: src/main/resources/application.yml

    # Confluent Cloud settings. Every secret comes from an environment variable.
    # For a local broker add: --spring.profiles.active=local
    spring:
      application:
        name: spring-kafka-orders
      kafka:
        bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS}   # e.g. pkc-abc123.eu-west-1.aws.confluent.cloud:9092
        security:
          protocol: SASL_SSL
        properties:
          "[sasl.mechanism]": PLAIN
          "[sasl.jaas.config]": >-
            org.apache.kafka.common.security.plain.PlainLoginModule required
            username='${KAFKA_API_KEY}' password='${KAFKA_API_SECRET}';
          "[client.dns.lookup]": use_all_dns_ips
        producer:
          key-serializer: org.apache.kafka.common.serialization.StringSerializer
          value-serializer: org.apache.kafka.common.serialization.StringSerializer
          acks: all
        consumer:
          group-id: order-audit
          auto-offset-reset: earliest
          key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
          value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    
    orders:
      replication-factor: ${KAFKA_REPLICATION_FACTOR:3}
      retry:
        initial-interval-ms: 1000

    The bootstrap, security protocol, serializer, group and acks properties are listed in the Spring Boot properties appendix. SASL settings have no dedicated Boot property, so they go through spring.kafka.properties.*, which passes raw Kafka client properties (dotted keys in brackets). They are common properties, so they apply to the producer, consumer and admin clients at once. The client.dns.lookup entry follows Confluent's Java client advice for Cloud, and matters most with PrivateLink where one hostname resolves to several addresses.

    Export the three values before you start the app (the values are placeholders):

    export KAFKA_BOOTSTRAP_SERVERS=pkc-abc123.eu-west-1.aws.confluent.cloud:9092
    export KAFKA_API_KEY=YOUR_API_KEY
    export KAFKA_API_SECRET=YOUR_API_SECRET

    For a broker on your own machine, a second profile overrides the security settings. The tests also use it:

    File: src/main/resources/application-local.yml

    # Local broker, no authentication. The tests use it too (@ActiveProfiles("local")).
    spring:
      kafka:
        bootstrap-servers: localhost:9092
        security:
          protocol: PLAINTEXT
        properties:
          "[sasl.mechanism]": ""
          "[sasl.jaas.config]": ""
    
    orders:
      replication-factor: 1

    Step 3: Create the topics

    The app writes to orders and publishes failed records to orders.DLT. Create both with 3 partitions. In the Console: open your cluster, choose Topics, then Create topic, and either create with defaults or use Customize settings to set the partition count. With the Confluent CLI: confluent kafka topic create takes a --partitions flag, and --cluster and --environment select the target (confirm the flags with --help for your CLI version):

    confluent kafka topic create orders --partitions 3
    confluent kafka topic create orders.DLT --partitions 3

    Or let the app create them. Spring Boot creates a topic on startup for every NewTopic bean, and ignores the bean if the topic already exists. The example declares both topics, with the replication factor taken from the orders.replication-factor property (3 by default, 1 for the local profile):

    File: src/main/java/com/example/orders/KafkaConfig.java

    package com.example.orders;
    
    import org.apache.kafka.clients.admin.NewTopic;
    import org.apache.kafka.common.TopicPartition;
    import org.springframework.beans.factory.annotation.Value;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    import org.springframework.kafka.config.TopicBuilder;
    import org.springframework.kafka.core.KafkaOperations;
    import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
    import org.springframework.kafka.listener.DefaultErrorHandler;
    import org.springframework.kafka.support.ExponentialBackOffWithMaxRetries;
    
    @Configuration
    public class KafkaConfig {
    
        public static final String ORDERS_TOPIC = "orders";
        public static final String DEAD_LETTER_TOPIC = "orders.DLT";
    
        @Bean
        NewTopic ordersTopic(@Value("${orders.replication-factor}") short replicationFactor) {
            return TopicBuilder.name(ORDERS_TOPIC).partitions(3).replicas(replicationFactor).build();
        }
    
        // Same partition count as "orders": the recoverer below sends each dead letter to the same partition number
        @Bean
        NewTopic deadLetterTopic(@Value("${orders.replication-factor}") short replicationFactor) {
            return TopicBuilder.name(DEAD_LETTER_TOPIC).partitions(3).replicas(replicationFactor).build();
        }
    
        // Spring Boot adds a CommonErrorHandler bean to the auto-configured listener container factory.
        @Bean
        DefaultErrorHandler errorHandler(KafkaOperations<Object, Object> template,
                                         @Value("${orders.retry.initial-interval-ms}") long initialIntervalMs) {
            var recoverer = new DeadLetterPublishingRecoverer(template,
                (record, ex) -> new TopicPartition(DEAD_LETTER_TOPIC, record.partition()));
            // 2 retries after the first failure (3 attempts): wait 1 s, then 2 s, then give up and dead-letter
            var backOff = new ExponentialBackOffWithMaxRetries(2);
            backOff.setInitialInterval(initialIntervalMs);
            backOff.setMultiplier(2.0);
            return new DefaultErrorHandler(recoverer, backOff);
        }
    }

    The dead letter topic has the same partition count as orders on purpose: the recoverer in step 8 sends each failed record to the same partition number, so the Spring for Apache Kafka documentation says the dead letter topic must have at least as many partitions as the original.

    When the app must not create topics. Confluent documents that automatic topic creation is disabled by default in Confluent Cloud, and creating a topic from an application needs the DeveloperManage role (section 15, step 5). In production most teams create topics through the Console, the CLI or infrastructure as code, and give the application key only DeveloperWrite and DeveloperRead. Switch the startup creation off with spring.kafka.admin.auto-create: false (the Boot 4.1.2 configuration metadata describes it as "Whether to automatically create topics during context initialization"). I did not test what Confluent Cloud does when NewTopic asks for a replication factor that your cluster type does not offer, so check that before relying on the bean.

    Step 4: The producer

    The producer is a Spring bean with a constructor-injected KafkaTemplate. The key is the order id, so every event for one order goes to one partition and keeps its order. send returns a CompletableFuture<SendResult<K, V>>; whenComplete handles the outcome without blocking.

    File: src/main/java/com/example/orders/OrderProducer.java

    package com.example.orders;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.context.annotation.Profile;
    import org.springframework.kafka.core.KafkaTemplate;
    import org.springframework.stereotype.Service;
    
    @Service
    @Profile("!json")
    public class OrderProducer implements OrderPublisher {
    
        private static final Logger log = LoggerFactory.getLogger(OrderProducer.class);
    
        private final KafkaTemplate<String, String> kafkaTemplate;
    
        public OrderProducer(KafkaTemplate<String, String> kafkaTemplate) {
            this.kafkaTemplate = kafkaTemplate;
        }
    
        @Override
        public void publish(OrderCreated order) {
            // key = orderId keeps every event for one order on one partition
            kafkaTemplate.send(KafkaConfig.ORDERS_TOPIC, order.orderId(), "Order created: " + order.orderId())
                .whenComplete((result, ex) -> {
                    if (ex != null) {
                        // log, alert, or route to a retry path: do not ignore a failed send
                        log.error("Publish failed for order {}", order.orderId(), ex);
                    } else {
                        var meta = result.getRecordMetadata();
                        log.info("Sent topic={} partition={} offset={} key={}",
                            meta.topic(), meta.partition(), meta.offset(), order.orderId());
                    }
                });
        }
    }

    The class implements a small interface, so the controller in step 5 does not care whether the payload is a String or JSON (step 7):

    File: src/main/java/com/example/orders/OrderPublisher.java

    package com.example.orders;
    
    /** What the REST controller needs. Two implementations: String payload (default) and JSON ("json" profile). */
    public interface OrderPublisher {
    
        void publish(OrderCreated order);
    }

    The business event is a Java record. It is the HTTP request body, and in the json profile it is also the Kafka value:

    File: src/main/java/com/example/orders/OrderCreated.java

    package com.example.orders;
    
    import java.math.BigDecimal;
    
    /** The business event. Used as the HTTP request body and, in the "json" profile, as the Kafka value. */
    public record OrderCreated(String orderId, String customerId, BigDecimal amount) {
    }

    The infographic's snippet omits the constructor and the result handling, which is where production bugs hide: a failed send is easy to miss if nobody looks at the future. Log it, alert on it, or route it to a retry path.

    Step 5: The REST endpoint that triggers it

    The controller accepts an order and hands it to the producer. It returns 202 Accepted, because the send is asynchronous: the response does not mean the broker has stored the record.

    File: src/main/java/com/example/orders/OrderController.java

    package com.example.orders;
    
    import org.springframework.http.HttpStatus;
    import org.springframework.web.bind.annotation.PostMapping;
    import org.springframework.web.bind.annotation.RequestBody;
    import org.springframework.web.bind.annotation.ResponseStatus;
    import org.springframework.web.bind.annotation.RestController;
    
    @RestController
    public class OrderController {
    
        private final OrderPublisher publisher;
    
        public OrderController(OrderPublisher publisher) {
            this.publisher = publisher;
        }
    
        // 202 Accepted: the send is asynchronous, so the response does not say the broker has stored the record yet
        @PostMapping("/orders")
        @ResponseStatus(HttpStatus.ACCEPTED)
        public void create(@RequestBody OrderCreated order) {
            publisher.publish(order);
        }
    }

    Step 6: The consumer

    The consumer needs a group and a deserializer. They are already in application.yml (step 2): group-id: order-audit, auto-offset-reset: earliest (a new group reads the existing records instead of only new ones) and the two String deserializers. The listener receives the whole ConsumerRecord, so it can log the topic, partition, offset and key next to the value.

    File: src/main/java/com/example/orders/OrderConsumer.java

    package com.example.orders;
    
    import org.apache.kafka.clients.consumer.ConsumerRecord;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.context.annotation.Profile;
    import org.springframework.kafka.annotation.KafkaListener;
    import org.springframework.stereotype.Component;
    
    @Component
    @Profile("!json")
    public class OrderConsumer {
    
        private static final Logger log = LoggerFactory.getLogger(OrderConsumer.class);
    
        @KafkaListener(topics = KafkaConfig.ORDERS_TOPIC, groupId = "order-audit")
        public void onOrder(ConsumerRecord<String, String> record) {
            log.info("Received topic={} partition={} offset={} key={} value={}",
                record.topic(), record.partition(), record.offset(), record.key(), record.value());
            // process idempotently: at-least-once delivery means duplicates can arrive
            if (record.value().contains("poison")) {
                throw new IllegalArgumentException("Cannot process order " + record.key());
            }
        }
    }

    At-least-once delivery means duplicates can arrive, so process each record idempotently. The poison check is a deliberate failure for step 8: an order whose id starts with poison always throws. Remove it in real code.

    Step 7: JSON payloads

    Strings are fine for a first message, but real events are structured. The json profile switches the producer to Spring Kafka's JSON serializer and the consumer to its JSON deserializer. Spring Boot 4 uses Jackson 3, so the classes are named JacksonJsonSerializer and JacksonJsonDeserializer (older Spring Kafka versions call them JsonSerializer and JsonDeserializer). The consumer only builds objects from packages you trust, so set spring.json.trusted.packages, and give it a default type in case the record carries no type header:

    File: src/main/resources/application-json.yml

    # JSON payloads: add --spring.profiles.active=json (or json,local for a local broker)
    spring:
      kafka:
        producer:
          value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
        consumer:
          value-deserializer: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
          properties:
            "[spring.json.trusted.packages]": com.example.orders
            "[spring.json.value.default.type]": com.example.orders.OrderCreated

    The two classes that apply in this profile (the String ones above are switched off with @Profile("!json")):

    File: src/main/java/com/example/orders/JsonOrderProducer.java

    package com.example.orders;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.context.annotation.Profile;
    import org.springframework.kafka.core.KafkaTemplate;
    import org.springframework.stereotype.Service;
    
    @Service
    @Profile("json")
    public class JsonOrderProducer implements OrderPublisher {
    
        private static final Logger log = LoggerFactory.getLogger(JsonOrderProducer.class);
    
        private final KafkaTemplate<String, OrderCreated> kafkaTemplate;
    
        public JsonOrderProducer(KafkaTemplate<String, OrderCreated> kafkaTemplate) {
            this.kafkaTemplate = kafkaTemplate;
        }
    
        @Override
        public void publish(OrderCreated order) {
            kafkaTemplate.send(KafkaConfig.ORDERS_TOPIC, order.orderId(), order)
                .whenComplete((result, ex) -> {
                    if (ex != null) {
                        log.error("Publish failed for order {}", order.orderId(), ex);
                    } else {
                        var meta = result.getRecordMetadata();
                        log.info("Sent topic={} partition={} offset={} key={}",
                            meta.topic(), meta.partition(), meta.offset(), order.orderId());
                    }
                });
        }
    }

    File: src/main/java/com/example/orders/JsonOrderConsumer.java

    package com.example.orders;
    
    import org.apache.kafka.clients.consumer.ConsumerRecord;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.context.annotation.Profile;
    import org.springframework.kafka.annotation.KafkaListener;
    import org.springframework.stereotype.Component;
    
    @Component
    @Profile("json")
    public class JsonOrderConsumer {
    
        private static final Logger log = LoggerFactory.getLogger(JsonOrderConsumer.class);
    
        @KafkaListener(topics = KafkaConfig.ORDERS_TOPIC, groupId = "order-audit")
        public void onOrder(ConsumerRecord<String, OrderCreated> record) {
            OrderCreated order = record.value();
            log.info("Received topic={} partition={} offset={} key={} value={}",
                record.topic(), record.partition(), record.offset(), record.key(), order);
            if (order.orderId().startsWith("poison")) {
                throw new IllegalArgumentException("Cannot process order " + order.orderId());
            }
        }
    }

    Start it with --spring.profiles.active=json. The topic then holds text such as {"orderId":"J-1","customerId":"c-9","amount":19.99}, which a test in the project reads back with a plain String consumer. One gap remains, and the example does not cover it: a record that is not valid JSON fails inside the deserializer, before your listener runs. The Spring for Apache Kafka documentation describes ErrorHandlingDeserializer for that case, and lists DeserializationException among the exceptions that are never retried. In a real service, wrap the JSON deserializer with it.

    Step 8: Error handling and the dead letter topic

    A listener that throws must not block its partition forever. DefaultErrorHandler retries the record with a back-off and then hands it to a recoverer; DeadLetterPublishingRecoverer publishes it to another topic. In Spring Boot you only add the error handler as a @Bean and Boot attaches it to the auto-configured listener container factory. The bean is in KafkaConfig (step 3):

    File: src/main/java/com/example/orders/KafkaConfig.java (the error handler, excerpt of the file above)

        // Spring Boot adds a CommonErrorHandler bean to the auto-configured listener container factory.
        @Bean
        DefaultErrorHandler errorHandler(KafkaOperations<Object, Object> template,
                                         @Value("${orders.retry.initial-interval-ms}") long initialIntervalMs) {
            var recoverer = new DeadLetterPublishingRecoverer(template,
                (record, ex) -> new TopicPartition(DEAD_LETTER_TOPIC, record.partition()));
            // 2 retries after the first failure (3 attempts): wait 1 s, then 2 s, then give up and dead-letter
            var backOff = new ExponentialBackOffWithMaxRetries(2);
            backOff.setInitialInterval(initialIntervalMs);
            backOff.setMultiplier(2.0);
            return new DefaultErrorHandler(recoverer, backOff);
        }

    What it does, as exercised by the tests:

    • ExponentialBackOffWithMaxRetries(2) allows 2 retries after the first failure, so 3 attempts in total, waiting 1 second and then 2 seconds with the default orders.retry.initial-interval-ms: 1000. The tests set it to 100 ms to run fast.
    • After the last attempt the record goes to orders.DLT. The destination resolver is explicit because the Spring for Apache Kafka 4.1.2 documentation says the default dead letter topic is named <originalTopic>-dlt, not .DLT. The resolver keeps the original partition number, which is why both topics have 3 partitions.
    • Retrying happens in place on the consumer thread, so keep the back-off short; later records of the same partition wait. Failures that retrying cannot fix should not be retried at all: handler.addNotRetryableExceptions(...) marks them.

    Something has to read the dead letter topic. In this example a second listener only logs what arrived, standing in for an alert, a parking table or a replay tool:

    File: src/main/java/com/example/orders/DeadLetterConsumer.java

    package com.example.orders;
    
    import org.apache.kafka.clients.consumer.ConsumerRecord;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.kafka.annotation.KafkaListener;
    import org.springframework.stereotype.Component;
    
    /** Stands in for whatever your team does with failed records: alert, park, inspect, replay. */
    @Component
    public class DeadLetterConsumer {
    
        private static final Logger log = LoggerFactory.getLogger(DeadLetterConsumer.class);
    
        @KafkaListener(topics = KafkaConfig.DEAD_LETTER_TOPIC, groupId = "order-dlt-inspector")
        public void onDeadLetter(ConsumerRecord<String, Object> record) {
            log.warn("Dead letter topic={} partition={} offset={} key={} value={}",
                record.topic(), record.partition(), record.offset(), record.key(), record.value());
        }
    }

    Step 9: Run it and send a test request

    Run the app against your cluster, with the environment variables from step 2 set, or use the local profile for a broker on localhost:9092:

    ./gradlew bootRun
    ./gradlew bootRun --args='--spring.profiles.active=local'

    Send an order. This is the HTTP call the controller accepts; the three fields match the OrderCreated record:

    curl -X POST http://localhost:8080/orders \
      -H "Content-Type: application/json" \
      -d '{"orderId":"1001","customerId":"c-1","amount":19.99}'

    You get an empty 202 Accepted. In the app log you should see one Sent ... line from the producer and one Received ... line from the consumer (step 11 shows what they look like). To see the dead letter path, send an order whose id starts with poison:

    curl -X POST http://localhost:8080/orders \
      -H "Content-Type: application/json" \
      -d '{"orderId":"poison-1","customerId":"c-1","amount":1.00}'

    The consumer logs three attempts, then the dead letter listener logs the record from orders.DLT.

    17. The Kafka CLI alternative

    The infographic builds the client config inline with a process substitution. A properties file is easier to read and debug. Keep the file out of source control and out of your shell history. The project includes a template with placeholders:

    File: client.properties.example (copy it to client.properties and fill in your values)

    # client.properties for the Kafka CLI tools (illustrative: replace the placeholders from your environment)
    bootstrap.servers=pkc-abc123.eu-west-1.aws.confluent.cloud:9092
    security.protocol=SASL_SSL
    sasl.mechanism=PLAIN
    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='YOUR_API_KEY' password='YOUR_API_SECRET';
    # Produce (type a line, press Enter, Ctrl+C to exit). Confirm option names with --help for your Kafka version.
    kafka-console-producer --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" \
      --producer.config client.properties --topic orders
    
    # Consume
    kafka-console-consumer --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" \
      --consumer.config client.properties --topic orders --from-beginning

    Apache Kafka's quickstart shows the plain form (--topic, --bootstrap-server, --from-beginning); confirm the client config option names with --help, because the quickstart does not show them. The Confluent CLI is an alternative that handles authentication: confluent kafka topic produce <topic> (with --parse-key and --delimiter) and confluent kafka topic consume <topic> --from-beginning (with --group and --print-key), both accepting --api-key and --api-secret.

    A message typed into the console producer has no key unless you use --parse-key, and it is not JSON, so in the json profile the app cannot deserialize it. Use the same profile on both sides.

    18. Verify the message three ways, then test it

    Step 10: Verify

    Method How What you should see
    Consumer app Start the Spring Boot app in the same environment Your listener logs the record: topic, partition, offset, key and value
    CLI consumer Run the console consumer or confluent kafka topic consume orders --from-beginning (add --print-key to see the key) The record's value, such as Order created: 1001
    Confluent Cloud Console Open Topics, then orders, then Messages The record with its partition, offset and timestamp

    For the dead letter path, run the same CLI consumer on orders.DLT, or open that topic in the Console. Verify from inside the same private network: a laptop consumer proves the topic works, not that the PrivateLink path does.

    The infographic's expected consumer output, Order created: 1001, is illustrative, not captured from a real run. Step 11 shows log lines that were captured from a test run; your partitions and offsets will differ.

    Step 11: Test it without a cloud account

    The tests start an in-process Kafka broker (KRaft mode) with @EmbeddedKafka, so they need no Docker and no Confluent account. The test dependency is spring-boot-starter-kafka-test (step 1), as the Spring for Apache Kafka testing documentation recommends for Spring Boot. The annotation sets spring.kafka.bootstrap-servers to the embedded broker by default, so the app connects to it without extra code. @ActiveProfiles("local") is what switches off the SASL settings, which an embedded broker does not use.

    File: src/test/java/com/example/orders/OrderFlowTest.java (class setup)

    @SpringBootTest(
        webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = "orders.retry.initial-interval-ms=100")
    @EmbeddedKafka(partitions = 3)
    @ActiveProfiles("local")
    @DirtiesContext
    class OrderFlowTest {
    
        @LocalServerPort
        int port;
    
        @Autowired
        OrderPublisher publisher;
    
        @Autowired
        KafkaAdmin kafkaAdmin;
    
        LogCapture logs;

    Produce, then consume: the test posts an order through the REST endpoint and waits until the consumer has logged it with the right key and value.

    File: src/test/java/com/example/orders/OrderFlowTest.java

        @Test
        void postedOrderIsConsumedWithItsKeyAndValue() {
            RestClient.create("http://localhost:" + port).post()
                .uri("/orders")
                .header("Content-Type", "application/json")
                .body("{\"orderId\":\"1001\",\"customerId\":\"c-1\",\"amount\":19.99}")
                .retrieve()
                .toBodilessEntity();
    
            await().atMost(Duration.ofSeconds(30)).untilAsserted(() ->
                assertThat(logs.lines("Received topic=orders"))
                    .anySatisfy(line -> assertThat(line)
                        .contains("key=1001")
                        .contains("value=Order created: 1001")));
        }

    Key to partition: the same key always maps to the same partition. The expected partition is computed with the same murmur2 hash the default partitioner uses.

    File: src/test/java/com/example/orders/OrderFlowTest.java

        @Test
        void sameKeyAlwaysGoesToTheSamePartition() {
            for (int i = 0; i < 5; i++) {
                publisher.publish(new OrderCreated("stable-key", "c-2", java.math.BigDecimal.TEN));
            }
    
            await().atMost(Duration.ofSeconds(30)).untilAsserted(() ->
                assertThat(logs.lines("Sent topic=orders partition=")).hasSize(5));
    
            Set<String> partitions = logs.lines("Sent topic=orders partition=").stream()
                .map(line -> line.split(" ")[2])
                .collect(Collectors.toSet());
            // the partitioner hashes the key bytes with murmur2 and takes the result modulo the partition count
            int expected = Utils.toPositive(Utils.murmur2("stable-key".getBytes(UTF_8))) % 3;
            assertThat(partitions).containsExactly("partition=" + expected);
        }

    A poison record: three attempts, then the record lands on orders.DLT on the same partition.

    File: src/test/java/com/example/orders/OrderFlowTest.java

        @Test
        void poisonRecordIsRetriedThenLandsOnTheDeadLetterTopic() {
            publisher.publish(new OrderCreated("poison-1", "c-3", java.math.BigDecimal.ONE));
    
            await().atMost(Duration.ofSeconds(30)).untilAsserted(() ->
                assertThat(logs.lines("Dead letter topic=orders.DLT")).anySatisfy(line ->
                    assertThat(line).contains("key=poison-1").contains("value=Order created: poison-1")));
    
            // 1 first delivery + 2 retries = 3 attempts, all on the same partition and offset
            List<String> attempts = logs.lines("Received topic=orders").stream()
                .filter(line -> line.contains("key=poison-1"))
                .toList();
            assertThat(attempts).hasSize(3).allSatisfy(line -> assertThat(line).isEqualTo(attempts.get(0)));
    
            // the dead letter keeps the original partition number
            String originalPartition = attempts.get(0).split(" ")[2];
            assertThat(logs.lines("Dead letter topic=orders.DLT").getFirst()).contains(originalPartition);
        }

    Run all of them from examples/spring-kafka-orders:

    ./gradlew test

    There are 7 tests in 3 classes: 4 for the String flow (topic creation, produce and consume, key stability, poison to the dead letter topic), 2 for the JSON profile and 1 that writes the log below. All 7 passed on 2026-11-16 (Gradle 9.8.2, JDK 26 compiling for Java 25).

    The log of one session comes from GoldenRunTest, which sends orders 1001, 1002, 1003, 1001 again and one poison order, collects the producer, consumer and dead letter log lines, and writes them to golden/run.txt. A normal test run compares against that file, so the text below cannot drift from the code. The lines are sorted by partition and offset because a consumer does not read different partitions in a fixed order.

    Captured from a real run on an embedded Kafka broker (not Confluent Cloud):

    # producer (OrderProducer)
    Sent topic=orders partition=0 offset=0 key=1001
    Sent topic=orders partition=0 offset=1 key=1002
    Sent topic=orders partition=0 offset=2 key=1001
    Sent topic=orders partition=1 offset=0 key=1003
    Sent topic=orders partition=2 offset=0 key=poison-1
    
    # consumer (OrderConsumer, group order-audit)
    Received topic=orders partition=0 offset=0 key=1001 value=Order created: 1001
    Received topic=orders partition=0 offset=1 key=1002 value=Order created: 1002
    Received topic=orders partition=0 offset=2 key=1001 value=Order created: 1001
    Received topic=orders partition=1 offset=0 key=1003 value=Order created: 1003
    Received topic=orders partition=2 offset=0 key=poison-1 value=Order created: poison-1
    Received topic=orders partition=2 offset=0 key=poison-1 value=Order created: poison-1
    Received topic=orders partition=2 offset=0 key=poison-1 value=Order created: poison-1
    
    # dead letter topic (DeadLetterConsumer)
    Dead letter topic=orders.DLT partition=2 offset=0 key=poison-1 value=Order created: poison-1

    Read it like this: orders 1001 and 1002 hash to partition 0 in order (offsets 0, 1 and 2, the second 1001 coming after 1002), 1003 went to partition 1, and poison-1 to partition 2. The consumer saw the poison record three times at the same partition and offset, which is the retry, and then one copy arrived on orders.DLT on the same partition 2.

    What this does not prove: behaviour against Confluent Cloud (authentication, PrivateLink, role bindings, what the cluster does with the replication factor), throughput, rebalancing with several consumers, or exactly-once. It proves that the producer, the consumer, the JSON path and the retry and dead letter path work together on a real Kafka broker in-process.

    19. Security and access checklist

    Check Why
    Use service accounts, not personal users Keys survive staff changes and can be rotated per application
    Grant least-privilege roles per topic and group A producer should not be able to read, and no app needs CloudClusterAdmin
    Create one service account per application Revocation and audit stay narrow
    Rotate API keys regularly Keys remain valid until rotated or deleted
    Store key and secret in AWS Secrets Manager or Parameter Store The secret is shown once and must never reach Git or logs
    Enforce TLS (SASL_SSL) PLAIN sends the secret as a password, so TLS is mandatory
    Restrict network access PrivateLink, plus security groups scoped to your VPC CIDR
    Check access before deleting accounts Deleting a service account revokes access through its keys

    20. Production readiness checklist

    • Cluster type and partition counts sized from throughput and consumer needs.
    • Retention, compaction and cleanup policies defined per topic.
    • Monitoring in place: Confluent metrics plus AWS CloudWatch for your side, with alerts on consumer lag and errors.
    • Backup, disaster-recovery strategy and multi-AZ or multi-region choice agreed and tested.
    • Cost controls for data transfer, storage, Schema Registry and connectors.
    • Load test of the producer and consumer, with service-level objectives set from the results.
    • Dead-letter topic and retry policy for poison records.
    • Idempotent consumers, because at-least-once delivery means duplicates happen.

    21. Common pitfalls and fixes

    Symptom Likely cause Fix
    Connection timeout Network path or DNS wrong: endpoint, security group, private DNS, or a missing zone Verify the PrivateLink or peering setup. Check the endpoint's security group allows 443 and 9092 from your VPC, and that the bootstrap name resolves to private addresses
    Authentication failed Wrong API key or secret, wrong mechanism, a key from another cluster, or a mangled JAAS string Recheck sasl.mechanism=PLAIN, the JAAS line, and that the key belongs to this cluster and a service account
    Topic not found Name typo, wrong cluster or environment, or the topic was never created Create the topic first. Auto-creation is disabled by default in Confluent Cloud
    Authorization error The service account has no permission on the topic or group Grant produce or consume permission (roles or ACL) for the topic and the consumer group
    No message received Different topic or group id, offsets already past the record, no data yet, wrong partition assumption Check topic and group.id. Use --from-beginning or auto-offset-reset: earliest for a new group. Inspect lag and partition assignment

    A brand-new consumer group defaults to auto.offset.reset=latest, so it skips anything published before it joined unless you ask for earliest.

    Where this article refines the infographics

    Infographics compress. Where wording was loose or incomplete, this article follows the official documentation:

    • CloudClusterAdmin is a cluster-wide administrator, not a least-privilege role. Use DeveloperWrite and DeveloperRead (plus group access) scoped to the topic.
    • PrivateLink or VPC peering is not open to every cluster type (section 13). The PrivateLink security group rule is inbound (443, 9092) from your VPC CIDR, not egress.
    • Control Center and REST Proxy are Confluent Platform features, and Confluent Platform provides a storage implementation for Kafka tiered storage. Confluent Cloud has the Console, a Kafka REST API and Infinite Storage.
    • Confluent Cloud documents SASL PLAIN (API key) and OAUTHBEARER for clients. Confluent Cloud also supports mTLS, where you upload your own certificate authority (see the Confluent Cloud mTLS documentation in the references).
    • Producers fetch metadata from brokers, not specifically the controller. A partition has zero or more followers. Compaction keeps at least the latest value per key.
    • A replication factor of 2 is a diagram value. Production defaults are 3 with a deliberate min.insync.replicas.
    • Exactly-once holds inside Kafka only; effects on external systems still need idempotent handling.
    • In the AWS infographic's CLI example, the client settings file is passed with --producer.config (check --help for your CLI version). In the Spring example, KafkaTemplate is injected through a constructor and the result of send should be handled, as the code in this article shows.

    Key takeaways

    • Use Kafka to build scalable, real-time, event-driven systems. Topics store immutable, ordered events in partitions.
    • Producers publish, and consumers read independently through consumer groups. Parallelism is capped by partitions.
    • Design for reliability: replication, acks=all, idempotence, retries and monitoring. Remember that exactly-once stops at Kafka's edge.
    • Use Confluent services (Schema Registry, Connect, Streams, ksqlDB, Flink) as needed, and check whether each is a Confluent Cloud feature or a Confluent Platform component.
    • Plan retention, compaction, security and operations before production, not after.
    • On AWS: pick a cluster type that supports your network, connect privately, use a service account with scoped roles, create the topic yourself, publish, then verify from inside the same network.

    Official references

    Related reading on this site: Kafka Scalability Patterns, Event-driven AI with Kafka Streams and Multi-agent Java and Kafka architecture.

    Keep reading — it's free

    How Confluent Kafka works from producers and partitions to consumer groups and delivery guarantees, plus onboarding Confluent Cloud on AWS with Spring Boot.

    In the rest of this article

    • 14. Target architecture: PrivateLink
    • 15. The seven onboarding steps
    • 16. Publish a message from Spring Boot
    • 17. The Kafka CLI alternative
    • 18. Verify the message three ways, then test it
    • 19. Security and access checklist

    A free account unlocks the rest of this article, the full prompt library, quiz answer reviews and completion certificates. The newsletter is optional.

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Was this helpful?
    Go deeper

    Want to go from reading to building?

    Take it further with the free, hands-on courses — structured paths that turn these ideas into working systems, with code and exercises.

    Article: How Confluent Kafka Works: Architecture, Delivery Guarantees and Publishing from Spring Boot on AWS

    Free: LLM Production Readiness Checklist (PDF)

    50 checks across observability, rate limiting, cost optimization, failure handling, and security — for teams shipping AI features to production.

    No spam. Unsubscribe any time.

    Comments