Integration / Apache Pulsar Interview questions

Apache Pulsar is an all-in-one messaging and streaming platform. Messages can be consumed and acknowledged individually or consumed as streams with less than 10ms of latency. Its layered architecture allows rapid scaling across hundreds of nodes, without data reshuffling.

Its features include multi-tenancy with resource separation and access control, geo-replication across regions, tiered storage and support for six official client languages. It supports up to one million unique topics and is designed to simplify your application architecture.

Apache Pulsar is a distributed, cloud-native messaging and streaming platform, originally built at Yahoo and now an Apache Software Foundation project, designed to handle both traditional queuing and real-time stream processing in one system.

Its defining trait is a segmented architecture that separates the serving layer (stateless brokers) from the storage layer (Apache BookKeeper), letting each scale independently instead of coupling compute and storage on the same node.

Built-in multi-tenancy, geo-replication, and a unified pub-sub/queue model make it a common alternative to running Kafka alongside a separate queuing system.

A broker is the stateless component that handles all producer and consumer traffic for the topics it owns - accepting published messages, dispatching them to subscribed consumers, and enforcing policies - without storing the actual message data itself.

Because brokers hold no persistent state, any broker can take over a topic; on failure only ownership needs to move, not data, letting failover happen in seconds instead of minutes.

Brokers write and read message entries by talking to a BookKeeper ensemble behind the scenes, and cache recently read entries in memory to serve consumers without always hitting BookKeeper directly.

Apache BookKeeper is the distributed, replicated log storage system Pulsar uses to durably persist every message; it's a separate Apache project that predates Pulsar and provides low-latency, append-only storage.

Storage is organized into ledgers, split into entries; a ledger's entries are striped across multiple storage nodes called bookies according to configurable ensemble, write-quorum, and ack-quorum settings for durability.

Because BookKeeper - not the broker - owns durability, a broker can be replaced without any data movement, and bookies can be added or removed to scale storage independently of the compute layer.

A tenant is the top-level unit of multi-tenancy in Pulsar, typically mapped to an organization or team, used to apply resource quotas and authentication/authorization boundaries.

A namespace is a logical grouping of topics within a tenant - for example separating dev, staging, and prod - and it's the unit at which most operational policies (retention, replication, backlog quotas) are configured for every topic inside it.

Together, tenant/namespace/topic form Pulsar's hierarchical naming structure, e.g. persistent://my-tenant/my-namespace/my-topic, letting large organizations share one cluster safely.

A topic is the named channel through which producers publish messages and consumers subscribe to receive them - the fundamental unit of pub-sub in Pulsar.

Topics can be persistent (durably backed by BookKeeper) or non-persistent (kept only in broker memory for lower latency at the cost of durability), and either can be partitioned or non-partitioned.

A topic's full name always includes its tenant and namespace, e.g. persistent://tenant/namespace/topic, which is what enables multi-tenant isolation even when different teams reuse the same topic name.

A producer is a client process that publishes messages to a specific topic; it connects to the broker currently owning that topic (or partition) and sends messages that BookKeeper then durably stores.

Producers can be configured with batching, compression, and a routing mode for partitioned topics (round-robin, single-partition, or key-based) that determines which partition each message lands on.

Producers also support message deduplication using a producer name and per-message sequence ID, so the broker can detect and discard retried duplicates from the same producer.

A consumer is a client process that subscribes to a topic (via a named subscription) to receive and process published messages, then acknowledges each message once it's been handled.

A consumer's behavior - which messages it sees, and whether other consumers share the load - is determined by the subscription type it uses (Exclusive, Shared, Failover, or Key_Shared).

Unacknowledged messages remain in the subscription's backlog and will be redelivered - immediately on a negative ack, or after a timeout for an unresponsive consumer - so acknowledgment is what actually advances the subscription's read position.

A subscription is a named cursor that tracks a consumer group's read position on a topic, independent from the topic's own data - it's what makes Pulsar behave like a durable queue and a pub-sub system at once, depending on how it's used.

Multiple subscriptions can exist on the same topic simultaneously, each tracking its own independent position, so different consumer groups can read the same stream of messages at their own pace without affecting each other.

The chosen subscription type controls how consumers attached to that subscription divide up the topic's messages.

A partitioned topic is a topic actually implemented as a set of internal sub-topics (partitions), each of which can be owned and served by a different broker, letting a single logical topic scale throughput beyond what one broker could handle alone.

Producers route each message to a specific partition (round-robin, key hash, or custom logic), and consumers on a Shared or Key_Shared subscription can be spread across partitions to parallelize consumption.

Unlike a non-partitioned topic (served entirely by one broker), a partitioned topic's ordering guarantee only applies within each partition, not across the topic as a whole.

The metadata store (traditionally ZooKeeper, though Pulsar also supports other backends) holds cluster-wide coordination state - broker membership, topic-to-broker ownership assignments, namespace policies, and BookKeeper ledger metadata locations.

Brokers watch the metadata store to know which topics they own and to detect when other brokers join or leave the cluster, which is how load rebalancing and failover decisions get triggered.

It does not store message data itself - that's BookKeeper's job - so metadata store load stays comparatively light and predictable even as message throughput scales.

Pulsar Functions are a lightweight, serverless-style compute framework built into Pulsar that let you run simple processing logic - a Java, Python, or Go function - directly against messages on a topic without standing up a separate stream-processing cluster.

A function reads from one or more input topics, applies user code to each message, and optionally writes results to an output topic, with Pulsar handling deployment, scaling, and fault tolerance for you.

They suit lightweight transformations, filtering, or enrichment; for complex stateful stream processing at scale, teams typically still reach for a dedicated engine like Flink, but Functions cover a large share of everyday use cases with far less operational overhead.

Pulsar IO is a connector framework, built on top of Pulsar Functions, that provides ready-made sources (to pull data into Pulsar from external systems) and sinks (to push data from Pulsar to external systems) without writing custom integration code.

Common connectors include sources/sinks for Kafka, JDBC databases, Elasticsearch, and cloud storage, each configurable declaratively rather than requiring a bespoke client application.

Because connectors run as managed Functions, they inherit the same deployment, scaling, and fault-tolerance handling, so operators manage data pipelines the same way they manage any other Pulsar Function.

Geo-replication is Pulsar's built-in capability to asynchronously replicate messages published in one cluster to one or more other clusters, without requiring an external tool.

It's configured at the namespace level by listing which clusters a namespace should replicate to; once enabled, each locally published message is automatically forwarded to the configured remote clusters while a replication cursor tracks progress independently per destination.

This supports both active-active setups (regional low-latency access to shared data) and active-passive setups (disaster recovery), because Pulsar's cluster model was designed multi-region from the start rather than bolted on afterward.

Tiered storage lets Pulsar automatically offload older message segments from BookKeeper to cheaper, long-term storage like Amazon S3, Google Cloud Storage, or Azure Blob once they age past a configured threshold.

Offloaded segments remain fully readable through the same topic and consumer APIs - Pulsar transparently fetches them from the offload target when a consumer needs to read that far back - so there's no separate archive workflow to manage.

This decouples retention length from BookKeeper disk cost, letting teams keep months or years of history queryable without provisioning that much expensive replicated disk on the bookies themselves.

A non-persistent topic keeps messages only in broker memory and never writes them to BookKeeper, trading durability for lower publish latency and no storage overhead.

If a broker restarts or crashes, any messages still in flight on a non-persistent topic are lost, and there's no backlog to replay - consumers only see messages published while they're actively connected.

It suits use cases where occasionally losing a message is acceptable in exchange for speed, such as ephemeral real-time metrics or best-effort notifications, not data that must be durably retained.

Pulsar offers four subscription types: Exclusive (one consumer, full topic ordering), Failover (multiple registered consumers but only one active, providing hot-standby failover), Shared (many consumers round-robin messages with no ordering guarantee), and Key_Shared (many consumers, but same-key messages always go to the same consumer, preserving per-key order).

Choosing between them is really a choice about how strictly you need ordering versus how much you need to parallelize consumption across multiple consumer instances.

All four types can be used on the same topic simultaneously via different subscription names, since each subscription tracks its own independent state.

Retention determines how long already-acknowledged messages are kept in BookKeeper after every subscription has consumed them, since by default Pulsar deletes data once no subscription needs it anymore.

Configuring a retention policy (size- and/or time-based) at the namespace or topic level keeps messages around even after acknowledgment, useful for replaying history or onboarding a new subscription that needs to read from an earlier point.

This differs from a backlog quota, which limits how much unacknowledged data can accumulate for slow or stalled consumers - retention protects already-read data, backlog quota protects against unread data growing unbounded.

Pulsar has a built-in schema registry that lets producers and consumers agree on message structure - supporting Avro, JSON, Protobuf, and plain primitives - without operating a separate schema service.

When a producer with a defined schema publishes to a topic, Pulsar validates and stores that schema version; consumers can then deserialize messages using the matching schema automatically instead of hand-parsing bytes.

Schema compatibility checks (backward, forward, full) can be enforced per topic, preventing an incompatible producer from breaking existing consumers when the message format evolves.

A Pulsar cluster is built from four core pieces: brokers (stateless, handle pub-sub traffic), bookies (BookKeeper nodes providing durable storage), a metadata store (cluster coordination and ownership state), and optionally a proxy layer (a stateless gateway fronting brokers for client connections).

Brokers and bookies scale independently of each other - you can add bookies to grow storage capacity without touching broker count, or add brokers to grow serving/compute capacity without adding storage.

This separation is the architectural core that differentiates Pulsar's operational model from a system where each node handles both serving and storage together.

The simplest way is auto-creation: publishing to or subscribing on a topic name that doesn't yet exist will create it automatically by default, using the namespace's default partition settings.

For explicit control, the pulsar-admin CLI (or the equivalent admin API) can create a topic deliberately - for example pulsar-admin topics create persistent://tenant/namespace/my-topic for a non-partitioned topic, or create-partitioned-topic with a partition count for a partitioned one.

Many production clusters disable auto-creation at the namespace level to avoid accidental topic sprawl, making explicit admin-driven creation the standard practice.

Kafka couples compute and storage on the same broker process - each broker both serves client traffic and stores partition data on its own local disk, so scaling storage means scaling brokers too, and a broker failure requires replica reassignment and data catch-up.

Pulsar splits these roles: brokers are stateless and only serve traffic, while BookKeeper bookies durably store the actual data as replicated ledgers - this segmented architecture lets broker count (compute) and bookie count (storage) scale independently based on which resource is actually the bottleneck.

The practical payoff is that Pulsar broker failover is fast (just reassign topic ownership, no data to catch up), while Kafka's tighter coupling can make rebalancing and scaling operations more disruptive at large scale.

Brokers handle all client-facing operations - accepting publishes, dispatching to consumers, enforcing policies - but hold no message data on local disk; when a broker needs to read or write an entry, it calls out to a BookKeeper ensemble over the network.

Bookies handle only durable, replicated storage of ledger entries, with no awareness of topics, subscriptions, or pub-sub semantics at all - that logic lives entirely in the broker layer.

Because neither layer needs to understand the other's internal concerns, you can add bookies purely to grow capacity, or add brokers purely to grow request-handling capacity, and either layer can be upgraded, restarted, or rebalanced without directly disrupting the other's state.

Multi-tenancy is built directly into Pulsar's naming hierarchy (tenant to namespace to topic) rather than layered on as a convention, so isolation and policy boundaries exist at the platform level from the start.

Each tenant can have its own authentication and authorization rules, and each namespace within a tenant can carry its own retention, replication, and quota policies - letting one shared physical cluster safely host many independent teams or applications.

This differs from systems where isolation is achieved only through naming conventions or separate clusters per team, which pushes governance work onto operators instead of the platform itself.

An Exclusive subscription allows exactly one consumer at a time; a second consumer attempting to attach with the same subscription name is rejected, and the single active consumer receives every message in strict order.

A Shared subscription allows many consumers to attach simultaneously, with messages round-robined across whichever consumers are currently connected - maximizing horizontal scalability of consumption but giving up any ordering guarantee across the topic.

Exclusive suits use cases needing strict single-consumer ordering; Shared suits use cases needing maximum throughput where message order between different consumers doesn't matter.

Key_Shared allows multiple consumers on one subscription, similar to Shared, but instead of pure round-robin distribution it hashes each message's key and consistently routes all messages sharing that key to the same consumer.

Ordering is preserved per key - all events for a given order ID or user ID, for example, always arrive at one consumer in the order they were published - while different keys can still be processed in parallel across multiple consumers.

If a consumer holding certain key ranges disconnects, Pulsar reassigns those key ranges to the remaining active consumers, preserving the per-key ordering guarantee even as the consumer set changes.

Use Failover when you want the strict single-active-consumer ordering guarantee of Exclusive, but also want automatic, fast promotion of a backup consumer if the active one disconnects, without the application having to detect and reconnect manually.

With Failover, multiple consumers can register on the subscription, but only the one with the highest priority (or earliest registration, on a tie) is actively receiving messages; the rest sit as hot standbys ready to take over instantly on disconnect.

Exclusive, by contrast, simply refuses a second consumer outright, which is fine for a single-instance application but provides no built-in redundancy if that one instance goes down.

A ledger is BookKeeper's fundamental storage abstraction - an append-only sequence of entries that, once closed, is immutable; a Pulsar topic's data is really a chain of these ledgers over time, not one continuous file.

A segment, in Pulsar's terminology, generally refers to that same ledger viewed from the topic's perspective - the topic periodically "rolls over" to a new ledger (segment) based on size or time thresholds, and old segments are what tiered storage offloads once they age out.

This ledger-based structure is what makes operations like tiered storage and independent per-segment replication placement possible - each segment can, in principle, live on a different set of bookies or be offloaded to different storage entirely.

Producer-side deduplication relies on each producer being assigned a unique producer name and tagging every message with a monotonically increasing sequence ID; the broker tracks the highest sequence ID it has durably stored per producer.

If a producer retries a publish (say, after a network timeout where it's unsure whether the original request succeeded), the broker recognizes the sequence ID as already seen and discards the duplicate rather than storing it again, while still acknowledging success back to the producer.

This must be explicitly enabled per namespace or topic since it adds tracking overhead, and it protects specifically against producer-retry duplicates, not against an application publishing the same logical event twice under different sequence IDs.

Keeping brokers stateless is a deliberate architectural choice: it decouples the availability and durability of message data from the lifecycle of any individual broker process, so a broker can be restarted, replaced, or moved without any risk to stored data.

All durability responsibility is delegated to BookKeeper, whose bookies replicate every entry across an ensemble according to configured quorum settings, so data survives broker churn entirely independent of how brokers come and go.

This also simplifies broker-level operations like rolling upgrades and autoscaling, since adding or removing broker capacity is purely a compute decision with zero data rebalancing implications.

The metadata store detects the broker's session has expired (it stops renewing its ephemeral registration), and the cluster's load manager recognizes every topic the crashed broker owned now has no active owner.

Those orphaned topics are reassigned to healthy brokers based on current load; because no message data lived on the crashed broker, the new owning broker simply opens the same BookKeeper ledgers the topic was already using - there's no data recovery or replica catch-up step required.

Clients using the Pulsar client library detect the ownership change via a lookup and automatically reconnect to the new owning broker, typically resuming within seconds with no message loss for already-acknowledged data.

A namespace's set of topics is divided into fixed hash-range slices called bundles, which are the actual unit of load assignment - brokers own bundles, not individual topics directly, keeping ownership bookkeeping manageable even with millions of topics.

When a bundle grows too hot (too much throughput or too many topics concentrated in one hash range), the load manager can split it into two smaller bundles, each covering half the original hash range, which can then be assigned to different brokers.

This split happens without moving message data - it only changes how topic ownership is partitioned and assigned - so it's a lightweight rebalancing operation compared to physically relocating stored data.

Backlog quota caps how much unacknowledged data a subscription is allowed to accumulate; once the limit is hit, Pulsar applies the configured action - typically blocking producers or evicting the oldest unacked messages - to protect the cluster from an unbounded backlog caused by a stalled or slow consumer.

Retention policy, by contrast, governs already-acknowledged data - by default that data is eligible for deletion the moment every subscription has consumed it, and retention overrides that to keep it around longer anyway, for replay or late-joining subscriptions.

In short, they protect against opposite risks: backlog quota against too much unread data piling up, retention against read data disappearing too soon.

Use a Reader when you need direct, explicit control over the starting read position in a topic - for example replaying from a specific message ID or from the very beginning - without the subscription-based cursor tracking a Consumer relies on.

A Reader doesn't create a durable, named subscription that persists across restarts; it simply starts reading from wherever you tell it to (earliest, latest, or a specific message ID) each time it connects, which suits stateless replay tools or one-off inspection scripts.

A Consumer, by contrast, is the right choice for typical application workloads where you want Pulsar to durably track progress on your behalf so processing can resume exactly where it left off after a restart.

Compaction rewrites a topic's backlog into a compacted ledger that keeps only the most recent message for each distinct key, discarding older messages with the same key - similar in spirit to Kafka's log compaction, useful for topics representing "latest state per entity."

It runs as a background job (triggered manually or automatically past a configured threshold) that reads the existing topic data and writes a new compacted ledger, without disrupting normal producer/consumer traffic on the topic in the meantime.

Consumers must explicitly opt in to reading the compacted view; by default, consumers still see the full uncompacted message history unless they request the compacted alternative.

Ensemble size (E) is the total number of bookies eligible to hold entries for a given ledger, while write quorum (Qw) is how many of those bookies each individual entry actually gets written to - Qw is always less than or equal to E.

Separating the two lets an operator stripe writes for a single ledger across more bookies than any one entry needs (for example E=5 with Qw=3), spreading load and improving read parallelism, without requiring every bookie in the ensemble to store every entry.

Ack quorum (Qa), a third related setting, is how many of the Qw bookies must confirm a write before it's considered durable - so a typical configuration like E=3, Qw=2, Qa=2 means each entry lands on 2 of 3 possible bookies and needs both to acknowledge before the write returns success.

Persistent topics durably store every message in BookKeeper before acknowledging the publish, so data survives broker restarts, and slow consumers can catch up later from the retained backlog.

Non-persistent topics keep messages only in broker memory and never touch BookKeeper, trading that durability for lower publish latency and zero storage overhead - a broker restart loses any in-flight messages with no way to recover them.

Because of this, non-persistent topics suit only data where occasional loss is acceptable in exchange for speed, while persistent topics are the default choice for anything that must not silently disappear.

A producer can mark an individual message with a delay (a duration or an absolute delivery time), and the broker holds that message out of normal dispatch until the delay elapses, at which point it becomes visible to consumers like any other message.

Internally, delayed messages are tracked in a time-indexed structure the broker consults on each dispatch cycle, checking whether any delayed messages have now become due, rather than the message sitting unusably in the ledger until manually re-checked by a client.

This is commonly used for scheduling retries with backoff, deferred notifications, or workflow steps that need to fire at a specific future time, without needing an external scheduler service.

A negative acknowledgment (nack) tells the broker that this specific consumer failed to process the message and it should be redelivered - typically after a configurable redelivery delay - rather than being treated as successfully handled.

Unlike an acknowledgment timeout (which waits for a consumer to go silent for a period before assuming failure), a nack is an explicit, immediate signal from the consumer's own processing logic that something went wrong with that particular message.

Depending on the subscription type, the redelivered message may go to the same consumer or a different one, and if it keeps failing repeatedly, a configured dead letter policy can route it to a separate dead letter topic instead of endlessly retrying.

A producer connects to the broker currently owning the target topic (discovered via a lookup against the metadata store) and sends a message, optionally batched with others and assigned a sequence ID for deduplication.

The owning broker validates the message against the topic's configured schema, applies routing (for partitioned topics), and forwards the entry to a BookKeeper ensemble, waiting for the configured ack quorum of bookies to durably confirm the write before acknowledging success back to the producer.

Once durably stored, the message becomes available to every active subscription on the topic; the broker dispatches it according to each subscription's type, pushing it to the appropriate connected consumer(s).

A consumer processes the message and sends an acknowledgment back to the broker, which advances that subscription's cursor position (itself persisted, via a managed ledger) so the message is no longer part of that subscription's backlog; if left unacknowledged, it remains eligible for redelivery.

Once every subscription on the topic has acknowledged a message and the configured retention window (if any) allows it, the message becomes eligible for deletion from BookKeeper, or for offload to tiered storage if it's aged past the offload threshold instead.

sequenceDiagram
  participant P as Producer
  participant B as Broker
  participant BK as BookKeeper ensemble
  participant C as Consumer
  P->>B: Publish message (lookup + send)
  B->>BK: Write entry to ledger
  BK-->>B: Ack quorum confirms durable write
  B-->>P: Publish acknowledged
  B->>C: Dispatch per subscription type
  C-->>B: Acknowledge
  B->>B: Advance subscription cursor

Enable producer-side batching and compression so many small messages are grouped and compressed into fewer, larger network writes and BookKeeper entries, dramatically reducing per-message overhead at high publish rates.

Use partitioned topics so a single logical topic's load spreads across multiple brokers, and correspondingly potentially different bookies, instead of bottlenecking on one broker's CPU/network and one ledger's write path.

Tune BookKeeper's ensemble/write-quorum/ack-quorum settings deliberately: a smaller write quorum reduces the number of bookies each entry waits on, lowering write latency, at the cost of some durability margin - the right balance depends on how much replication risk is acceptable.

On the consumption side, prefer Shared or Key_Shared subscriptions with multiple parallel consumer instances over a single Exclusive consumer wherever strict global ordering isn't required, since that's what actually lets consumption scale horizontally alongside publish throughput.

Finally, size broker and bookie hardware appropriately - especially bookie journal/ledger disks, ideally on separate fast disks such as NVMe/SSD - since BookKeeper's write path is disk-latency sensitive and undersized storage is a common throughput ceiling in practice.

Start by identifying whether the backlog growth is on a specific subscription or across the whole topic - per-subscription backlog metrics will show whether one consumer group is falling behind while others keep pace, pointing at a consumer-side problem rather than a producer or broker issue.

Check whether consumers are actually connected and actively acknowledging: a subscription with zero connected consumers, or consumers stuck failing to process messages and repeatedly nacking or timing out, will show backlog climbing steadily with no processing happening at all.

If consumers are connected but processing slowly, compare consumption rate against publish rate directly - a sustained mismatch means you need more parallel consumers (on Shared/Key_Shared) or need to speed up per-message processing, not just wait it out.

Rule out downstream issues in consumer application logic itself, such as a slow external call or a poison-pill message causing repeated failures and redelivery loops, by checking consumer-side logs and negative-acknowledgment/redelivery counts - a backlog can also be a symptom of one bad message stuck in a redelivery loop rather than genuinely high volume.

If backlog quota is configured, also confirm it hasn't already been hit and started blocking producers or evicting messages, which changes the nature of the problem from "processing is behind" to "data is actively being lost or refused."

Every broker maintains a live session with the metadata store using a short-lived, renewable registration; as long as the broker is healthy, it keeps renewing that session, implicitly proving it's still alive and reachable.

When a broker crashes or becomes network-partitioned, it stops renewing its session, and after the session timeout elapses, the metadata store expires it - this is the trigger the rest of the cluster watches for, not any direct crash notification from the failed broker itself.

The cluster's load manager, running as logic within the surviving brokers, observes the expired session, determines which bundles the failed broker owned, and reassigns each one to a healthy broker based on current load - no data is copied because it already lives durably in BookKeeper, independent of which broker owned it.

The newly assigned broker opens the relevant managed ledgers (topic data) and cursors (subscription positions) directly from BookKeeper metadata, resuming service from exactly where the previous owner left off.

Clients performing a topic lookup during or after this window get redirected to the new owning broker automatically by the Pulsar client library's built-in retry/backoff and lookup-refresh logic, so from the application's perspective, service resumes with a brief pause rather than requiring manual reconnection logic.

flowchart TD
  A[Broker session stops renewing] --> B[Metadata store session expires]
  B --> C[Load manager detects orphaned bundles]
  C --> D[Reassign bundles to healthy broker]
  D --> E[New broker opens existing ledgers/cursors from BookKeeper]
  E --> F[Clients redirected via lookup on reconnect]

Configure an offload threshold (by size or age) at the namespace or topic level so ledger segments older than the threshold are automatically moved from replicated BookKeeper disk to cheaper object storage like S3, without manual intervention once set up.

Choose the threshold deliberately based on actual access patterns: frequently read "hot" data should stay on BookKeeper's low-latency disks, while data rarely accessed except for occasional replay or compliance should be pushed to offload as early as reasonably possible to minimize the replicated-disk footprint.

Because offloaded segments remain transparently readable through the normal topic/consumer APIs, you can safely set aggressive (short) offload thresholds without breaking any existing consumer or replay workflow - the only cost is somewhat higher read latency the rare times older data actually gets accessed.

Combine this with a sensible retention policy: for data you don't need indefinitely, letting it expire and delete outright, rather than retaining and offloading forever, is cheaper still than paying even discounted object storage costs for data nobody will ever read again.

A Pulsar transaction begins when a client requests a transaction ID from a dedicated internal component, the Transaction Coordinator, which tracks the transaction's state (open, committing, committed, aborting, aborted) throughout its life.

As the client produces messages and acknowledges consumed messages within that transaction, those operations are recorded as "pending" against the relevant topics and subscriptions - visible internally, but not yet visible to other consumers reading normally, and not yet permanently marking source messages as acknowledged.

On commit, the Transaction Coordinator writes a commit marker and drives all participating topics/subscriptions to make the pending produces visible and the pending acknowledgments permanent atomically from the consumer's perspective, even though the operations may span multiple partitions or topics.

On abort, or a coordinator/client failure before commit, the pending produced messages are marked as effectively invisible to consumers and any pending acknowledgments are rolled back, so those source messages return to being unacknowledged and eligible for normal redelivery.

This is what allows exactly-once-style, multi-topic atomic operations, like consuming from topic A and producing to topic B as one unit, that plain per-message deduplication alone can't provide, since deduplication only protects a single producer's own retries, not multi-step consume-then-produce atomicity.

They solve different, narrower problems, so "better" depends on what's actually being protected - it isn't really a straight substitute-for-substitute comparison.

Idempotent, deduplicating producers protect against a single producer accidentally publishing the same message twice due to its own retries, for example after an ambiguous network timeout; they're cheap, low-overhead, and sufficient when the risk is purely "did my publish get double-sent."

Transactions protect a broader case: atomicity across multiple operations, especially the common consume-process-produce pattern spanning different topics or partitions, ensuring the whole set of produces and acknowledgments happens or none of it does - something producer-side deduplication alone cannot guarantee, since it has no awareness of related acknowledgments elsewhere.

In practice, use plain deduplication for simple single-topic publish reliability where overhead should stay minimal, and reach for transactions specifically when a processing step must atomically span multiple topics or combine consuming and producing into one all-or-nothing unit - reserving the extra coordination overhead for the cases that actually need it.

Check the cluster's current bundle-to-broker assignment and per-broker resource metrics (CPU, throughput, connection count) via pulsar-admin or a monitoring dashboard to confirm the imbalance is real and see which brokers are over- or under-loaded.

A common cause is a small number of very hot bundles sitting on the same broker; check whether automatic bundle splitting is enabled and whether its thresholds are appropriately tuned for the actual traffic pattern - overly conservative thresholds delay splitting hot bundles.

Confirm the load manager's shedding configuration is active - Pulsar's load manager can actively unload bundles from overloaded brokers onto less-loaded ones, and if this is disabled or too conservative, imbalance can persist even after hot bundles are correctly split.

Check for "sticky" traffic from non-partitioned topics: a single very high-traffic non-partitioned topic is inherently confined to one broker, since it can't be split across brokers the way a partitioned topic can - if that's the imbalance source, converting it to a partitioned topic is the actual fix, not load-manager tuning.

Finally, verify client-side connection distribution isn't itself skewed, for example a proxy or DNS layer routing disproportionately to a subset of brokers, since load imbalance can sometimes originate outside Pulsar's own load manager entirely.

When a namespace is created, its topic hash range is divided into an initial, configurable number of bundles - each owning an equal slice of the hash space that any topic in the namespace could fall into based on a hash of its name.

As topics are created and traffic flows, each bundle gets ownership assigned to exactly one broker at a time, tracked via the metadata store, and that broker handles all pub-sub traffic for every topic whose name hashes into that bundle's range.

The load manager continuously monitors per-bundle load signals - throughput, topic count, resource usage; when a bundle's load crosses a configured threshold, it's flagged as a split candidate rather than being split reflexively on any brief spike.

On split, the bundle's hash range divides into two new, smaller-range bundles, each initially owned by the same broker as before to avoid a disruptive double-move, and the load manager can subsequently reassign one of the two to a different, less-loaded broker as a separate rebalancing step.

This split only changes routing metadata, which broker is responsible for which hash range, and involves no movement of underlying message data in BookKeeper, since bundles are purely a load-management and ownership-assignment construct layered on top of storage that was already independent of any broker.

flowchart LR
  A[Namespace created] --> B[Initial bundles: equal hash-range slices]
  B --> C[Broker assigned ownership per bundle]
  C --> D{Load manager: threshold crossed?}
  D -- No --> C
  D -- Yes --> E[Split bundle into two smaller ranges]
  E --> F[Both initially owned by same broker]
  F --> G[Optional: reassign one to a less-loaded broker]

Ensure message keys have enough distinct values and reasonably even distribution across them - if most messages share one or a few keys, that traffic still funnels to a single consumer each, capping parallelism regardless of how many consumer instances are added.

Scale the number of consumer instances to roughly match, or stay below, the practical number of distinct, active keys at any given time, since adding more consumers than there are actively-flowing keys just leaves extra consumers idle rather than increasing throughput.

Tune receiver queue size and permits per consumer appropriately - too small a queue can throttle a fast consumer waiting on network round-trips for more messages, while an oversized queue on a slow consumer can cause it to hold an unfair share of in-flight, unacknowledged key ranges.

Watch for consumer churn: every time a consumer joins or leaves a Key_Shared subscription, key-range ownership is recomputed, briefly pausing processing for the reassigned ranges - so a subscription with frequent connect/disconnect cycles, from crash-looping or aggressive autoscaling, sees lower effective throughput than the same consumer count running stably.

The function's instance(s) subscribe to the configured input topic(s) as a regular Pulsar consumer under the hood, receiving each message through the normal subscription and dispatch mechanism just like any other consumer application.

For each received message, the Pulsar Functions runtime invokes the user-provided process method with the message's deserialized value (using the topic's configured schema) plus a context object exposing metadata like message properties, topic name, and access to any configured state store.

Whatever the function returns is published to the configured output topic using the function's own internal producer, with the same durability guarantees as any other Pulsar publish - and by default, only after that output publish succeeds does the function acknowledge the original input message, giving Functions at-least-once (and, in transactional mode, atomic) processing semantics.

If the function needs to maintain state across invocations, like a running count, it can read and write to Pulsar's built-in state store, itself backed by BookKeeper's table service, without the operator needing to stand up a separate external database.

The Functions runtime handles scaling instances, restarting failed instances, and redistributing partitions among instances automatically, so operationally a function behaves like any other managed Pulsar workload rather than a bespoke deployed service.

flowchart LR
  A[Input topic] --> B[Function consumer receives message]
  B --> C[Runtime invokes process with value + context]
  C --> D[Optional: read/write state store]
  C --> E[Publish result to output topic]
  E --> F[Acknowledge input message]

First confirm whether producer-side deduplication is actually enabled on the namespace or topic - it's off by default in many configurations, so what looks like a "bug" causing duplicates may simply be expected behavior from retried publishes with dedup disabled.

If dedup is enabled but duplicates still appear, check whether the application is creating a new producer instance, with an effectively different producer name each time, on every retry rather than reusing one stable producer - deduplication tracks state per producer name, so a fresh producer name defeats it even with the feature turned on.

Check the deduplication window/snapshot interval configuration - the broker only remembers recent sequence IDs for a bounded time or entry count, and a retry arriving long after that window has rolled off won't be recognized as a duplicate even with dedup correctly enabled and the producer name unchanged.

Distinguish producer-level duplicates from application-level duplicates: if the same logical event is published twice under two legitimately new sequence IDs, because application-layer retry logic sits above the Pulsar client and calls publish twice, no amount of Pulsar-side deduplication will catch that - the fix belongs in the application's own idempotency handling, not Pulsar configuration.

Finally, check consumer-side handling isn't the actual source: a message redelivered after a nack or ack timeout is a different failure than true producer-side duplication, and easy to mistake for one if consumer logs aren't inspected for negative-acknowledgment or timeout events alongside the "duplicate" message IDs.