#2 This is how Kafka Partitioning works

#2 This is how Kafka Partitioning works

Kafka Partitions: You can configure the number of partitions for a Kafka topic. Inside one partition the order of the records is guaranteed, but ordering is not guaranteed across partitions. Kafka will put records with the same key in the same partition, when no valid partition is specified from the producer.

The producer is responsible for choosing which record to assign to which partition within the topic

 Kafka Consumer distribution to Partitions: The maximum number of consumers in one consumer group equals the number of partitions. Each partition is consumed by exactly one consumer per consumer group but each consumer can consume more than one partition. Kafka will automatically distribute the partitions to all consumers in one group, if one dies it will reassign the partitions to another consumer.

Kafka Partition distribution on Consumers of Consumer groups of different sizes

Replication of topics: Each partition is replicated on a configured number of servers. One of them is the leader and handles all incoming requests to the partition. The others (followers) passively duplicate the data. Writing is only considered as complete once replication is complete. This way Kafka ensures no data loss if the leader fails.

Further reading:

http://kafka.apache.org/documentation/#intro_distribution

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert