The set of partitions described by the topic subscription is distributed across all the members of a consumer group (generally scaled replicas of the same application). This can be represented as a single topic, multiple topics, or a set of individual topic partitions, that we want to consume from. One of the required parameters needed to setup a consumer is a topic subscription. However, committing offsets to Kafka for simple streaming applications, or in addition to managing offsets yourself, enables an easy way to track the progress of all partitions being consumed in a consumer group. HDFS, Ceph, etc.) to enable stateful streaming workloads in a fault tolerant manner.Ĭustom user applications may want to perform their own offset management so they can easily replay a partition’s messages from different positions in the log. Stream processing frameworks like Spark and Flink will perform offset management internally on fault tolerant distributed block storage (i.e. The main purpose behind committing is to provide an easy way for applications to manage their current position in a partition so that if a consumer group member stops for any reason (error, consumer group rebalance, graceful shutdown) that it can resume from the last committed offset (+1) when it’s active again.Ĭommitting offsets to Kafka is not strictly necessary to maintain consumer group position–you may also choose to store offsets yourself.
Member applications of a consumer group may commit offsets to Kafka to indicate that they’ve been successfully processed (at-least-once semantics) or successfully received (at-most-once semantics). Distribute the consumption of messages across 1 or more consumer group members.At a high level, they allow us to do the following. When consuming messages from Kafka it is common practice to use a consumer group, which offer a number of features that make it easier to scale up/out streaming applications.
Kafka consumer group lag is one of the most important metrics to monitor on a data streaming platform. This project was started to facilitate an easy way to discover consumer group lag & latency of Akka Streams and Spark streamlets in Lightbend Pipelines, but more generally it can report consumer group metrics of any Kafka application that commits offsets back to Kafka.īefore discussing Kafka Lag Exporter’s features, it’s important to have an understanding of Kafka consumer group lag. We’ve helped many of our clients to run high throughput, low latency data streaming applications on Lightbend Platform, and understanding consumer group lag is critical to ensuring low latency processing.
Lightbend has spent a lot of time working with Apache Kafka on Kubernetes. Kafka Lag Exporter can run anywhere, but it provides features to run easily on Kubernetes clusters against Strimzi Kafka clusters using the Prometheus and Grafana monitoring stack. Introducing Kafka Lag Exporter, a tool to make it easy to view consumer group metrics using Kubernetes, Prometheus, and Grafana.
0 kommentar(er)
