Table of Contents |
---|
Overview
Apache Kafka is an open source project for a distributed publish-subscribe messaging system rethought as a distributed commit log.
...
Created by LinkedIn and is now an Open Source project maintained by Confluent.
Kafka Use Cases
Some use cases for using Kafka:
- Messaging System
- Activity Tracking
- Gathering metrics from many different sources
- Application Logs gathering
- Stream processing (with the Kafka Streams API or Spark for example)
- De-coupling of system dependencies
- Integration with Spark, Flink, Storm, Hadoop and many other Big Data technologies
Architecture
- Source Connectors pull data from sources
- Data is sent to Kafka cluster
- Transformation of topic data into another topic can be done with Streams
- Sink Connectors in Connect cluster pull data from Kafka
- Sink Connectors push data to sinks
Kafka
Topics and Partitions
Topics: a particular stream of data
- similar to a table in a database(without constraints)
- you can have as many topics as you want
- a topic is identified by it's name
Topics are split into partitions
- each partition is ordered
- each message with a partition gets an incremental id, called offset.
- offsets are only relevant for a particular partition
- order is guaranteed only in a partition (not across partitions)
- data is assigned to a random partition unless a key is provided
- you can have as many partitions per topic as you want
- specifying a key, ensures that your message is written to the same partition (which ensures order).
Installation on Kubernetes
Installing Kafka Cluster
We are using the bitnami helm chart:
...
Code Block |
---|
$ kubectl get pods NAME READY STATUS RESTARTS AGE kafka-0 1/1 Running 3 3h9m kafka-zookeeper-0 1/1 Running 0 3h9m |
Installing Kafka Connect Cluster
...
References
Reference | URL |
---|---|
Apache Kafka in 5 minutes | https://www.youtube.com/watch?v=PzPXRmVHMxI |
Kafka Topics, Partitions and Offsets Explained | https://www.youtube.com/watch?v=_q1IjK5jjyU |
Kafka Helm Chart | https://bitnami.com/stack/kafka/helm |
Kafka Helm Chart Source | https://github.com/bitnami/charts/tree/master/bitnami/kafka/#installing-the-chart |
...