...
| Code Block |
|---|
## Creating new Topics
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--create \
--topic kafka.learning.tweets \
--partitions 1 \
--replication-factor 1
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--create \
--topic kafka.learning.alerts \
--partitions 1 \
--replication-factor 1
## Listing Topics
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--list
## Getting details about a Topic
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--describe
## Publishing Messages to Topics
./kafka-console-producer.sh \
--bootstrap-server localhost:29092 \
--topic kafka.learning.tweets
## Consuming Messages from Topics
./kafka-console-consumer.sh \
--bootstrap-server localhost:29092 \
--topic kafka.learning.tweets \
--from-beginning
## Deleting Topics
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--delete \
--topic kafka.learning.alerts
#Create a Topic with multiple partitions
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--create \
--topic kafka.learning.orders \
--partitions 3 \
--replication-factor 1
#Check topic partitioning
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--topic kafka.learning.orders \
--describe
## Publishing Messages to Topics with keys
./kafka-console-producer.sh \
--bootstrap-server localhost:29092 \
--property "parse.key=true" \
--property "key.separator=:" \
--topic kafka.learning.orders
## Consume messages using a consumer group
./kafka-console-consumer.sh \
--bootstrap-server localhost:29092 \
--topic kafka.learning.orders \
--group test-consumer-group \
--property print.key=true \
--property key.separator=" = " \
--from-beginning
## Check current status of offsets
./kafka-consumer-groups.sh \
--bootstrap-server localhost:29092 \
--describe \
--all-groups
## Creating the Topic
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--create \
--topic kafka.usecase.students \
--partitions 2 \
--replication-factor 1
## Describe the Topic
./kafka-topics.sh \
--zookeeper zookeeper:2181 \
--topic kafka.usecase.students \
--describe
## Publish to the Topic
./kafka-console-producer.sh \
--bootstrap-server localhost:29092 \
--property "parse.key=true" \
--property "key.separator=:" \
--topic kafka.usecase.students
## Consume Message from the Topic
./kafka-console-consumer.sh \
--bootstrap-server localhost:29092 \
--topic kafka.usecase.students \
--group usecase-consumer-group \
--property print.key=true \
--property key.separator=" = " \
--from-beginning |
REST Api
Pushing Message to Topic
Publish a Message
| Code Block | ||||
|---|---|---|---|---|
| ||||
curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" \
--data ' { "records": [ { "value": {"log":"value"} }]}' \
"http://localhost:8082/topics/ncyd_test_in" |
| Code Block |
|---|
{"offsets":[{"partition":0,"offset":0,"error_code":null,"error":null}],"key_schema_id":null,"value_schema_id":null}% |
Consuming Messages from a Topic
Create a Consumer
| Code Block | ||||
|---|---|---|---|---|
| ||||
curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" \
--data '{"name": "test_consumer", "format": "json", "auto.offset.reset": "earliest"}' \
http://localhost:8082/consumers/test_consumer |
| Code Block |
|---|
{"instance_id":"test_consumer","base_uri":"http://ckaf-rest-0.ckaf-rest-headless.default.svc.cluster.local:8082/consumers/test_consumer/instances/test_consumer"}% |
Subscribe to a Topic
| Code Block | ||||
|---|---|---|---|---|
| ||||
curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" \
--data '{"topics":["ncyd_test_in"]}' \
http://localhost:8082/consumers/test_consumer/instances/test_consumer/subscription |
Consume the Messages
| Code Block | ||||
|---|---|---|---|---|
| ||||
curl -X GET -H "Accept: application/vnd.kafka.json.v2+json" \
http://localhost:8082/consumers/test_consumer/instances/test_consumer/records |
| Code Block |
|---|
[{"topic":"ncyd_test_in","key":null,"value":{"log":"value"},"partition":0,"offset":0}]% |
References
| Reference | URL |
|---|---|
| Apache Kafka in 5 minutes | https://www.youtube.com/watch?v=PzPXRmVHMxI |
| Nokia Learning - Kafka | https://nokialearn.csod.com/ui/lms-learning-details/app/course/dc425d19-5642-535f-916c-211768f90a00 |
| Kafka Topics, Partitions and Offsets Explained | https://www.youtube.com/watch?v=_q1IjK5jjyU |
| Kafka Helm Charts | https://github.com/confluentinc/cp-helm-charts |
Confluent for Kubernetes | https://docs.confluent.io/operator/current/overview.html#operator-about-intro |
...