...
| Connector | Type | URL |
|---|---|---|
| Azure | Kafka Sink Connector | https://www.confluent.de/hub/chaitalisagesh/kafka-connect-log-analytics |
| Fluent | Output Plugin for fluentbit | https://github.com/fluent/fluent-plugin-kafka |
| Elasticsearch | Kafka Source Connector | https://www.confluent.io/hub/dariobalinzo/kafka-connect-elasticsearch-source |
| Elasticsearch | Kafka Sink Connector | https://github.com/confluentinc/kafka-connect-elasticsearch |
| *Fluent | Output Plugin for fluentbit | https://docs.fluentbit.io/manual/pipeline/outputs/kafka |
Functions
These call can all be made from the kafka-connect pod
Login to the kafka-connect pod
| Code Block |
|---|
kubectl exec -it kafka-cp-kafka-connect-<ID> -c cp-kafka-connect-server bash |
List Connectors
| Code Block |
|---|
$ curl -s -X GET -H "Content-Type: application/json" http://kafka-cp-kafka-connect:8083/connectors
["azure-sink-connector"] |
Get connector details
| Code Block |
|---|
$ curl -s -X GET -H "Content-Type: application/json" http://kafka-cp-kafka-connect:8083/connectors/azure-sink-connector
{
"name": "azure-sink-connector",
"config": {
"connector.class": "io.kafka.connect.log.anlaytics.sink.LogAnalyticsSinkConnector",
"workspace.id": "7e0d2c8e-a46c-4fd9-b274-4b07f0ba555c",
"topics": "john-test",
"value.converter.schemas.enable": "false",
"name": "azure-sink-connector",
"workspace.key": "y3n6lvRaKhDIaV6UuGn6+nuh/BoRsQI0fy9S13ZdrL/w56LUOuqrRK3ajAAnxjo8W4PAzxId0V09bJWxmtrNLA==",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.storage.StringConverter"
},
"tasks": [
{
"connector": "azure-sink-connector",
"task": 0
}
],
"type": "sink"
} |
Delete a Connector
| Code Block |
|---|
curl -s -X DELETE -H "Content-Type: application/json" http://kafka-cp-kafka-connect:8083/connectors/azure-sink-connector |
Send Message to a topic
| Code Block |
|---|
echo '{"test": 213}'| kafka-console-producer --broker-list kafka-cp-kafka-headless:9092 --topic john-test |
References
| Reference | URL |
|---|---|
| Apache Kafka in 5 minutes | https://www.youtube.com/watch?v=PzPXRmVHMxI |
| Nokia Learning - Kafka | https://nokialearn.csod.com/ui/lms-learning-details/app/course/dc425d19-5642-535f-916c-211768f90a00 |
| Kafka Topics, Partitions and Offsets Explained | https://www.youtube.com/watch?v=_q1IjK5jjyU |
| Kafka Helm Charts | https://github.com/confluentinc/cp-helm-charts |
Confluent for Kubernetes | https://docs.confluent.io/operator/current/overview.html#operator-about-intro |
...