Kafka

Deploying Kafka in a project

Getting started

To get started with creating and managing a Kafka Cluster in a project, you first need to deploy the Kafka operator and the ZooKeeper operator in the workspace where the project exists.

After deploying the Kafka operator, create Kafka Clusters by applying a KafkaCluster custom resource on each attached cluster in a project’s namespace. Refer to the Kafka operator repository for examples of the custom resources and their configurations.

Example deployment

NOTE: If you need to manage these custom resources across all clusters in a project, it is recommended you use Project Deployments which enables you to leverage GitOps to deploy the resources. Otherwise, you will need to create the custom resources manually in each cluster.

This example deployment walks you through first deploying a ZooKeeper cluster and then a Kafka cluster in a project namespace. The result of this procedure is a running ZooKeeper cluster and Kafka cluster ready for use in your project’s namespace.

  1. Set the PROJECT_NAMESPACE environment variable to the name of your project’s namespace:

    export PROJECT_NAMESPACE=<project namespace>

  2. Create a ZooKeeper Cluster custom resource in your project’s namespace:

    kubectl apply -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: zookeeper
  namespace: ${PROJECT_NAMESPACE}
spec:
  replicas: 1
EOF

  3. Check the status of your ZooKeeper cluster using kubectl:

    kubectl -n ${PROJECT_NAMESPACE} get zookeeperclusters

  4. Download the sample Kafka Cluster file.

  5. Update the following attribute zkAddresses, replacing zookeeper-client.zookeeper:2181 with zookeeper-client.<project namespace>:2181. You can use sed to update the file:

    • MacOS sed

      # If you are using sed that comes installed on macOS
sed -i '' "s/zookeeper-client.zookeeper:2181/zookeeper-client.${PROJECT_NAMESPACE}:2181/g" simplekafkacluster.yaml

    • GNU sed

      # If you are using GNU sed
sed -i "s/zookeeper-client.zookeeper:2181/zookeeper-client.${PROJECT_NAMESPACE}:2181/g" simplekafkacluster.yaml

  6. Verify the file contains the following line:

    ...
zkAddresses:
    - “zookeeper-client.<project namespace>:2181”
...

  7. Apply the KafkaCluster to your project’s namespace:

    kubectl apply -n ${PROJECT_NAMESPACE} -f simplekafkacluster.yaml

  8. Check the status of your Kafka cluster using kubectl:

    kubectl -n ${PROJECT_NAMESPACE} get kafkaclusters

With both the ZooKeeper cluster and Kafka cluster running in your project’s namespace, refer to the Kafka Operator documentation for information on how to test and verify they are working as expected in. When performing those steps, ensure you substitute: zookeeper-client.<project namespace>:2181 anywhere that the zookeeper client address is mentioned.

Deleting Kafka custom resources

Follow these steps to delete the Kafka custom resources.

  1. View all Kafka resources in the cluster:

    kubectl get kafkaclusters -A
kubectl get kafkausers -A
Kubectl get kafkatopics -A

  2. Delete a KafkaCluster example:

    kubectl -n ${PROJECT_NAMESPACE} delete kafkacluster <name of KafkaCluster>

Resources