Versions

In Kafka 2.3.0-1.1.0 and later, these topics have been divided up among the Getting Started and Security sections. Previous versions will still need the information below.

Configuring DC/OS Access

This topic describes how to configure DC/OS access for Kafka. Depending on your security mode, Kafka requires service authentication for access to DC/OS.

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

• DC/OS CLI installed and be logged in as a superuser.
• Enterprise DC/OS CLI 0.4.14 or later installed.
• If your security mode is permissive or strict, you must get the root cert before issuing the curl commands in this section.

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created uses the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

Tip: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (<service-account-id>) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d "Kafka service account" <service-account-id>

Tip: You can verify your new service account using the following command.

dcos security org service-accounts show <service-account-id>

Create a Secret

Create a secret (kafka/<secret-name>) with your service account (<service-account-id>) and private key specified (<private-key>.pem).

Tip: If you store your secret in a path that matches the service name (e.g. service name and secret path are kafka), then only the service named kafka can access it.

Permissive

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> kafka/<secret-name>

Strict

dcos security secrets create-sa-secret --strict <private-key>.pem <service-account-id> kafka/<secret-name>

Tip: You can list the secrets with this command:

dcos security secrets list /

Create and Assign Permissions

Use the following curl commands to rapidly provision the Kafka service account with the required permissions.

Tips:

• Any / character in a resource must be replaced with %252F before it can be passed in a curl command.
• When using the API to manage permissions, you must first create the permissions and then assign them. Sometimes, the permission may already exist. In this case, the API returns an informative message. You can regard this as a confirmation and continue to the next command.

1. Create the permission.

   Important: These commands use the default Kafka role value of kafka-role. If you’re running multiple instances of Kafka, replace the instances of kafka-role with the correct name (<name>-role). For example, if you have a Kafka instance named kafka2, you would replace each role value in the code samples to kafka2-role.

   Permissive

   Run these commands with your service account name (<service-account-id>) specified.

   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody \
   -d '{"description":"Allows Linux user nobody to execute tasks"}' -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:kafka-role \
   -d '{"description":"Controls the ability of kafka-role to register as a framework with the Mesos master"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:kafka-role \
   -d '{"description":"Controls the ability of kafka-role to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:kafka-role \
   -d '{"description":"Controls the ability of kafka-role to access volumes"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
   -H 'Content-Type: application/json'
   

   Strict

   Run these commands with your service account name (<service-account-id>) specified.

   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:kafka-role \
   -d '{"description":"Controls the ability of kafka-role to register as a framework with the Mesos master"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:kafka-role \
   -d '{"description":"Controls the ability of kafka-role to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:kafka-role \
   -d '{"description":"Controls the ability of kafka-role to access volumes"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
   -H 'Content-Type: application/json'
   

2. Grant the permissions and the allowed actions to the service account using the following commands.

   Run these commands with your service account name (<service-account-id>) specified.

   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:kafka-role/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:kafka-role/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:kafka-role/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id>/users/<service-account-id>/delete
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id>/users/<service-account-id>/delete
   

Create a config.json file

Create a custom configuration file that will be used to install Kafka and save as config.json.

Specify the service account (<service-account-id>) and secret path (kafka/<secret-name>).

{
  "service": {
    "service_account": "<service-account-id>",
    "service_account_secret": "kafka/<secret-name>"
  }
}

Install Kafka

Install Kafka with this command:

dcos package install --options=config.json beta-kafka