Updating Confluent Kafka

Enterprise DC/OS 1.10 and later

Enterprise DC/OS 1.10 introduced a convenient command line option that allows for easier updates to a service’s configuration and version, as well as allowing users to inspect the status of an update, to pause and resume updates, and to restart or complete steps if necessary.

In Enterprise DC/OS 1.11 and later, configuration updates may be done from the UI as well as from the CLI, but managing the phases and steps of an update must be done from the CLI. Service version updates must always be done from the CLI.

Prerequisites

Enterprise DC/OS 1.10 or later
A DC/OS SDK-based Service with a version later than 2.0.0-x
The DC/OS CLI installed and available
The service’s subcommand available and installed on your local machine
- You can install just the subcommand CLI by running dcos package install --cli confluent-kafka.
- If you are running an older version of the subcommand CLI that does not have the update command, uninstall and reinstall your CLI.
```
dcos package uninstall --cli confluent-kafka
dcos package install --cli confluent-kafka
```

Updating service version

Viewing available versions

The update package-versions command allows you to view the versions of the Confluent Kafka service that you can upgrade or downgrade to.

The CLI command is:

dcos confluent-kafka update package-versions

The result will be of the form:

Current package version is: "1.0.1-1.2.3"
Package can be downgraded to: ["1.0.0-1.2.3"]
Package can be upgraded to: ["1.0.2-1.2.3"]

Upgrading or downgrading a service

Before updating the service itself, update its CLI subcommand to the new version:

dcos package uninstall --cli confluent-kafka
dcos package install --cli confluent-kafka --package-version="<version>"

After the CLI subcommand has been updated, call the update start command, passing in the version

dcos confluent-kafka update start --package-version="<version>"

If you are missing mandatory configuration parameters, the update command will return an error.

To supply missing configuration values or to override configuration values, you can also provide an options.json file (see Updating configuration below):

dcos confluent-kafka update start --options=options.json --package-version="<version>"

The default behavior on update is to merge ‘Default’, ‘Stored’ and ‘Provided’ configurations, in that order, and then validate against the schema. In some situations, such as when a schema option has been removed, the default behavior might result in an invalid configuration. You can work around this with --replace=true which, when specified, will override the ‘Stored’ options with the ‘Provided’ options.

dcos confluent-kafka update start --options=options.json --replace=true --package-verion="<version>"

See Advanced update actions for commands you can use to inspect and manipulate an update after it has started.

Updating configuration

Preparing configuration

If you installed the service with Enterprise DC/OS 1.10, you can fetch the full configuration of a service (including any default values that were applied during installation).

dcos confluent-kafka describe > options.json

Make any configuration changes to this options.json file.

Updating after DC/OS 1.9 to DC/OS 1.10 Upgrade

If the service was previously installed on DC/OS 1.9, and the cluster has now been upgraded to DC/OS 1.10, it will not have the necessary metadata for the update commands to succeed. You will need to perform an initial update to create the metadata.

Using the version controlled options file of the service, start a configuration update:

dcos confluent-kafka update start --options=<options-file>

This will update the service metadata with the correct stored options. The service scheduler will be restarted, but service tasks will not. Going forward, you can use the update commands as described elsewhere.

WARNING: It is critical to do the update with the exact options the service is currently running with in this upgrade step. Any option not specified in the options file will be lost and the default used.

Starting the update

Once you are ready to begin, initiate an update using the DC/OS CLI, passing in the updated options.json file:

dcos confluent-kafka update start --options=options.json

You will receive an acknowledgement message and the DC/OS package manager will restart the service scheduler which will proceed to update the service.

See Advanced update actions for commands you can use to inspect and manipulate an update after it has started.

Advanced update actions

The following sections describe advanced commands that may be used to interact with an update in progress.

Monitoring the update

Once the service scheduler has been restarted, it will begin a new deployment plan as individual pods are restarted with the new configuration. Depending on the high availability characteristics of the service being updated, you may experience a service disruption.

You can query the status of the update as follows:

dcos confluent-kafka update status

If the service scheduler is still restarting, DC/OS will not be able to route to it and this command will return an error message. Wait a short while and try again. You can also go to the Services tab of the DC/OS GUI to check the status of the restart.

Pause

To pause an ongoing update, issue a pause command:

dcos confluent-kafka update pause

You will receive an error message if the plan has already completed or has already been paused. Once completed, the update will enter the WAITING state.

Resume

If the update is in a WAITING state, as a result of being paused or reaching a breakpoint that requires manual operator verification, you can use the resume command to continue the update:

dcos confluent-kafka update resume

You will receive an error message if you attempt to resume an update that is already in progress or has already completed.

Force Complete

In order to manually “complete” a step (such that the service scheduler stops attempting to launch a task), you can issue a force-complete command. This will instruct the service scheduler to mark a specific step within a phase of the update as complete. You need to specify both the phase and the step:

dcos confluent-kafka update force-complete <phase> <step>

The names of the phases and the steps can be determined by running the update status command as described above.

Force Restart

Similar to force complete, you can also force a restart. This can either be done for an entire update, a phase of the update, or just for a specific step within a phase.

To restart the entire update:

dcos confluent-kafka update force-restart

Or for all steps in a single phase:

dcos confluent-kafka update force-restart <phase>

Or for a specific step within a specific phase:

dcos confluent-kafka update force-restart <phase> <step>

Open Source DC/OS, DC/OS 1.9, and Earlier

If you do not have Enterprise DC/OS 1.10 or later, the update commands above are not available.

Configuration Updates

For Open Source DC/OS of any version, or Enterprise DC/OS 1.9 and earlier, you can perform changes from the DC/OS GUI by editing the service scheduler’s configuration.

Go to the Services tab of the DC/OS GUI and click into the service you want to change.
Click on the service scheduler task (it will have the name of the service).
Click the three dots on the right hand side of the page and select Edit.
Find the configuration parameter to change in the list of environment variables.
Change the parameter and click Change and deploy.
The service scheduler will be restarted and roll out the configuration change to the service.

Finding the correct environment variable

While DC/OS Enterprise 1.10 and later support changing the configuration using the option schema directly, DC/OS Open Source and versions 1.9 and earlier require mapping those options to the environment variables that are passed to the service scheduler.

The correct environment variable for a given setting can vary depending on the service. For instance, some services have multiple types of nodes, each with separate count settings.

To find the correct environment variable, map from the service configuration to the service scheduler’s Marathon environment. However, most environment variable names are self-explanatory.

Service Version Updates

In the DC/OS web interface, destroy the confluent-kafka service scheduler to be updated.
Verify that you no longer see it in the DC/OS web interface.
Install the latest version of the confluent-kafka package with the version controlled service options:

dcos package install confluent-kafka -—options=options.json

Upgrading from Portworx Confluent Kafka

Customers who are using Portworx flavour of Confluent Kafka should follow these steps to migrate and upgrade to DC/OS Confluent Kafka.

Take a backup of the configuration of existing running service

dcos confluent-kafka --name=<name-of-service> describe > options-backup.json

Uninstall the service either from UI or dcos cli using

dcos package uninstall --app-id=<service-name> confluent-kafka

Create a file confluent-kafka.json with external volumes options enabled.

{
  "service": {
    "community": false,
      "user": "root"
   },
    "brokers": {
      "external_volume": {
        "enabled": true,
           "volume_name": "kafkaVolume",
           "volume_driver_options": "",
           "volume_path": "kafka-broker-data",
        }
      }
    }

IMPORTANT: Confluent Kafka will be installed with default parameter values. If you have different values in your backup configuration, please include them in the above `confluent-kafka.json` file as well to avoid service config related issues. The service will pick up the existing volumes named `kafkaVolume`. If your volume name is different, please change it in the above config to avoid creating new volumes with provided names.

Install the service

dcos package install confluent-kafka --options=confluent-kafka.json

Verify all the topics are migrated successfully.

Upgrading from 4.1.2 to 5.1.2(or higher)

The inter_broker_protocol_version now defaults to the newer, 2.1. This has a few implications, as described below:
- Confluent 4.1.2 supports inter_broker_protocol_version: 1.1 maximum, and by default it is set to 1.0.
- Confluent 5.1.2 supports inter_broker_protocol_versions up to 2.1.
- Confluent 5.3.0 supports inter_broker_protocol_versions up to 2.3.
- Confluent 5.4.0 supports inter_broker_protocol_versions up to 2.4.
- If you haven’t specified a inter_broker_protocol_version in your options file, the new default will be used and changed to 2.1.

To avoid any downtime, during the upgrade, some Kafka nodes will be on Confluent 4.1.2 using inter_broker_protocol_version 1.0 and others will be on Kafka 5.1.2 using protocol 2.1.

To avoid any potential downtime caused by this, change the protocol version used when upgrading Confluent.

Set up CLI to connect to a soak cluster.

Update your options_file.json with the following contents:

{
	"service": {
		"community": false
	},
	"kafka": {
		...
		"inter_broker_protocol_version": "1.0"
		...
	}
	...
}
	```

And update your service like so:

~$ dcos package install --cli --yes confluent-kafka
~$ dcos confluent-kafka --name=confluent-kafka update start \
	--package-version=2.9.1-5.4.0 \
	--options=options_file.json