Updating

Enterprise DC/OS 1.10 and later

Enterprise DC/OS 1.10 introduced a convenient command line option that allows for easier updates to a service’s configuration and version, as well as allowing users to inspect the status of an update, to pause and resume updates, and to restart or complete steps if necessary.

In Enterprise DC/OS 1.11 and later, configuration updates may be done from the UI as well as from the CLI, but managing the phases and steps of an update must be done from the CLI. Service version updates must always be done from the CLI.

Prerequisites

Enterprise DC/OS 1.10 or later
A DC/OS SDK-based Service with a version later than 2.0.0-x
The DC/OS CLI installed and available
The service’s subcommand available and installed on your local machine
- You can install just the subcommand CLI by running dcos package install --cli hdfs.
- If you are running an older version of the subcommand CLI that does not have the update command, uninstall and reinstall your CLI.
```
dcos package uninstall --cli hdfs
dcos package install --cli hdfs
```

Updating service version

Viewing available versions

The update package-versions command allows you to view the versions of the Apache HDFS service that you can upgrade or downgrade to.

The CLI command is:

dcos hdfs update package-versions

The result will be of the form:

Current package version is: "1.0.1-1.2.3"
Package can be downgraded to: ["1.0.0-1.2.3"]
Package can be upgraded to: ["1.0.2-1.2.3"]

Upgrading or downgrading a service

Before updating the service itself, update its CLI subcommand to the new version:

dcos package uninstall --cli hdfs
dcos package install --cli hdfs --package-version="<version>"

After the CLI subcommand has been updated, call the update start command, passing in the version

dcos hdfs update start --package-version="<version>"

If you are missing mandatory configuration parameters, the update command will return an error.

To supply missing configuration values or to override configuration values, you can also provide an options.json file (see Updating configuration below):

dcos hdfs update start --options=options.json --package-version="<version>"

The default behavior on update is to merge ‘Default’, ‘Stored’ and ‘Provided’ configurations, in that order, and then validate against the schema. In some situations, such as when a schema option has been removed, the default behavior might result in an invalid configuration. You can work around this with --replace=true which, when specified, will override the ‘Stored’ options with the ‘Provided’ options.

dcos hdfs update start --options=options.json --replace=true --package-verion="<version>"

See Advanced update actions for commands you can use to inspect and manipulate an update after it has started.

Updating configuration

Preparing configuration

If you installed the service with Enterprise DC/OS 1.10, you can fetch the full configuration of a service (including any default values that were applied during installation).

dcos hdfs describe > options.json

Make any configuration changes to this options.json file.

Updating after DC/OS 1.9 to DC/OS 1.10 Upgrade

If the service was previously installed on DC/OS 1.9, and the cluster has now been upgraded to DC/OS 1.10, it will not have the necessary metadata for the update commands to succeed. You will need to perform an initial update to create the metadata.

Using the version controlled options file of the service, start a configuration update:

dcos hdfs update start --options=<options-file>

This will update the service metadata with the correct stored options. The service scheduler will be restarted, but service tasks will not. Going forward, you can use the update commands as described elsewhere.

WARNING: It is critical to do the update with the exact options the service is currently running with in this upgrade step. Any option not specified in the options file will be lost and the default used.

Starting the update

Once you are ready to begin, initiate an update using the DC/OS CLI, passing in the updated options.json file:

dcos hdfs update start --options=options.json

You will receive an acknowledgement message and the DC/OS package manager will restart the service scheduler which will proceed to update the service.

See Advanced update actions for commands you can use to inspect and manipulate an update after it has started.

Advanced update actions

The following sections describe advanced commands that may be used to interact with an update in progress.

Monitoring the update

Once the service scheduler has been restarted, it will begin a new deployment plan as individual pods are restarted with the new configuration. Depending on the high availability characteristics of the service being updated, you may experience a service disruption.

You can query the status of the update as follows:

dcos hdfs update status

If the service scheduler is still restarting, DC/OS will not be able to route to it and this command will return an error message. Wait a short while and try again. You can also go to the Services tab of the DC/OS GUI to check the status of the restart.

Pause

To pause an ongoing update, issue a pause command:

dcos hdfs update pause

You will receive an error message if the plan has already completed or has already been paused. Once completed, the update will enter the WAITING state.

Resume

If the update is in a WAITING state, as a result of being paused or reaching a breakpoint that requires manual operator verification, you can use the resume command to continue the update:

dcos hdfs update resume

You will receive an error message if you attempt to resume an update that is already in progress or has already completed.

Force Complete

In order to manually “complete” a step (such that the service scheduler stops attempting to launch a task), you can issue a force-complete command. This will instruct the service scheduler to mark a specific step within a phase of the update as complete. You need to specify both the phase and the step:

dcos hdfs update force-complete <phase> <step>

The names of the phases and the steps can be determined by running the update status command as described above.

Force Restart

Similar to force complete, you can also force a restart. This can either be done for an entire update, a phase of the update, or just for a specific step within a phase.

To restart the entire update:

dcos hdfs update force-restart

Or for all steps in a single phase:

dcos hdfs update force-restart <phase>

Or for a specific step within a specific phase:

dcos hdfs update force-restart <phase> <step>

Open Source DC/OS, DC/OS 1.9, and Earlier

If you do not have Enterprise DC/OS 1.10 or later, the update commands above are not available.

Configuration Updates

For Open Source DC/OS of any version, or Enterprise DC/OS 1.9 and earlier, you can perform changes from the DC/OS GUI by editing the service scheduler’s configuration.

Go to the Services tab of the DC/OS GUI and click into the service you want to change.
Click on the service scheduler task (it will have the name of the service).
Click the three dots on the right hand side of the page and select Edit.
Find the configuration parameter to change in the list of environment variables.
Change the parameter and click Change and deploy.
The service scheduler will be restarted and roll out the configuration change to the service.

Finding the correct environment variable

While DC/OS Enterprise 1.10 and later support changing the configuration using the option schema directly, DC/OS Open Source and versions 1.9 and earlier require mapping those options to the environment variables that are passed to the service scheduler.

The correct environment variable for a given setting can vary depending on the service. For instance, some services have multiple types of nodes, each with separate count settings.

To find the correct environment variable, map from the service configuration to the service scheduler’s Marathon environment. However, most environment variable names are self-explanatory.

Service Version Updates

In the DC/OS web interface, destroy the hdfs service scheduler to be updated.
Verify that you no longer see it in the DC/OS web interface.
Install the latest version of the hdfs package with the version controlled service options:

dcos package install hdfs -—options=options.json