Enterprise DC/OS 1.10 and later

Enterprise DC/OS 1.10 introduced a convenient command line option that allows for easier updates to a service’s configuration and version, as well as allowing users to inspect the status of an update, to pause and resume updates, and to restart or complete steps if necessary.

In Enterprise DC/OS 1.11 and later, configuration updates may be done from the UI as well as from the CLI, but managing the phases and steps of an update must be done from the CLI. Service version updates must always be done from the CLI.

Prerequisites

Enterprise DC/OS 1.10 or later
A DC/OS SDK-based Service with a version later than 2.0.0-x
The DC/OS CLI installed and available
The service’s subcommand available and installed on your local machine
- You can install just the subcommand CLI by running dcos package install --cli data-science-engine.
- If you are running an older version of the subcommand CLI that does not have the update command, uninstall and reinstall your CLI.
```
dcos package uninstall --cli data-science-engine
dcos package install --cli data-science-engine
```

Updating service version

Viewing available versions

The update package-versions command allows you to view the versions of the DC/OS Data Science Engine service that you can upgrade or downgrade to.

The CLI command is:

dcos data-science-engine update package-versions

The result will be of the form:

Current package version is: "1.0.1-1.2.3"
Package can be downgraded to: ["1.0.0-1.2.3"]
Package can be upgraded to: ["1.0.2-1.2.3"]

Upgrading or downgrading a service

Before updating the service itself, update its CLI subcommand to the new version:

dcos package uninstall --cli data-science-engine
dcos package install --cli data-science-engine --package-version="<version>"

After the CLI subcommand has been updated, call the update start command, passing in the version

dcos data-science-engine update start --package-version="<version>"

If you are missing mandatory configuration parameters, the update command will return an error.

To supply missing configuration values or to override configuration values, you can also provide an options.json file (see Updating configuration below):

dcos data-science-engine update start --options=options.json --package-version="<version>"

The default behavior on update is to merge ‘Default’, ‘Stored’ and ‘Provided’ configurations, in that order, and then validate against the schema. In some situations, such as when a schema option has been removed, the default behavior might result in an invalid configuration. You can work around this with --replace=true which, when specified, will override the ‘Stored’ options with the ‘Provided’ options.

dcos data-science-engine update start --options=options.json --replace=true --package-verion="<version>"

See Advanced update actions for commands you can use to inspect and manipulate an update after it has started.

Updating configuration

Preparing configuration

If you installed the service with Enterprise DC/OS 1.10, you can fetch the full configuration of a service (including any default values that were applied during installation).

dcos data-science-engine describe > options.json

Make any configuration changes to this options.json file.

Updating after DC/OS 1.9 to DC/OS 1.10 Upgrade

If the service was previously installed on DC/OS 1.9, and the cluster has now been upgraded to DC/OS 1.10, it will not have the necessary metadata for the update commands to succeed. You will need to perform an initial update to create the metadata.

Using the version controlled options file of the service, start a configuration update:

dcos data-science-engine update start --options=<options-file>

This will update the service metadata with the correct stored options. The service scheduler will be restarted, but service tasks will not. Going forward, you can use the update commands as described elsewhere.

WARNING: It is critical to do the update with the exact options the service is currently running with in this upgrade step. Any option not specified in the options file will be lost and the default used.

Starting the update

Once you are ready to begin, initiate an update using the DC/OS CLI, passing in the updated options.json file:

dcos data-science-engine update start --options=options.json

You will receive an acknowledgement message and the DC/OS package manager will restart the service scheduler which will proceed to update the service.

See Advanced update actions for commands you can use to inspect and manipulate an update after it has started.

Advanced update actions

The following sections describe advanced commands that may be used to interact with an update in progress.

Monitoring the update

Once the service scheduler has been restarted, it will begin a new deployment plan as individual pods are restarted with the new configuration. Depending on the high availability characteristics of the service being updated, you may experience a service disruption.

You can query the status of the update as follows:

dcos data-science-engine update status

If the service scheduler is still restarting, DC/OS will not be able to route to it and this command will return an error message. Wait a short while and try again. You can also go to the Services tab of the DC/OS GUI to check the status of the restart.

Pause

To pause an ongoing update, issue a pause command:

dcos data-science-engine update pause

You will receive an error message if the plan has already completed or has already been paused. Once completed, the update will enter the WAITING state.

Resume

If the update is in a WAITING state, as a result of being paused or reaching a breakpoint that requires manual operator verification, you can use the resume command to continue the update:

dcos data-science-engine update resume

You will receive an error message if you attempt to resume an update that is already in progress or has already completed.

Force Complete

In order to manually “complete” a step (such that the service scheduler stops attempting to launch a task), you can issue a force-complete command. This will instruct the service scheduler to mark a specific step within a phase of the update as complete. You need to specify both the phase and the step:

dcos data-science-engine update force-complete <phase> <step>

The names of the phases and the steps can be determined by running the update status command as described above.

Force Restart

Similar to force complete, you can also force a restart. This can either be done for an entire update, a phase of the update, or just for a specific step within a phase.

To restart the entire update:

dcos data-science-engine update force-restart

Or for all steps in a single phase:

dcos data-science-engine update force-restart <phase>

Or for a specific step within a specific phase:

dcos data-science-engine update force-restart <phase> <step>

Open Source DC/OS, DC/OS 1.9, and Earlier

If you do not have Enterprise DC/OS 1.10 or later, the update commands above are not available.

Configuration Updates

For Open Source DC/OS of any version, or Enterprise DC/OS 1.9 and earlier, you can perform changes from the DC/OS GUI by editing the service scheduler’s configuration.

Go to the Services tab of the DC/OS GUI and click into the service you want to change.
Click on the service scheduler task (it will have the name of the service).
Click the three dots on the right hand side of the page and select Edit.
Find the configuration parameter to change in the list of environment variables.
Change the parameter and click Change and deploy.
The service scheduler will be restarted and roll out the configuration change to the service.

Finding the correct environment variable

While DC/OS Enterprise 1.10 and later support changing the configuration using the option schema directly, DC/OS Open Source and versions 1.9 and earlier require mapping those options to the environment variables that are passed to the service scheduler.

The correct environment variable for a given setting can vary depending on the service. For instance, some services have multiple types of nodes, each with separate count settings.

To find the correct environment variable, map from the service configuration to the service scheduler’s Marathon environment. However, most environment variable names are self-explanatory.

Service Version Updates

In the DC/OS web interface, destroy the data-science-engine service scheduler to be updated.
Verify that you no longer see it in the DC/OS web interface.
Install the latest version of the data-science-engine package with the version controlled service options:

dcos package install data-science-engine -—options=options.json

Upgrades from 2.x versions

It is possible to upgrade directly to 2.0.1 from version 2.0.0. However, you’ll need to run the following command to successfully upgrade your DC/OS Data Science Engine package:
```
dcos data-science-engine update start --package-version=2.0.1 --replace
```

Upgrades from 1.x versions

DC/OS Data Science Engine 2.x does not support in-place upgrades from 1.x versions. In order to transition to a new version you must manually back up your data located in your jupyter_data folder and upload it to a new instance of DC/OS Data Science Engine, running on the latest version using the following guide.

Navigate to the Notebook terminal from Jupyter Notebook UI and run the following command to compress the contents of your jupyter_data folder:
```
nobody@95602332-7dd1-4a24-b2a5-b0fa028c0998:/mnt/mesos/sandbox$ tar -cvf jupyter_data.tar.gz jupyter_data
```
NOTE: If you have files you would like to keep from outside of the jupyter_data folder, you should also back up these files, but keep in mind that only the data in the jupyter_data folder persists between service restarts. If you copy files from outside of the jupyter_data directory, please ensure you do not copy any hidden files that could interfere with the operation of DC/OS Data Science Engine after being uploaded to the new instance.
Use the Download button to download jupyter_data.tar.gz.
Deploy a new instance of DC/OS Data Science Engine, specifying the upgrade version. You may need to tear down your existing DC/OS Data Science Engine instance depending on the available resources in the cluster.
From the new instance’s UI, select the Upload button to transfer jupyter_data.tar.gz to the sandbox of the new DC/OS Data Science Engine instance.
Then run the following command to uncompress the contents of the archive to the jupyter_data folder:
```
nobody@e08af121-e9e7-4c0f-8188-23bfead5080a:/mnt/mesos/sandbox$ tar -xvf jupyter_data.tar.gz
```

Upgrade

Upgrading DC/OS Data Science Engine