Spark is available in the Universe and can be installed by using either the DC/OS GUI or the DC/OS CLI.

Prerequisites

DC/OS and DC/OS CLI installed
Depending on your security mode, Spark requires service authentication for access to DC/OS.

Security mode	Service account
Disabled	Not available
Permissive	Optional
Strict	Required

For more information about service accounts, see Security:

Default installation

To install the DC/OS Apache Spark service, run the following command on the DC/OS CLI. This installs the Spark DC/OS service, Spark CLI, dispatcher, and, optionally, the history server. See Custom installation to install the history server.

dcos package install spark

Go to the Services > Deployments tab of the DC/OS GUI to monitor the deployment. When it has finished deploying, visit Spark at http://<dcos-url>/service/spark/.

You can also install Spark via the DC/OS GUI.

Spark CLI

You can install the Spark CLI with this command. This is useful if you already have a Spark cluster running, but need the Spark CLI.

IMPORTANT: If you install Spark via the DC/OS GUI, you must install the Spark CLI as a separate step from the DC/OS CLI.

dcos package install spark --cli

Custom installation

You can customize the default configuration properties by creating a JSON options file and passing it to dcos package install --options. For example, to launch the Dispatcher using the Universal Container Runtime (UCR), create a file called options.json.

To customize the installation:

Create the options.json configuration file.

{
  "service": {
    "UCR_containerizer": true
  }
}

Install Spark with the configuration specified in the options.json file:
```
dcos package install --options=options.json spark
```
Run this command to see all configuration options:
```
dcos package describe spark --config
```

Customize Spark distribution

DC/OS Apache Spark does not support arbitrary Spark distributions, but Mesosphere does provide multiple pre-built distributions, primarily used to select Hadoop versions.

To use one of these distributions, select the corresponding Docker image from here, then use that value to set the following configuration variable:

{
  "service": {
    "docker-image": "<docker-image>"
  }
}

Customize Spark user

DC/OS Spark user defaults to nobody, to override it set the following configuration variable:

{
  "service": {
    "user": "<user>"
  }
}

Spark runs all of its components in Docker containers. Since the Docker image contains a full Linux userspace with its own /etc/users file, it is possible for the user nobody to have a different UID inside the container than on the host system. Although user nobody has UID 65534 by convention on many systems, this is not always the case. As Mesos does not perform UID mapping between Linux user namespaces for a Docker containerizer, specifying a service user of nobody in this case will cause access failures when the container user attempts to open or execute a filesystem resource owned by a user with a different UID, preventing the service from launching. If the hosts in your cluster have a UID for nobody other than 65534, you will need to a provide valid UID for nobody in the configuration to run the service successfully. For example, on RHEL/Centos based distributions:

{
  "service": {
    "user": "nobody",
    "docker_user": "99"
  }
}

Configure Spark virtual network

DC/OS Spark can be launched in a virtual network and configured with network labels.

Here’s an example of a Spark configuration for DC/OS overlay network:

{
  "service": {
    "virtual_network_enabled": true,
    "virtual_network_name": "dcos",
    "virtual_network_plugin_labels": [
        {"key": "key_1", "value": "value_1"},
        {"key": "key_2", "value": "value_2"}
    ]
  }
}

When DC/OS Spark is deployed in a virtual network, all submitted jobs will run in the same network until another network is specified in job submit arguments.

You can check the existing limitations of virtual network support here.

Configure Spark role enforcement

DC/OS Spark by default launches submitted applications (Drivers) with the same role it is running itself. Users can always provide spark.mesos.role as a part of their application configuration and override the role of Dispatcher. For the cases when Dispatcher role needs to be enforced so that users are not allowed to override it, add the following configuration flag:

{
  "service": {
    "role": "<dispatcher role>",
    "enforce_role": true
  }
}

If spark.mesos.role is not provided via spark-submit or dcos spark run the Dispatcher will assign its own role to the Drivers. In case spark.mesos.role has a different value than the Dispatcher’s role - the submission will be rejected.

NOTE: When Spark is installed in a Marathon group with group role enforced, it will use Marathon group to register with Mesos and will automatically enforce that role.

Minimal installation

For development purposes, use dcos-vagrant to install Spark on a local DC/OS cluster.

Install a minimal DC/OS Vagrant according to the instructions here.
Install Spark:
```
dcos package install spark
```

Run a simple job:

dcos spark run --submit-args="--class org.apache.spark.examples.SparkPi https://downloads.mesosphere.com/spark/assets/spark-examples_2.12-3.0.1.jar 30"

IMPORTANT: A limited resource environment such as DC/OS Vagrant restricts some of the features available in DC/OS Apache Spark. For example, you must have enough resources to start up a 5-agent cluster, otherwise you will not be able to install DC/OS HDFS an enable the history server.

Also, a limited resource environment can restrict how you size your executors, for example with spark.executor.memory.

Multiple installations

Installing multiple instances of the DC/OS Apache Spark package provides basic multi-team support. Each dispatcher displays only the jobs submitted to it by a given team, and each team can be assigned different resources.

To install multiple instances of the DC/OS Apache Spark package, set each service.name to a unique name (for example, spark-dev) in your JSON configuration file during installation. For example, create a JSON options file name multiple.json:

{
  "service": {
    "name": "spark-dev"
  }
}

Install Spark with the options file specified:

dcos package install --options=multiple.json spark

To specify which instance of Spark to use add --name=<service_name> to your CLI, for example
```
dcos spark --name=spark-dev run ...
```

Installation for strict mode

If your cluster is set up for strict security then you will follow these steps to install and run Spark.

Service accounts and secrets

Install the dcos-enterprise-cli to get CLI security commands (if you have not already done so):
```
dcos package install dcos-enterprise-cli
```
Create a 2048-bit RSA public-private key pair using the Enterprise DC/OS CLI. Create a public-private key pair and save each value into a separate file within the current directory.
```
dcos security org service-accounts keypair <your-private-key>.pem <your-public-key>.pem
```
For example:
```
dcos security org service-accounts keypair private-key.pem public-key.pem
```
Create a new service account, service-account-id (for example, spark-principal) containing the public key, your-public-key.pem.
```
dcos security org service-accounts create -p <your-public-key>.pem -d "Spark service account" <service-account>
```
For example:
```
dcos security org service-accounts create -p public-key.pem -d "Spark service account" spark-principal
```
In Mesos parlance, a service-account is called a principal and so we use the terms interchangeably here.

You can verify your new service account using the following command.
```
dcos security org service-accounts show <service-account>
```

Create a secret (for example, spark/<secret-name>) with your service account, service-account, and private key specified, your-private-key.pem.

# permissive mode
dcos security secrets create-sa-secret <your-private-key>.pem <service-account> spark/<secret-name>
# strict mode
dcos security secrets create-sa-secret --strict <private-key>.pem <service-account> spark/<secret-name>

For example, on a strict-mode DC/OS cluster:

dcos security secrets create-sa-secret --strict private-key.pem spark-principal spark/spark-secret

Use the dcos security secrets list / command to verify that the secrets were created:
```
dcos security secrets list /
```

Assigning permissions

Permissions must be created so that the Spark service will be able to start Spark jobs and so the jobs themselves can launch the executors that perform the work on their behalf. There are a few points to keep in mind depending on your cluster:

Spark runs by default under the Mesos default role, which is represented by the * symbol. You can deploy multiple instances of Spark without modifying this default. If you want to override the default Spark role, you must modify these code samples accordingly. We use spark-service-role to designate the role used below.

Permissions can also be assigned through the GUI.

Run the following to create the required permissions for Spark:

dcos security org users grant <service-account> dcos:mesos:master:task:user:<user> create --description "Allows the Linux user to execute tasks"
dcos security org users grant <service-account> dcos:mesos:master:framework:role:< spark-service-role> create --description "Allows a framework to register with the Mesos master using the Mesos default role"
dcos security org users grant <service-account> dcos:mesos:master:task:app_id:/<service_name> create --description "Allows reading of the task state"

Note that above the dcos:mesos:master:task:app_id:/<service_name> will likely be dcos:mesos:master:task:app_id:/spark

For example, continuing from above:

dcos security org users grant spark-principal dcos:mesos:master:task:user:nobody create --description "Allows the Linux user to execute tasks"
dcos security org users grant spark-principal dcos:mesos:master:framework:role:* create --description "Allows a framework to register with the Mesos master using the Mesos default role"
dcos security org users grant spark-principal dcos:mesos:master:task:app_id:/spark create --description "Allows reading of the task state"

Here, we are using the service account spark-principal and the user nobody.

If you are running the Spark service as nobody (as we are in this example) you will need to add an additional permission for Marathon:

dcos security org users grant dcos_marathon dcos:mesos:master:task:user:nobody create --description "Allow Marathon to launch containers as nobody"

Install Spark with necessary configuration

Make a configuration file with the following before installing Spark, these settings can also be set through the GUI:

cat spark-strict-options.json
{
  "service": {
    "service_account": "<service-account-id>",
    "user": "<user>",
    "service_account_secret": "spark/<secret_name>"
  }
}

A minimal example would be:

{
  "service": {
    "service_account": "spark-principal",
    "user": "nobody",
    "service_account_secret": "spark/spark-secret"
  }
}

Then install:

dcos package install spark --options=spark-strict-options.json

Additional configuration for Spark jobs

You must add configuration parameters to your Spark jobs when submitting them.

Running jobs in strict mode cluster

To run a job on a strict mode cluster, you must add the principal to the command line. For example:

dcos spark run --verbose --submit-args=" \
--conf spark.mesos.principal=<service-account> \
--conf spark.mesos.containerizer=mesos \
--class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com/spark/assets/spark-examples_2.12-3.0.1.jar 100"

Running jobs as a different user

Spark Mesos Dispatcher uses the same user for running Spark jobs as itself and defaults to nobody. If you run Dispatcher as root and want to submit a job as a different user e.g. nobody, you must provide user property in the following way.

Universal Container Runtime

For UCR containerizer it is sufficient to provide spark.mesos.driverEnv.SPARK_USER=nobody configuration property when submitting a job:

dcos spark run --verbose --submit-args="\
--conf spark.mesos.driverEnv.SPARK_USER=nobody \
--class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com/spark/assets/spark-examples_2.12-3.0.1.jar 100"

Docker Engine

If you want to use the Docker Engine instead of the Universal Container Runtime, you must specify spark.mesos.executor.docker.parameters=user=nobody in addition to spark.mesos.driverEnv.SPARK_USER=nobody to run the Docker container as this user:

dcos spark run --verbose --submit-args="\
--conf spark.mesos.driverEnv.SPARK_USER=nobody \
--conf spark.mesos.executor.docker.parameters=user=nobody \
--class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com/spark/assets/spark-examples_2.12-3.0.1.jar 100"

If the hosts in your cluster have a UID for nobody other than 65534 (see Customize Spark user), you will need to provide a valid UID as a parameter to Docker containerizer via --conf spark.mesos.executor.docker.parameters=user=UID:

dcos spark run --verbose --submit-args="\
--conf spark.mesos.driverEnv.SPARK_USER=nobody \
--conf spark.mesos.executor.docker.parameters=user=99 \
--class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com/spark/assets/spark-examples_2.12-3.0.1.jar 100"

NOTE: UID should typically be set to 99 when running as nobody (default) on RHEL/CentOS

Running jobs in virtual network

To run a job in a virtual network and/or with network plugin labels assigned, one need to specify network name and labels in submit arguments:

dcos spark run --verbose --submit-args="\
--conf spark.mesos.network.name=dcos \
--conf spark.mesos.network.labels=key_1:value_1,key_2:value_2 \
--class org.apache.spark.examples.GroupByTest http://downloads.mesosphere.com/spark/assets/spark-examples_2.12-3.0.1.jar"

Install and Customize

Installing and customizing your DC/OS Apache Spark service