DC/OS Hive Metastore Security

• The DC/OS Hive Metastore service allows you to create a service account to configure access for Hive Metastore. The service allows you to create and assign permissions as required for access.

• The DC/OS Hive Metastore service supports Hive Metastore’s native transport encryption mechanisms. The service provides automation and orchestration to simplify the usage of the following features.

NOTE: These security features are only available on DC/OS Enterprise 1.10 and later.

Provisioning a service account

This section describes how to configure DC/OS access for Hive Metastore. Depending on your security mode, Hive Metastore may require service authentication for access to DC/OS.

A service like Hive Metastore typically performs certain privileged actions on the cluster, which might require authenticating with the cluster. A service account associated with the service is used to authenticate with the DC/OS cluster. It is recommended to provisioning a separate service account for each service that would perform privileged operations. Service accounts authenticate using public-private keypair. The public key is used to create the service account in the cluster, while the corresponding private key is stored in the secret store. The service account and the service account secret are passed to the service as install time options.

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

• DC/OS CLI installed and be logged in as a superuser.
• Enterprise DC/OS CLI 0.4.14 or later installed.

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created using the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

NOTE: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (for example, hive-metastore) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d <description> hive-metastore

You can verify your new service account using the following command.

dcos security org service-accounts show hive-metastore

Create a Secret

Create a secret (hive-metastore/<secret-name>) with your service account and private key specified (<private-key>.pem).

NOTE: If you store your secret in a path that matches the service name, for example, service name and secret path are both hive-metastore, then only the service named hive-metastore can access it.

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> hive-metastore/<secret-name>

NOTE: If you are running, now EOL-ed, DC/OS 1.11 or older you would need to add --strict to the above command. For example, dcos security secrets --strict create-sa-secret .pem hive-metastore/sa-secret .

You can list the secrets with this command:

dcos security secrets list /

Create and Assign Permissions

Use the following DC/OS CLI commands to rapidly provision the Hive Metastore service account with the required permissions.

1. Create the permission.

IMPORTANT: The value to be used for <service-role> will be based on the service name, package version and DC/OS version. The table below shows a few examples of service names and the corresponding Mesos roles they would use. This version of hive-metastore is quota support built in. To determine whether the service group, you are deploying a service to, has enforceRole set to true or false please check this KB article.

If you need help configuring the permissions for hive-metastore, please feel to reach out to D2iQ support by filing a support ticket. Replace the instances of <service-role> with the correct name (<name>-role).

Permissive

Run these commands with the service account name you created for the service in the Create a Service Account step above. For example we are using hive-metastore

dcos security org users grant hive-metastore dcos:mesos:master:framework:role:<service-role> create --description "Allow registering as a framework of role <service-role> with Mesos master"
dcos security org users grant hive-metastore dcos:mesos:master:reservation:role:<service-role> create --description "Allow creating Mesos resource reservations of role <service-role>"
dcos security org users grant hive-metastore dcos:mesos:master:volume:role:<service-role> create --description "Allow creating Mesos persistent volumes of role <service-role>"
dcos security org users grant hive-metastore dcos:mesos:master:reservation:principal:hive-metastore delete --description "Allow unreserving Mesos resource reservations with principal hive-metastore"
dcos security org users grant hive-metastore dcos:mesos:master:volume:principal:hive-metastore delete --description "Allow deleting Mesos persistent volumes with principal hive-metastore"

Strict

Run these commands with the service account name you created for the service in the Create a Service Account step above. For example we are using hive-metastore

dcos security org users grant hive-metastore dcos:mesos:master:task:user:nobody create --description "Allow running a task as linux user nobody"
dcos security org users grant hive-metastore dcos:mesos:master:framework:role:<service-role> create --description "Allow registering as a framework of role <service-role> with Mesos master"
dcos security org users grant hive-metastore dcos:mesos:master:reservation:role:<service-role> create --description "Allow creating Mesos resource reservations of role <service-role>"
dcos security org users grant hive-metastore dcos:mesos:master:volume:role:<service-role> create --description "Allow creating Mesos persistent volumes of role <service-role>"
dcos security org users grant hive-metastore dcos:mesos:master:reservation:principal:hive-metastore delete --description "Allow unreserving Mesos resource reservations with principal hive-metastore"
dcos security org users grant hive-metastore dcos:mesos:master:volume:principal:hive-metastore delete --description "Allow deleting Mesos persistent volumes with principal hive-metastore"

Using the Secret Store for Passwords

Enterprise DC/OS provides a Secrets store to enable access to sensitive data such as database passwords, private keys, and API tokens. DC/OS manages secure transportation of secret data, access control and authorization, and secure storage of secret content. Detailed information can be found here.

All tasks defined in the pod will have access to secret data. If the content of the secret is changed, the relevant pod needs to be restarted so that it can get updated content from the secret store.

We can use secrets in Hive Metastore to store database passwords. We can use the secret store as follows in order to store and use secrets in Hive Metastore service:

1. From the left-side navigation menu, click on Secrets.

2. From the Secrets page, click on the + icon in the top-right corner of the screen to create a new secret key-value pair.

3. In the ID field, provide a unique ID for the key-pair we want to create. This ID will be used later to enable secrets. In the Value field, enter the value of the secret; for example, a database password, private key, or API token. Lastly, click on Create Secret.

4. Go to the Hive Metastore’s service configuration page for a fresh deployment of service using secrets. Click on the Hive Metastore configuration menu to proceed.

5. Scroll down and look for the Enable Secrets checkbox. Check the box to enable secrets.

6. Now, enter the ID of the secret created earlier in the Database Password field. Click Review & Run. The service will now be deployed using secrets.

Authentication

DC/OS Hive Metastore supports Kerberos authentication.

Kerberos Authentication

Kerberos authentication relies on a central authority to verify that Metastore clients are who they say they are. DC/OS Hive Metastore integrates with your existing Kerberos infrastructure to verify the identity of clients.

Prerequisites

• The hostname and port of a KDC reachable from your DC/OS cluster
• Sufficient access to the KDC to create Kerberos principals
• Sufficient access to the KDC to retrieve a keytab for the generated principals
• The DC/OS Enterprise CLI
• DC/OS Superuser permissions

Configure Kerberos Authentication

Create principals

The DC/OS Hive Metastore service requires Kerberos principals for each node to be deployed. The overall topology of the Metastore service is:

• 1 init node
• A configurable number of server nodes

The required Kerberos principals will have the form:

<service primary>/node-0-init.<service subdomain>.autoip.dcos.thisdcos.directory@<service realm>
<service primary>/node-<data-index>-server.<service subdomain>.autoip.dcos.thisdcos.directory@<service realm>

with:

• service primary = service.security.kerberos.primary
• data index = 0 up to n
• service subdomain = service.name with all /'s removed
• service realm = service.security.kerberos.realm

For example, if installing with these options:

{
    "service": {
        "name": "a/good/example",
        "security": {
            "kerberos": {
                "enabled": true,
                "primary": "example",
                "realm": "EXAMPLE"
            }
        }
    },
    "hive_metastore": {
        "count": 3
    }
}

then the principals to create would be:

example/node-0-init.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE
example/node-0-server.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE
example/node-1-server.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE
example/node-2-server.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE

Active Directory

Microsoft Active Directory can be used as a Kerberos KDC. Doing so requires creating a mapping between Active Directory users and Kerberos principals.

The utility ktpass can be used to both create a keytab from Active Directory and generate the mapping at the same time.

The mapping can, however, be created manually. For a Kerberos principal like <primary>/<host>@<REALM>, the Active Directory user should have its servicePrincipalName and userPrincipalName attributes set to,

servicePrincipalName = <primary>/<host>
userPrincipalName = <primary>/<host>@<REALM>

For example, with the Kerberos principal ``, then the correct mapping would be,

servicePrincipalName = 
userPrincipalName = 

If either mapping is incorrect or not present, the service will fail to authenticate that Principal. The symptom in the Kerberos debug logs will be an error of the form

KRBError:
sTime is Wed Feb 07 03:22:47 UTC 2018 1517973767000
suSec is 697984
error code is 6
error Message is Client not found in Kerberos database
sname is krbtgt/AD.MESOSPHERE.COM@AD.MESOSPHERE.COM
msgType is 30

when the userPrincipalName is set incorrectly, and an error of the form

KRBError:
sTime is Wed Feb 07 03:44:57 UTC 2018 1517975097000
suSec is 128465
error code is 7
error Message is Server not found in Kerberos database
sname is kafka/kafka-1-broker.confluent-kafka.autoip.dcos.thisdcos.directory@AD.MESOSPHERE.COM
msgType is 30

when the servicePrincipalName is set incorrectly.

Place Service Keytab in DC/OS Secret Store

The DC/OS Hive Metastore service uses a keytab containing all node principals (service keytab). After creating the principals above, generate the service keytab making sure to include all the node principals. This will be stored as a secret in the DC/OS Secret Store.

NOTE: DC/OS 1.10 does not support adding binary secrets directly to the secret store, only text files are supported. Instead, first base64 encode the file, and save it to the secret store as /desired/path/__dcos_base64__secret_name. The DC/OS security modules will handle decoding the file when it is used by the service.

The service keytab should be stored at service/path/name/service.keytab. As noted above. for DC/OS 1.10, it would be __dcos_base64__service.keytab), where service/path/name matches the path and name of the service. For example, if installing with the options

{
    "service": {
        "name": "a/good/example"
    }
}

then the service keytab should be stored at a/good/example/service.keytab.

Documentation for adding a file to the secret store can be found here.

NOTE: Secrets access is controlled by DC/OS Spaces, which function like namespaces. Any secret in the same DC/OS Space as the service will be accessible by the service. However, matching the two paths is the most secure option. Additionally the secret name service.keytab is a convention and not a requirement.

Install the Service

Install the DC/OS Hive Metastore service with the following options in addition to your own:

{
    "service": {
        "security": {
            "kerberos": {
                "enabled": true,
                "kdc": {
                    "hostname": "<kdc host>",
                    "port": <kdc port>
                },
                "primary": "<service primary default hive-metastore>",
                "realm": "<realm>",
                "keytab_secret": "<path to keytab secret>",
                "debug": <true|false default false>
            }
        }
    }
}

Authorization

DC/OS Hive Metastore supports Storage Based Authorization.

Storage Based Authorization

The DC/OS Hive Metastore service supports Metastore’s storage based authorization. When metastore server security is configured to use Storage Based Authorization, it uses the file system permissions for folders corresponding to the different metadata objects as the source of truth for the authorization policy.

• Storage based authorization authorizes read privilege on database and tables.
• Use of Storage Based Authorization in metastore is recommended.

Install the Service

Install the DC/OS Hive Metastore service with the following options in addition to your own:

{
    "service": {
        "security": {
            "authorization": {
                "enabled": true
            },
        }
    }
}

Transport Encryption

With transport encryption enabled, DC/OS Hive Metastore will automatically deploy all nodes with the correct configuration to encrypt communication via SSL. The nodes will communicate securely between themselves using SSL.

The service uses the DC/OS CA to generate the SSL artifacts that it uses to secure the service. Any client that trusts the DC/OS CA will consider the service’s certificates valid.

Prerequisites

• A DC/OS Service Account with a secret stored in the DC/OS Secret Store.
• DC/OS Superuser permissions for modifying the permissions of the Service Account.

Configure Transport Encryption

Set up the service account

Grant the service account the correct permissions.

• In DC/OS 1.10, the required permission is dcos:superuser full.
• In DC/OS 1.11 and later, the required permissions are:

dcos:secrets:default:/<service name>/* full
dcos:secrets:list:default:/<service name> read
dcos:adminrouter:ops:ca:rw full
dcos:adminrouter:ops:ca:ro full

where <service name> is the name of the service to be installed.

Run the following DC/OS Enterprise CLI commands to set permissions for the service account on a strict cluster:

dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:task:app_id:<service/name> create
dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:reservation:principal:dev_hdfs create
dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:volume:principal:dev_hdfs create

Install the service

Install the DC/OS Hive Metastore service including the following options in addition to your own:

{
    "service": {
        "service_account": "<your service account name>",
        "service_account_secret": "<full path of service secret>",
        "security": {
            "ssl_enabled": true
        }
    }
}

Transport Encryption for Clients

With Transport Encryption enabled, service clients will need to be configured to use the DC/OS CA bundle to verify the connections they make to the service. Consult your client’s documentation for trusting a CA and configure your client appropriately.

Forwarding DNS and Custom Domain

Every DC/OS cluster has a unique cryptographic ID which can be used to forward DNS queries to that cluster. To securely expose the service outside the cluster, external clients must have an upstream resolver configured to forward DNS queries to the DC/OS cluster of the service as described here.

With only forwarding configured, DNS entries within the DC/OS cluster will be resolvable at <task-domain>.autoip.dcos.<cryptographic-id>.dcos.directory. However, if you configure a DNS alias, you can use a custom domain. For example, <task-domain>.cluster-1.acmeco.net. In either case, the DC/OS Hive Metastore service will need to be installed with an additional security option:

{
    "service": {
        "security": {
            "custom_domain": "<custom-domain>"
        }
    }
}

where <custom-domain> is one of autoip.dcos.<cryptographic-id>.dcos.directory or your organization’s specific domain (e.g., cluster-1.acmeco.net).

As a concrete example, using the custom domain of cluster-1.acmeco.net the node 0 task would have a host of node-0-server.<service-name>.cluster-1.acmeco.net.