Enabling Alertmanager

Alertmanager is off by default. There is no default configuration that the Alertmanager is automatically run with. To enable Alertmanager, the DC/OS Monitoring service must be configured to load the Alertmanager configurations from a Git repository. See the Alertmanager documentation to learn how to create the configuration file.

Save Alertmanager configurations

The Alertmanager configuration file (YAML format) must be named config.yml. You should save config.yml as well as all template files in a Git repository. Assume that the repository is https://github.com/company/alertmanager-configs.

You can put config.yml and template files in a subfolder in the repository. For instance, https://github.com/company/alertmanager-configs/production.

{
  "alertmanager": {
    "config_repository": {
      "url": "https://github.com/company/alertmanager-configs",
      "path": "/production"
    }
  }
}

The Git repository can be either public or private. You can omit the credentials section if the Git repository is public. If the Git repository is private, you must configure the credentials to access the Git repository.

Create secrets for Git repository credentials

If the Git repository containing the Alertmanager configurations is private, you must configure the secrets first. Currently, the following Auth types are supported.

IMPORTANT: If the Git repository contains submodules, all the submodules must use the same Auth type as what is used for the repository. For example, if HTTP Auth is configured for the Git repository, all of the submodules in the repository's `.gitmodules` file must use the `https` URL scheme.

HTTP Auth

dcos security secrets create --value=<GITHUB_USERNAME> githubusername-secret
dcos security secrets create --value=<GITHUB_PASSWORD> githubpassword-secret

For GitHub, the password can be the API token.

Create a custom option file (options.json) like the following.

{
  "alertmanager": {
    "config_repository": {
      "url": "https://github.com/company/alertmanager-configs",
      "path": "/production",
      "credentials": {
        "username_secret": "githubusername-secret",
        "password_secret": "githubpassword-secret"
      }
    }
  }
}

SSH Auth

dcos security secrets create -f <PATH_TO_PRIVATE_KEY> gitsshkey-secret

For GitHub, you must add the Deployment Key (that is, the public key) to the repository.

Create a custom option file (options.json) like the following.

{
  "alertmanager": {
    "config_repository": {
      "url": "git@github.com:company/alertmanager-configs",
      "path": "/production",
      "credentials": {
        "deploy_key_secret": "gitsshkey-secret"
      }
    }
  }
}

NOTE: You will have to use `git@github.com:/.git` instead of `https` as the scheme of the URL.

Secrets in the Alertmanager configuration file

Instead of putting plain text secrets in the configuration file, you can save the secrets in the secret store, and the service will automatically configure the global default values in the Alertmanager configuration file.

The following secrets in the configuration file are supported:

Slack API URL (Webhook URL)

The service will automatically add the following global default value for slack_api_url in the configuration file.

global:
- slack_api_url: '<SLACK_API_URL>'

Then, you need to save the secret in the secret store.

dcos security secrets create --value=<SLACK_API_URL> slackapiurl-secret

When installing the service, you must configure the secrets section to point to the secret created. You may omit the credentials section if the Git repository is public.

{
  "alertmanager": {
    "secrets": {
      "slack_api_url": "slackapiurl-secret"
    },
    "config_repository": {
      "url": "https://github.com/company/alertmanager-configs",
      "path": "/production",
      "credentials": {
        "username_secret": "githubusername-secret",
        "password_secret": "githubpassword-secret"
      }
    }
  }
}

Install DC/OS Monitoring service

Install the service using the custom options file options.json that you’ve created:

dcos package install dcos-monitoring --options=options.json

The Alertmanager configurations defined in the repository will be automatically loaded when the service finishes deploying.

Triggering a reload of Alertmanager configurations

It is possible to trigger a reload of the Alertmanager configurations after the service is installed.

dcos monitoring plan start reload-alertmanager-config

Managing Alertmanager Configurations

Enabling and configuring Alertmanager