Configuring Alertmanager to alert via Slack

Create an Alertmanager configuration file

See the Alertmanager configuration documentation and the Slack configuration section for a detailed overview of how to create a configuration file with a Slack receiver.

Below is an example of a simple configuration file with a Slack receiver. The configuration file must be named config.yml. In this example, the Slack channel is set to #prometheus-alerts. You can set this to another existing Slack channel, or create the Slack channel #prometheus-alerts.

config.yml

# The root route on which each incoming alert enters.
route:
  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  #
  # To aggregate by all possible labels use '...' as the sole label name.
  # This effectively disables aggregation entirely, passing through all
  # alerts as-is. This is unlikely to be what you want, unless you have
  # a very low alert volume or your upstream notification system performs
  # its own grouping. Example: group_by: [...]
  group_by: ['alertname', 'cluster', 'service']

  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 30s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: slack_general

receivers:
- name: slack_general
  slack_configs:
  - channel: '#prometheus-alerts'

Save config.yml into a Git repository, in a folder called slack. Assume that the repository is https://github.com/company/alertmanager-configs.

Create the Slack API URL secret

The service is automatically configured to add the following global default value for slack_api_url in the configuration file.

global:
- slack_api_url: '<SLACK_API_URL>'

Create the secret slackapiurl-secret for the Slack API URL (Webhook URL) to use for notifications.

dcos security secrets create --value=<SLACK_API_URL> slackapiurl-secret

When installing the service, you must configure the secrets section to point to the secret slackapiurl-secret created.

{
  "alertmanager": {
    "secrets": {
      "slack_api_url_secret": "slackapiurl-secret"
    },
    "config_repository": {
      "url": "https://github.com/company/alertmanager-configs",
      "path": "/slack"
    }
  }
}

If the Git repository is private, you will need to configure the credentials to access the Git repository.

Install DC/OS Monitoring service

Install the service using the custom options file (options.json) created in the above steps:

dcos package install dcos-monitoring --options=options.json

The Alertmanager configurations defined in the repository will be automatically loaded when the service finishes deploying.

Triggering a reload of Alertmanager configurations

It is possible to trigger a reload of the Alertmanager configurations after the service is installed.

dcos monitoring plan start reload-alertmanager-config

Testing Slack alerts

You can test that Alertmanager is configured correctly by using SSH to connect to the node on which Alertmanager is running and manually hitting the Alertmanager endpoint to trigger an alert. Alertmanager should route the alert to the Slack channel you configured it to notify.

curl -H "Content-Type: application/json" -d '[{"labels":{"alertname":"TestAlert1"}}]' localhost:9093/api/v1/alerts

Check the #prometheus-alerts Slack channel (or whatever channel configured to receive alerts) and an alert called TestAlert1 should show up shortly.

Navigating to https://<CLUSTER_URL>/service/dcos-monitoring/alertmanager/#/alerts should also show the alert.