This guide expects that you already have a running DC/OS cluster based on Universal Installer 0.3. To learn more about running DC/OS with the Universal Installer have a look into the Guide for DC/OS on AWS using the Universal Installer.

You will learn how to place additional infrastructure into a AWS remote region. Remote regions will be connected to each other by using the AWS VPC Peering Feature.

Prerequisites

A running DC/OS Enterprise cluster set up with Universal Installer 0.3 modules
A subnet range for your remote region

Getting started with remote region

We expect your already running DC/OS clusters main.tf will look similar to this example. To deploy a remote region we have to do some changes to your main.tf

provider "aws" {
  # Change your default region here
  region = "us-east-1"
}

# Used to determine your public IP for forwarding rules
data "http" "whatismyip" {
  url = "http://whatismyip.akamai.com/"
}

module "dcos" {
  source  = "dcos-terraform/dcos/aws"
  version = "~> 0.3.0"

  providers = {
    aws = aws
  }

  cluster_name        = "my-dcos-demo"
  ssh_public_key_file = "<path-to-public-key-file>"
  admin_ips           = ["${data.http.whatismyip.body}/32"]

  num_masters        = "3"
  num_private_agents = "2"
  num_public_agents  = "1"

  dcos_version = "2.2"

  dcos_variant              = "ee"
  dcos_license_key_contents = "${file("./license.txt")}"

  # Make sure to set your credentials if you do not want the default EE
  # dcos_superuser_username      = "superuser-name"
  # dcos_superuser_password_hash = "${file("./dcos_superuser_password_hash.sha512")}"

  dcos_instance_os             = "centos_7.5"
  bootstrap_instance_type      = "m5.large"
  masters_instance_type        = "m5.2xlarge"
  private_agents_instance_type = "m5.xlarge"
  public_agents_instance_type  = "m5.xlarge"
}

output "masters-ips" {
  value = module.dcos.masters-ips
}

output "cluster-address" {
  value = module.dcos.masters-loadbalancer
}

output "public-agents-loadbalancer" {
  value = module.dcos.public-agents-loadbalancer
}

Remote region provider

The first change we have to apply to your main.tf is adding a specific provider statement for the remote region. In this example we will use us-west-2 with the alias usw2 as our remote region. We also add an alias statement to the provider used deploying our region holding the master instances.

This needs to be done so our modules know which account credentials to use.

Note: Some resources have name length limitations which is the reason we shorten our region name.

provider "aws" {
  # Change your default region here
  region = "us-east-1"
  alias  = "master"
}

provider "aws" {
  # Change your default region here
  region = "us-west-2"
  alias  = "usw2"
}

# ...

Shared config options

To create the remote region and its infrastructure we will use the same underlying modules as in our master region. This also means there will be some information needed for both infrastructures like cluster_name, admin_ips and ssh_public_key_file. To make the operation easier you should define local variables in your main.tf that will be used in every module.

#...

// lets define variables which are shared between all regions
locals {
  ssh_public_key_file = "~/.ssh/id_rsa.pub"
  cluster_name        = "my-dcos-demo"
  admin_ips           = ["${data.http.whatismyip.body}/32"]
}

#...

Internal subnetworks

Part of the shared information is which internal subnets are used in your infrastructure. This information needs to be known by all parts of the DC/OS so traffic can be routed and is allowed by the security groups. If you did not specify subnet_range, terraform uses the default which is 172.16.0.0/16. The remote region we want to specify needs its own subnet.

IMPORTANT: You should not take the next free network of 172.16/12 as 172.17.0.0/16 is dockers internal network default which will lead to problems.

To have a clear separation between our master and our remote regions we will take 10.128.0.0/16 as our remote regions subnet. Also, we will use a map variable to assign the networks to regions. This will make it easier when adding additional regions in the future.

The locals section will now look like this

#...

// lets define variables which are shared between all regions
locals {
  ssh_public_key_file = "~/.ssh/id_rsa.pub"
  cluster_name        = "my-dcos-demo"
  admin_ips           = ["${data.http.whatismyip.body}/32"]

  region_networks = {
    // dont use 172.17/26 as its used by docker.
    "master" = "172.16.0.0/16" // this is the default
    "usw2"   = "10.128.0.0/16"
  }
}

#...

Allowed internal networks

To let our main region allow traffic from the remote region and vice versa we have to specify the accepted_internal_networks variable in both. This variable will inform the security group which allows the agents and masters to communicate to each other. accepted_internal_networks can contain the regions network which makes it extremly easy to let terraform calculate the value for this variable. We will use the values method to retrieve the subnets from locals.region_networks which we previously defined.

The locals section will now look like this:

#...

// lets define variables which are shared between all regions
locals {
  ssh_public_key_file = "<path-to-public-key-file>"
  cluster_name        = "my-dcos-demo"

  region_networks = {
    // dont use 172.17/26 as its used by docker.
    "master" = "172.16.0.0/16" // this is the default
    "usw2"   = "10.128.0.0/16"
  }

  accepted_internal_networks = values(local.region_networks)
}

#...

The remote region

Before we start changing values within the dcos module we will append the infrastructure definition of the remote region to your main.tf. In our example case we only want to have private agents in our remote region, and both private and public agents can be put in a remote region.

IMPORTANT: Running master instances in remote regions is not supported.

To only start private agents we will set num_masters = 0 and num_public_agents = 0. Due to some internal limitation we also have to tell the infrastructure module not preparing load balancers for masters and public agents with lb_disable_public_agents and lb_disable_masters

Another important topic to mention is naming. To distinguish between instances of your main and your remote region we introduced the name_prefix variable which allows you to add a prefix to the name of every resource. In this example we set the name_prefix to the short name of the remote region.

In the following example you will also find the shared config options being used in the module call referenced by e.g. local.admin_ips and the provider we specified for the region

#...

module "dcos-usw2" {
  source  = "dcos-terraform/infrastructure/aws"
  version = "~> 0.3.0"

  admin_ips   = ["${local.admin_ips}"]
  name_prefix = "usw2"

  cluster_name               = local.cluster_name
  accepted_internal_networks = values(local.region_networks)

  num_masters        = 0
  num_private_agents = 1
  num_public_agents  = 0

  lb_disable_public_agents = true
  lb_disable_masters       = true

  ssh_public_key_file = local.ssh_public_key_file
  subnet_range        = local.region_networks["usw2"]

  providers = {
    aws = aws.usw2
  }
}

Peering to the main DC/OS region

We now need to establish a connection between the two infrastructures. The Universal Installer provides a module for this task. In this module we reference data from both infrastructures the main region holding DC/OS masters and the remote region holding your remote private agents.

The only information this module needs to receive is the output of our dcos and dcos-usw2 modules. We will append this module to the end of your main.tf

Here is the example vpc-peering-section

#...

module "vpc-connection-master-usw2" {
  source  = "dcos-terraform/vpc-peering/aws" // module init the peering
  version = "~> 0.3.0"

  providers = {
    aws.local  = aws.master
    aws.remote = aws.usw2
  }

  local_vpc_id        = module.dcos.infrastructure_vpc_id
  local_subnet_range  = local.region_networks["master"]
  remote_vpc_id       = module.dcos-usw2.vpc_id
  remote_subnet_range = local.region_networks["usw2"]
}

Changes to dcos module

At this point its time to do changes to your dcos module so it knows about the remote region and is able to install the remote agents.

Choose a subnet range. In general this change is not needed but we wanted to make your example pretty specific.

subnet_range = local.region_networks["master"]

Add accepted internal networks. Same as in the remote region we need to specify the internal networks to allow internal traffic flow.

accepted_internal_networks = values(local.region_networks)

Change the cluster_name. As this is a shared resource we will make use of the local variable.

cluster_name = local.cluster_name

List the SSH key. This is also a shared resource and we can make use of the local variable

ssh_public_key_file = local.ssh_public_key_file

Add the admin IPs following the same pattern.

`admin_ips = local.admin_ips

Add the private agents. This nearly the most important new variable. This tells the DC/OS installation module which other agents need to be installed.

`additional_private_agent_ips = module.dcos-usw2.private_agents_private_ips

Update the providers section. As we change to explicit alias providers we have to point our dcos module to this specific provider.

providers = {
  aws = aws.master
}

Example dcos module

After the changes above have been applied, your dcos module should look like this

module "dcos" {
  source  = "dcos-terraform/dcos/aws"
  version = "~> 0.3.0"

  cluster_name        = local.cluster_name
  ssh_public_key_file = local.ssh_public_key_file
  admin_ips           = local.admin_ips
  subnet_range        = local.region_networks["master"]

  num_masters        = "1"
  num_private_agents = "2"
  num_public_agents  = "1"

  dcos_version = "2.2"

  dcos_instance_os             = "centos_7.5"
  bootstrap_instance_type      = "m5.large"
  masters_instance_type        = "m5.2xlarge"
  private_agents_instance_type = "m5.xlarge"
  public_agents_instance_type  = "m5.xlarge"

  accepted_internal_networks   = values(local.region_networks)
  additional_private_agent_ips = module.dcos-usw2.private_agents_private_ips

  providers = {
    aws = aws.master
  }

  dcos_variant              = "ee"
  dcos_license_key_contents = "${file("./license.txt")}"
}

Full main.tf example

Here is the complete main.tf you should see once you completed this guide.

provider "aws" {
  # Change your default region here
  region = "us-east-1"
  alias  = "master"
}

provider "aws" {
  # Change your default region here
  region = "us-west-2"
  alias  = "usw2"
}

// lets define variables which are shared between all regions
locals {
  ssh_public_key_file = "~/.ssh/id_rsa.pub"
  cluster_name        = "my-dcos-demo"
  admin_ips           = ["${data.http.whatismyip.body}/32"]

  region_networks = {
    // dont use 172.17/26 as its used by docker.
    "master" = "172.16.0.0/16" // this is the default
    "usw2"   = "10.128.0.0/16"
  }
}

module "dcos" {
  source  = "dcos-terraform/dcos/aws"
  version = "~> 0.3.0"

  cluster_name        = local.cluster_name
  ssh_public_key_file = local.ssh_public_key_file
  admin_ips           = local.admin_ips
  subnet_range        = local.region_networks["master"]

  num_masters        = "1"
  num_private_agents = "2"
  num_public_agents  = "1"

  dcos_version = "2.2"

  dcos_instance_os             = "centos_7.5"
  bootstrap_instance_type      = "m5.large"
  masters_instance_type        = "m5.2xlarge"
  private_agents_instance_type = "m5.xlarge"
  public_agents_instance_type  = "m5.xlarge"

  accepted_internal_networks   = values(local.region_networks)
  additional_private_agent_ips = module.dcos-usw2.private_agents_private_ips

  providers = {
    aws = aws.master
  }

  dcos_variant              = "ee"
  dcos_license_key_contents = "${file("./license.txt")}"
}

# Used to determine your public IP for forwarding rules
data "http" "whatismyip" {
  url = "http://whatismyip.akamai.com/"
}

output "masters-ips" {
  value = module.dcos.masters-ips
}

output "cluster-address" {
  value = module.dcos.masters-loadbalancer
}

output "public-agents-loadbalancer" {
  value = module.dcos.public-agents-loadbalancer
}

module "dcos-usw2" {
  source  = "dcos-terraform/infrastructure/aws"
  version = "~> 0.3.0"

  admin_ips   = local.admin_ips
  name_prefix = "usw2"

  cluster_name               = local.cluster_name
  accepted_internal_networks = values(local.region_networks)

  num_masters        = 0
  num_private_agents = 1
  num_public_agents  = 0

  lb_disable_public_agents = true
  lb_disable_masters       = true

  ssh_public_key_file = local.ssh_public_key_file
  subnet_range        = local.region_networks["usw2"]

  providers = {
    aws = aws.usw2
  }
}

module "vpc-connection-master-usw2" {
  source  = "dcos-terraform/vpc-peering/aws" // module init the peering
  version = "~> 0.3.0"

  providers = {
    aws.local  = aws.master
    aws.remote = aws.usw2
  }

  local_vpc_id        = module.dcos.infrastructure_vpc_id
  local_subnet_range  = local.region_networks["master"]
  remote_vpc_id       = module.dcos-usw2.vpc_id
  remote_subnet_range = local.region_networks["usw2"]
}

Multi-Region DC/OS on AWS using the Universal Installer

Guide for DC/OS on AWS using the Universal Installer adding a remote region.