Distributed Process Management

Understanding distributed process management in a DC/OS cluster

This section describes the management of processes in a DC/OS cluster, from the resource allocation to the execution of a process. At a high level, this interaction takes place between the DC/OS components when you launch a process. Communication occurs between the different layers, such as the user interacting with the scheduler, and within a layer, for example, a master communicating with agents.

Concept of distributed process management in DC/OS

Figure 1. Distributed process management in DC/OS

Here is an example using the Marathon service and a user launching a container based on a Docker image:

Example of distributed process management in DC/OS

Figure 2 - Distributed process management in DC/OS using Marathon and Docker

The chronological interaction between the above components looks like this. Notice that Executors and Task are folded into one block since in practice this is often the case:

Sequence diagram for distributed process management in DC/OS

Figure 3 - Sequence of distributed process management in DC/OS

In detail, here are the steps:

  1. Client/Scheduler initialization. In this step, the Client needs to know how to connect to the Scheduler to launch a process, for example via Mesos-DNS or DC/OS CLI.
  2. Mesos master sends resource offer to Scheduler; the resource offers are based on cluster resources managed through agents and the DRF algorithm in Mesos master.
  3. Scheduler declines resource offers because no process requests from Clients are pending. As long as no clients have initiated a process, the scheduler will reject offers from the master.
  4. Client initiates process launch. For example, this could be a user creating a Marathon app using the DC/OS Services tab or via the HTTP endpoint /v2/app.
  5. Mesos master sends the resource offers . For example, cpus(*):1; mem(*):128; ports(*):[21452-21452].
  6. If resource offer matches the requirements the Scheduler has for the process, it accepts the offer and sends a launchTask request to Mesos master.
  7. Mesos master directs Mesos agents to launch tasks.
  8. Mesos agent launches tasks via Executor.
  9. Executor reports task status to Mesos agent.
  10. Mesos agent reports task status to Mesos master.
  11. Mesos master reports task status to scheduler.
  12. Scheduler reports process status to client.