Project Deployments

Kommander Projects can be configured with GitOps based Continuous Deployments for federation of your Applications to associated clusters of the project. This is backed by Dispatch which enables software and applications to be continuously deployed (CD) using GitOps processes. GitOps enables the application to be deployed as per a manifest that is stored in a Git repository. This ensures that the application deployment can be automated, audited and declaratively deployed to the infrastructure.

What is GitOps?

GitOps is a modern software deployment strategy. The configuration describing how your application is deployed to a cluster are stored in a Git repository. The configuration is continuously synchronized from the Git repository to the cluster, ensuring that the specified state of the cluster always matches what is defined in the “GitOps” Git repository.

The benefits of using a GitOps deployment strategy are:

• Familiar, collaborative change and review process. Engineers are intimately familiar with Git-based workflows: branches, pull requests, code reviews, etc. GitOps leverages this experience to control deployment of software and updates to catch issues early.
• Clear change log and audit trail. The Git commit log serves as an audit trail to answer the question: “who changed what, and when?” Having such information available you can contact the right people when fixing or prioritizing a production incident to determine the why and correctly resolve the issue as quickly as possible. Additionally, Dispatch’s CD component (Flux CD) maintains a separate audit trail in the form of Kubernetes Events, as changes to a Git repository don’t include exactly when those changes were deployed.
• Avoid configuration drift. The scope of manual changes made by operators expands over time. It soon becomes difficult to know which cluster configuration is critical and which is left over from temporary workarounds or live debugging. Over time, changing a project configuration or replicating a deployment to a new environment becomes a daunting task. GitOps supports simple, reproducible deployment to multiple different clusters by having a single source of truth for cluster and application configuration.

That said, there are some cases when live debugging is necessary in order to resolve an incident in the minimum amount of time. In such cases, pull-request-based workflow adds precious time to resolution for critical production outages. Dispatch’s CD strategy supports this scenario by letting you disable the Auto Sync feature. After Auto Sync is disabled, Dispatch will stop synchronizing the cluster state from the GitOps git repository. This lets you use kubectl, helm or whichever tool you need to resolve the issue.

Prerequisites

This procedure requires that Dispatch is installed. Optionally you can configure Dispatch to be deployed in “CD-only” mode. Modify the cluster.yaml file as follows:

- name: dispatch
  enabled: true
  values: |
    ci:
      enabled: false

Continuous Deployments

After enabling Dispatch and configuring your project and its clusters, navigate to the Continuous Deployment (CD) tab under your Project. Here you create a GitOps source which is an SCM repository hosting the application definition. D2IQ recommends that you create a secret first then create a GitOps source accessed by the secret:

Setting up a secret for accessing GitOps source.

Create a secret that Dispatch uses to read the contents of your GitOps repository by creating SCM webhooks:

Refer to this guide to learn more about setting up credentials for Dispatch. Note that the above screen creates a types.kubefed.io/v1beta1, Kind=FederatedSecret and this is not yet supported by Dispatch CLI. Use the GUI, as shown above, to create a federated secret or create a FederatedSecret manifest and apply it to the project namespace. Learn more about FederatedSecrets.

Creating GitOps Source

After the secret is created, you can view it in the Secrets tab. Configure the GitOps source accessed by this secret:

While the username and password are mandatory fields, SCM details default to github if unspecified. It takes a few seconds for the GitOps Source to be reconciled and the manifests from the SCM repository at the given path to be federated to attached clusters. After the sync is complete, manifests from GitOps source are created in attached clusters.

After the manifests are federated, there are various commands that can be executed from the CLI to check the federation status.

On the management cluster, check for your FederatedKustomization and FederatedGitRepository resources. The status field reflects the propagation of manifests:

$ kubectl get federatedkustomization -n<PROJECT_NAMESPACE> <GITOPS_SOURCE_NAME> -oyaml
...
...
status:
  clusters:
  - name: attached-cluster
  conditions:
  - lastTransitionTime: "2021-01-27T02:06:03Z"
    lastUpdateTime: "2021-01-27T02:06:03Z"
    status: "True"
    type: Propagation
  observedGeneration: 1
...
...

If there are errors in federation, those events are populated in the status field of the FederatedKustomization and or FederatedGitRepository resources.

Suspending GitOps Source

Some times in production, you may want to suspend the auto sync between the GitOps repository and the associated clusters. This live debugging may be necessary to resolve an incident in the minimum amount of time without the burden of pull request based workflows. Select Suspend to manually suspend the GitOps reconciliation:

This lets you use kubectl, helm or another tool to resolve the issue. After the issue is resolved select Resume to sync the updated contents of GitOps source to associated clusters:

Similar to Suspend/Resume, you can use the Delete action to remove the GitOps source. Removing the GitOps source results in removal of all the manifests applied from the GitOps source.

You can have more than one GitOps Source in your Project to deploy manifests from various sources.

Troubleshooting

• Events related to federation are stored in respective FederatedGitRepository and or FederatedKustomization resources.
• View the events and or logs for deployments/dispatch-repository-controller in dispatch namespace in case there are any unexpected errors.
• Enabling the Dispatch repository controller for your project namespace will cause a number of related Flux controller components to be deployed into the namespace. These are necessary for the proper operation of the repository controller and should not be removed.

View Helm Releases

In addition to viewing the current GitOps Sources, you can also view the current Helm Releases that have been deployed.

On the Continuous Deployment (CD) page, select the Helm Releases button. All of the current Helm Release charts are displayed with their Chart Version and the names of the clusters. In this example daily is the name of the current cluster being displayed.