Continuous Integration and Deployment


Overview

Based on Tekton, Datapio provides a CI/CD platform with:

  • project management
  • webhook generation
  • concurrency settings
  • resources creation and garbage collection
  • history limitation
  • pipeline as code support

Setup

Prerequisites:

Get the source code

The source code is hosted on Github:

$ git clone https://github.com/datapio/opencore.git datapio-opencore
$ cd datapio-opencore

Create a dedicated namespace

Using your preferred method, create the following namespace:

---
apiVersion: v1
kind: Namespace
metadata:
  name: datapio-control-plane

Install operators

First, the PipelineRunServer Operator:

$ helm install \
    pipelinerunserver-operator \
    ./charts/pipelinerunserver-operator \
    --namespace datapio-control-plane
ValuesDescription
replicaCountNumber of replicas in the cluster (only one supported)
operator.image.nameDocker image name for the operator itself
operator.image.tagDocker image version
operator.image.pullPolicyKubernetes Pull Policy for image
archiver.image.nameDocker image name for the archiver
archiver.image.tagDocker image version
archiver.image.pullPolicyKubernetes Pull Policy for image
worker.image.nameDocker image name for the workers
worker.image.tagDocker image version
worker.image.pullPolicyKubernetes Pull Policy for image
rabbitmq.url_secretName of Kubernetes Secret containing a key url
rabbitmq.admin_secretName of Kubernetes Secret containing the keys protocol, host, port, username, password, and vhost

See default values here

Then, the Project Operator:

$ helm install \
    datapio-project-operator \
    ./charts/project-operator \
    --namespace datapio-control-plane
ValuesDescription
replicaCountNumber of replicas in the cluster (only one supported)
operator.image.nameDocker image name for the operator itself
operator.image.tagDocker image version
operator.image.pullPolicyKubernetes Pull Policy for image
tasks.git.image.nameDocker image name for the Tekton Task git cloning your repository
tasks.git.image.tagDocker image version
tasks.pipelineExec.image.nameDocker image name for the Tekton Task executing your pipeline’s code
tasks.pipelineExec.image.tagDocker image version
tasks.yarn.image.nameDocker image name for for the Tekton Task yarn installing your pipeline’s dependencies)
sidecars.docker.image.nameDocker image name for the tasks sidecar container running a Docker server
sidecars.docker.image.tagDocker image version
vault_secretName of Kubernetes Secret containing the keys VAULT_ADDR, VAULT_ROLE and VAULT_K8S_MOUNT_POINT

See default values here