Make your CI workflow portable

Problems with existing CIs

• The CI code is sticky to the underlying CI infrastructure.
• It's hard to migrate the CI from a runner to another.
• The CI syntax is different with each CI runner.
• Most CI workflows are represented using YAML.

Implement the CI workflow using dagger solves all of those problems.

Benefits of defining your CI workflow in Dagger

• Your CI only runs dagger. It moves all the CI logic from the non-portable CI syntax, to a dagger plan, that runs the same way everywhere.
• Develop and run your CI workflow locally. No need to create a Pull Request to trigger CI, you can run the workflow locally. Since dagger workflows are containerized, you can expect the same results no matter where the workflow is executed.
• Write once, Run anywhere. The same workflow can run in any CI, goodbye vendor lock in.
• Blazing Fast: Dagger will automatically build an optimized execution graph, completing the job as fast as possible. Spend less time staring at CI to complete and reduce costs.
• Effortless Cache Optimizations. Dagger will automatically re-use cached execution results if no changes are detected. Made a change to the frontend? The backend won't be built again.
• Composable & Reusable. Define re-usable steps and share them across all your projects. Or with the world. Dagger ships with dozens of re-usable components

Example

This example illustrates how to use Dagger to test and lint a Go project. The use of Go is irrelevant and the example can be easily adapted to other languages.

From the project repository, create a file named ci/main.cue and add the following configuration to it.

ci/main.cue

package main

import (
    "alpha.dagger.io/dagger"
    "alpha.dagger.io/os"
    "alpha.dagger.io/docker"
)

// Source directory of the repository. We'll connect this from the CLI using `dagger input`
source: dagger.#Artifact

// Here we define a test phase.
// We're using `os.#Container`, a built-in package that will spawn a container
// We're also using `docker.#Pull` to execute the container from a Docker image
// coming from a registry.
test: os.#Container & {
    image: docker.#Pull & {
        from: "golang:1.16-alpine"
    }
    mount: "/app": from: source
    command: "go test -v ./..."
    dir:     "/app"
}

// Likewise, here we define a lint phase.
lint: os.#Container & {
    image: docker.#Pull & {
        from: "golangci/golangci-lint:v1.39.0"
    }
    mount: "/app": from: source
    command: "golangci-lint run -v"
    dir:     "/app"
}

The configuration above defines:

• source code of the project. More on this later.
• test task which executes go test inside the source artifact using the golang Docker image
• lint task which executes golangci-lint inside the source artifact using the golangci-lint Docker image.

Before we can execute the configuration, we need to set up the Dagger project and environment.

# Initialize a dagger project at the root of your project
dagger init

# Create a CI environment using the CUE files in the `./ci` directory
dagger new ci -p ./ci

# Link the `source` artifact defined in the configuration with the project
# source code.
dagger input dir source .

Next, bring up the CI environment (e.g. execute the CI configuration):

$ dagger up
# ...
7:15PM INF test | computing    environment=ci
7:15PM INF lint | computing    environment=ci
# ...

Since test and lint do not depend on each other, they are executed in parallel.

Running dagger up a second time will return almost immediately: since no changes were made to source, Dagger will re-use the cached results for both test nor lint.

Integrating with CI

All it takes to execute a Dagger workflow in CI is to run dagger up.

We provide a GitHub Actions to make integration with GitHub easier.

Monorepos

Dagger workflows scale really well with CI complexity.

The following example illustrates how to test two different projects in the same repository: a Node frontend (located in ./frontend) along with a Go backend (located in ./backend).

From the project repository, update the file named ci/main.cue with the following configuration.

ci/main.cue

package main

import (
    "alpha.dagger.io/dagger"
    "alpha.dagger.io/os"
    "alpha.dagger.io/docker"
)

// Source directory of the repository. We'll connect this from the CLI using `dagger input`
source: dagger.#Artifact

backend: {
    // We use `os.#Dir` to grab a sub-directory of `source`
    code: os.#Dir & {
        from: source
        path: "./backend"
    }

    test: os.#Container & {
        image: docker.#Pull & {
            from: "golang:1.16-alpine"
        }
        mount: "/app": from: code
        command: "go test -v ./..."
        dir:     "/app"
    }
}

frontend: {
    // We use `os.#Dir` to grab a sub-directory of `source`
    code: os.#Dir & {
        from: source
        path: "./frontend"
    }

    test: os.#Container & {
        image: docker.#Pull & {
            from: "node:16-alpine"
        }
        mount: "/app": from: code
        command: """
            # Install Dependencies
            yarn

            # Run the test script
            yarn test
            """
        dir: "/app"
    }
}

tip

Larger configurations can be split into multiple files, for example backend.cue and frontend.cue

The configuration above defines a frontend and backend structure, each containing:

• A code directory, defined as a subdirectory of source
• A language specific test task using either yarn or go

$ dagger up
7:15PM INF frontend.test | computing    environment=ci
7:15PM INF backend.test | computing    environment=ci
7:15PM INF frontend.test | #8 0.370 yarn install v1.22.5    environment=ci
7:15PM INF frontend.test | #8 0.689 [1/4] Resolving packages...    environment=ci
7:15PM INF frontend.test | #8 1.626 [2/4] Fetching packages...    environment=ci
...

frontend.test and backend.test are running in parallel since there are no dependencies between each other.

If you were to make changes to the ./frontend directory, only frontend.test will be executed.