How to define a test pipeline to be reused with multiple setups

Goals

The potential users of a software are likely to be using it from within very different system setups. This means that the software should work with e.g. different compilers or third-party library versions, etc.

Using a base job

This is a very fairly easy to understand approach, but we believe the later (Using a trigger job) to be better in general due to better scaling properties.

In the GitLab CI , this can easily be tested by running a test pipeline with different container images that represent different user setups. This can be easily defined by a base job and multiple derivatives using different images. For instance, a .gitlab-ci.yml file that achieves this could look like this:

stages:
  - test

# base job
.test:
  stage: test
  script:
    - echo "Running tests"

# derived job using image with full setup
test-setup-1:
  extends: .test
  image: $CI_REGISTRY/my-group/my-containers/full-setup

# derived job using image with minimal setup
test-setup-2:
  extends: .test
  image: $CI_REGISTRY/my-group/my-containers/minimal-setup

In some cases, it may be beneficial to split the compilation of the test suite from the actual execution. With the above example, we could do the following:

stages:
  - build
  - test

.build:
  stage: build
  script:
    - echo "Building tests"

.test:
  stage: test
  script:
    - echo "Running tests"

build-setup-1:
  extends: .build
  image: $CI_REGISTRY/my-group/my-containers/full-setup

build-setup-2:
  extends: .build
  image: $CI_REGISTRY/my-group/my-containers/minimal-setup

test-setup-1:
  extends: .test
  image: $CI_REGISTRY/my-group/my-containers/full-setup
  needs:
    - job: build-setup-1

test-setup-2:
  extends: .test
  image: $CI_REGISTRY/my-group/my-containers/minimal-setup
  needs:
    - job: build-setup-2

The needs statements are there because in an actual pipeline you would pass the built tests as artifacts to the test job. Clearly, the above way of defining our pipelines does not scale well, as we need to define two new jobs for each setup that we want to test. Therefore, a better approach may be to use a trigger job:

Using a trigger job

gitlab-ci.yml

stages:
  - trigger-test-pipelines

.base-trigger:
  stage: trigger-test-pipelines
  trigger:
    include: .pipeline.yml
    strategy: depend

setup-1:
  extends: .base-trigger
  variables:
    IMAGE: $CI_REGISTRY/my-group/my-containers/full-setup

setup-2:
  extends: .base-trigger
  variables:
    IMAGE: $CI_REGISTRY/my-group/my-containers/minimal-setup

.pipeline.yml

Here, .pipeline.yml contains the actual test pipeline that is then executed with the different container images that we define by the ‘IMAGE’ variable.

This is what .pipeline.yml may look like:

default:
  image: $IMAGE

stages:
  - build
  - test

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE=="parent_pipeline"

build:
  stage: build
  script:
    - echo "Building tests"

test:
  stage: test
  script:
    - echo "Running tests"
  needs:
    - job: build

This approach scales well for adding

  • new setups to be tested as well as
  • new jobs or stages in the actual test pipeline.

See also