Terraform and Gitlab CI integration

This article documents the process of integrating gitlab CI with terraform. The purpose is so that any change in terraform files (.tf files) will trigger gitlab CI pipeline to provision the cloud infrastructure. The problems encountered along the way are also documented at the end of this article.

Step one: add Service Account

Go to GCP IAM & Admin dashboard, click Service Accounts menu, then click “Create Service Account”, you will see this screen.

After the above step is done and Service Account is created, in your laptop, go to terraform folder, which is workspace/access_management/stg, and run :

terraform plan
terraform apply

to add terraform permission to the SA.

Step two: add the Service Account to gitlab CI

Go to GCP IAM & Admin dashboard, click Service Accounts menu, click on the SA, click Keys, you will see this screen

Click Add key , and then you will get to download a json format private key to your laptop. Keep the file.

After you have the json format key, you go to gitlab ci webpage. Click Settings->CI/CD, and expand the Variables section.

Add TERRAFORM_SA_FILE variable, and add the content of the json key to the variable.

Step three:

Check the .gitlab-ci.yml , if you have a new gitlab ci yaml file, you add the yaml file to this .gitlab-ci.yml file.

Step four:

Check the gitlab ci yaml file, we use the below as an example

plan-stg-airflow_project:
  environment:
    url: workspaces/airflow_aerotrade/
extends:
  - .stg-env-tpl
  - .tf-build-tpl
only:
  refs:
    - merge_requests
    - master
  changes:
    - “workspaces/access_management/stg/*”
    - “workspaces/airflow_project/stg/*”
    - “modules/gcs_provider/*”
    - “modules/google_apis/*”
    - “modules/airflow/*”
    - “modules/compute/*”
    - “workspaces/compute/stg/*”
except:
  - triggers

The task plan-stg-airflow_project extends .stg-env-tpl and .tf-build-tpl, and refers to merge_requests and master branch, and listen to changes of the “workspaces/compute/stg” etc.

Any change in “workspaces/compute/stg” etc. of master branch and merge_requests will call the task in the gitlab ci pipeline.

More info on gitlab ci yaml file is shown below:

.tf-build-tpl:
  extends: .tf-base-build
only:
  - merge_requests
  - master
  - branches

.tf-base-build:
  extends: .tf-infra-tpl
  stage: build
  script:
    - echo “tf-base-build”
    - echo ${CI_ENVIRONMENT_URL}
    - echo ${CI_ENVIRONMENT_NAME}
    - cd ${CI_ENVIRONMENT_URL}/${CI_ENVIRONMENT_NAME}
    - ls -l
    - md5sum ${GOOGLE_APPLICATION_CREDENTIALS}
    - terraform init
    - terraform validate
    - export PLAN=current_plan.txt
    - terraform plan -lock=false -out=$PLAN
  artifacts:
    name: plan
    paths:
      - current_plan.txt

The .tf-build-tpl extends into .tf-base-build, and .tf-base-build further extends into .tf-infra.tpl

.tf-infra-tpl:
  tags:
    - blockchain
image:
  name: hashicorp/terraform:0.13.1
  entrypoint: [“”]
before_script:
  - echo “tf-infra-tpl”
  - export TF_IN_AUTOMATION=true
  - terraform version
  - echo ${TERRAFORM_SA_FILE} | base64 -d > creds.json
  - export GOOGLE_APPLICATION_CREDENTIALS=”${PWD}/creds.json”

The .tf-infra-tpl has before_script condition that exports GOOGLE_APPLICATION_CREDENTIALS. The GOOGLE_APPLICATION_CREDENTIALS is coming from ${TERRAFORM_SA_FILE}.

Terraform gitlab ci problems:

This happens when the state file is accessed by multiple terraform clients using different versions. It could be the initial state file was created by your local terraform and the ci-cd version of terraform is different, such as local is v1.0.1 , gitlab ci version is v0.13.1

This happens when the support email is not in the google group. Only Admins can check/add on the group ownership. Owner cannot do so. Solution is to ask IT to add the support email to group ownership.

This happens when the service account is not created or terraform storage access permission is not added to the service account.

This happens because Jobs need to share the same limitations. So the “only:” portion needs to be the same in apply and plan tasks. For example, the code below showing the only: portion

only:
  changes:
    - workspaces/access_management/prd/*
    - modules/gcs_provider/*
    - modules/google_apis/*
    - modules/ro_access/*
    - modules/oauth/*