Building An Automated Serverless Deployment Pipeline With Cloud Build

You’ve got a shiny new application ready to deploy to the cloud. After researching your options, you land on using Cloud Run with Cloud Build to build and push your containerized application code to an Artifact Registry repository. In three steps, using a Dockerfile and a Cloud Build configuration you build your container, push it to Artifact Registry, and deploy it to Cloud Run:

steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'us-central1-docker.pkg.dev/my-project/my-app-repo/shiny-new-app', '.']

# Push the container image to Artifact Registry
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'us-central1-docker.pkg.dev/my-project/my-app-repo/shiny-new-app']

# Deploy container image to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: gcloud
  args: ['run', 'deploy', 'my-serverless-app', '--image', 'us-central1-docker.pkg.dev/my-project/my-app-repo/shiny-new-app', '--region', 'us-central1']

images:
- us-central1-docker.pkg.dev/my-project/my-app-repo/shiny-new-app

The example above combines the build, push and deployment steps into one Cloud Build job. This blog will show you what this could look like as a series of manual deployment steps, and how it can be developed into an automatic serverless deployment pipeline that can be used as a jumping off point for more complex solutions. We’ll be using Cloud Build, Artifact Registry, Pub/Sub and Cloud Run.

We’ll use the open source GitHub project, Emblem, to model a working reference of Google Cloud serverless architecture. References to Emblem will be marked with the💠emoji.

A manual pipeline

Let’s start by examining the manual steps to deploy a containerized application to Cloud Run.

1. First, you make application code changes to your repository’s main branch.
2. When an application change is merged, you use Cloud Build to build a new container.
3. After a successful build, Cloud Build pushes the newly built container to Artifact Registry.
4. You update Cloud Run with a new configuration pointing to the new build.
5. Cloud Run deploys a new revision of your service. Your code changes are now deployed.

Of course, you would need to go through these steps each time there are changes to your application code. That’s not practical and can turn into a logistical nightmare for a team making continuous updates to the code. Not to mention the added complexity of staging changes to multiple environments or incorporating systematic testing or incremental rollouts. Let’s see how you can automate your lovely little pipeline by looking at it as two parts: the build and the deployment.

Automate the build

To automate the build step of your pipeline, Cloud Build should build and push when a change is committed to the application code in your repository. Here’s what’s needed to make this happen:

1. Connect your GitHub repository to your Cloud project

By connecting your GitHub repository to your project, Cloud Build can use repository events to initiate a Cloud Build trigger. Common repository events are supported including pushing to a specific branch, pushing a new tag, and creating a pull request.

2. Include a Cloud Build yaml configuration in your repository

You can configure a Cloud Build job with a build config file. This YAML file provides task-level instructions to Cloud Build. This file can live alongside your application’s Dockerfile, or in a separate directory in your repository. For an automatic build, your config file will tell Cloud Build to build the container image and push it to Artifact Registry.

💠The Emblem project continuously builds multiple containers and keeps corresponding build config files in a centralized ops/ directory. This allows for the separation of ownership of the Cloud Build configs and the application code they may build.

3. Create a Cloud Build trigger

A Cloud Build trigger can be invoked each time a change is pushed to your main branch. Its configuration will require the GitHub repository to be connected to the Google Cloud project, the name of the branch you want to use, and the path to the Cloud Build configuration file in the repo. The invocation of the Cloud Build trigger can be narrowed down further by specifying files and directories to include or ignore, so that a new build can be created only when certain files have changed.

💠 The automatic build triggers featured in Emblem use a Cloud Build config file that builds the container and pushes it to Artifact Registry:

Loading…

steps:
  # Docker Build 
  - name: 'gcr.io/cloud-builders/docker'
    args: 
      - 'build'
      - '-t'
      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/content-api/content-api:${_IMAGE_TAG}'

# Default to us-central1
substitutions:
  _REGION: us-central1
  _IMAGE_TAG: $SHORT_SHA

# Store in Artifact Registry
images:
  - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/content-api/content-api:${_IMAGE_TAG}'

The variables prefixed by an underscore (_) allow for substitutions to be provided when configuring the Cloud Build trigger. In the example above, _REGION can be overridden allowing this configuration to be used unchanged even if the container registry is moved to a new location. Substitutions without an underscore, such as $PROJECT_ID, are built-in and have values provided by Cloud Build. You can see the list of built-in substitutions in the documentation. This is helpful for using a single Cloud Build config for multiple triggers that have a similar function.

Automate the deployment

With the manual pipeline, you know when a new build is pushed so you can faithfully update the Cloud Run service yourself. For this to work automatically, there needs to be some way of signaling Cloud Run that a new build is available. You can do this with a little help from Pub/Sub and another Cloud Build trigger. Let’s look at this in more detail:

1. The “gcr” Pub/Sub topic

If your Google Cloud project includes a Pub/Sub topic named “gcr”, Artifact Registry will publish messages about changes in its repositories. A message will be published every time an image build is pushed, tagged, or deleted. These messages are delivered by a corresponding Pub/Sub subscription to your application or in our case, to a Cloud Build trigger.

2. Create another Cloud Build trigger

A second Cloud Build trigger is configured to deploy a new revision of your Cloud Run service. In addition to repository events, Cloud Build triggers support Pub/Sub events. You can select the gcr Pub/Sub topic as the trigger event to create a corresponding subscription. With that, your Cloud Run service will be updated automatically when Artifact Registry publishes a message to Pub/Sub.

While it is possible to have a single Cloud Build trigger build, push and deploy your application, separating the deployment from the build and the push allows each stage to run in a separate Cloud Build job and make it easier to develop each piece of the pipeline independently from the other.

💠 Emblem features two separate Cloud Build triggers that automatically deploy the website and content-api services to Cloud Run. They share a common Cloud Build config file:

steps:
  # Print the full Pub/Sub message for debugging.
  - name: gcr.io/cloud-builders/gcloud
    entrypoint: /bin/bash
    args:
      - '-c'
      - |
        echo ${_BODY}
  # Cloud Run Deploy
  - name: gcr.io/cloud-builders/gcloud
    args:
      - run
      - deploy
      - ${_SERVICE}
      - --image=${_IMAGE_NAME}
      - --region=${_REGION}
      - --revision-suffix=${_REVISION}
      - --project=${_TARGET_PROJECT}
      - --allow-unauthenticated

Once again, the config file uses variables for which values are provided via the trigger’s substitution variable settings. Values for certain variables, such as _BODY, _IMAGE_NAME and _REVISION are evaluated using the message received from the gcr Pub/Sub topic, while others are hardcoded:

Why stop there?

The value of this pipeline lies not in its simplicity, but in its potential to be developed further to include more functionality, such as staging changes in a separate Google Cloud project, incorporating automatic testing for each change to your application, or doing incremental rollouts to your Cloud Run service. These can all be achieved with a combination of additional Cloud Build triggers and Pub/Sub topics. Alternatively, with the recent addition of Cloud Run support, Cloud Deploy can be used as a delivery pipeline that will deploy to a Cloud Run target complete with rollbacks, approval, audit and delivery metrics.

💠Emblem features a more advanced automatic deployment pipeline based on this model. It includes multiple Google Cloud projects to support staging changes between multiple environments and incremental canary rollouts to production:

To see this in action, visit the Emblem GitHub repository and deploy the Emblem sample application yourself. For a step-by-step tutorial that will deploy this pipeline, The Squire’s guide to automated deployments with Cloud Build.

By: Roger Martinez (Developer Relations Engineer) and Patricia Shin (Cloud Developer Relations Engineer)
Source: Google Cloud Blog