GitOps takes DevOps best practices used for application development (such as version control and CI/CD) and applies them to infrastructure automation. In GitOps, the Git repository serves as the source of truth and the CD pipeline is responsible for building, testing, and deploying the application code and the underlying infrastructure.
Nowadays, an application is not just code running on infrastructure that you own and operate. It is usually a set of first-party and third-party microservices working together in an event-driven architecture or with a central service orchestrator such as Workflows.
From our partners:
Service orchestrations, which have their own definition files and deployment cycles, can benefit from a GitOps approach. This blog post describes how to set up a simple Git-driven development, testing, and deployment pipeline for Workflows using Cloud Build.
Architecture
Let’s take a look at the overall approach.
In this approach, you have a staging branch where you make changes to a workflow. This triggers a Cloud Build configuration that deploys a test staging workflow and runs some staging tests against it. If all tests pass, Cloud Build deploys the staging workflow. After more manual testing of the staging workflow, you merge changes from the staging branch to the main branch. This triggers the same Cloud Build configuration to deploy a test production workflow, run more production tests, and if all tests pass, deploy the production workflow.
This approach enables you to have an automated and staged rollout of workflow changes with tests along the way to minimize risk.
Configuration
Setting up such an automated workflow deployment pipeline is straightforward.
First, you need a workflow definition file that can benefit from such automation. You can use one of your workflow definition files or workflow.yaml
, which simply returns Hello World
.
Next, define a Cloud Build configuration file (see cloudbuild.yaml
) with all the stages. In this configuration, Cloud Build deploys a test workflow with the branch name and commit hash, runs the workflow and captures the output, deletes the test workflow, and tests the workflow with the supplied test script. If all the tests pass, it deploys the final workflow in the branch.
Tests for the branch are defined in appropriate test-{branchname}.sh
files. For example, test-staging.sh
runs against the workflows deployed in the staging branch and only checks the workflow execution state. On the other hand, test-main.sh
runs against the main branch, checks the workflow execution state, and also checks the output of the execution. You can add more tests as you see fit.
Connect your repository to Cloud Build
Now that you have the basic configuration in place, you connect your (or workflows-demos
) repository to Cloud Build before creating triggers. Follow the instructions here.
Create a Cloud Build trigger
You now create a Cloud Build trigger to watch for commits to the main and staging branches. General instructions are here.
Go to the Create Trigger
section of Cloud Build in the console and create a trigger with the following properties:
- Name:
workflows-trigger
- Event: Push to a branch
- Repository:
GoogleCloudPlatform/workflows-demos
- Branch:
^main$|^staging$
- Included files filter:
gitops/workflow.yaml
- Configuration type: Cloud build configuration file
- Cloud build configuration file location:
gitops/cloudbuild.yaml
- Add a Substitution variable with key/value:
_WORKFLOW_NAME
andworkflows-gitops
Test the staging workflow
You’re now ready to test the build pipeline with the staging branch.
Switch to the staging branch:
git checkout staging
Change Hello World
in workflow.yaml
to Bye World
:
- init:
assign:
- - message: "Hello World"
+ - message: "Bye World"
Commit and push the change to the staging branch:
git add workflow.yaml
git commit -m "Update workflow.yaml in staging"
git push
You should see the trigger running:
After a few seconds, the build (including all its stages) is successful:
And a staging workflow has been deployed:
Test the production workflow
Once you’re ready to deploy the staging workflow to production, simply merge the staging branch to the main branch.
git checkout main
git merge staging
git push
In this case, however, the build fails:
This is because the test script for the production workflow in test-main.sh
is expecting to see Hello World
as output of the workflow.
You need to go back to the staging branch, and change Bye World in workflow.yaml
back to Hello World
:
- init:
assign:
- - message: "Bye World"
+ - message: "Hello World"
Check in your changes to staging, see the build succeed, and merge to main. Finally, you should also see the build succeed and see that a production workflow has been deployed alongside staging:
Next steps
This post covered how to set up a Git-driven development, testing, and deployment pipeline for Workflows using Cloud Build. All the details and sample configuration files are in our workflows-demos/gitops
repository.
Of course, Cloud Build is not the only way to set up such a pipeline. GitHub Actions is another useful tool that can help to set up similar service orchestration pipelines. Feel free to contribute to our repository with GitHub Actions based pipelines and reach out to me on Twitter @meteatamel for any questions or feedback.
By: Mete Atamel (Developer Advocate)
Source: Google Cloud Blog
For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!
Our humans need coffee too! Your support is highly appreciated, thank you!