aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Software Engineering
  • Solutions
  • Technology

Avoiding GCF Anti-Patterns Part 5: How To Run Background Processes Correctly In Python

  • aster.cloud
  • November 23, 2021
  • 3 minute read

Editor’s note: Over the past several weeks, we’ve posted a series of blog posts focusing on best practices for writing Google Cloud Functions based on common questions or misconceptions as seen by the Support team.  We refer to these as “anti-patterns” and offer you ways to avoid them.  This article is the fifth post in the series.

Scenario

You see finished with status: 'timeout' in the logs before a background process has completed in your Python Function.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

Most common root issue

Although this timeout error can happen for Functions using any runtime, we most often see this issue occur when Python developers try to use os.fork() or  multiprocessing.Process() in their Cloud Function.

Why you should try to avoid async work in a Function:

A background task started by a Cloud Function is not guaranteed to complete. As soon as the Functions completes, e.g. the Function returns or a timeout error occurs, the Function instance can be terminated at any time. You can read more about the Function execution timeline in the documentation.

We often see customers test their functions locally where these execution timeouts do not exist. Additionally, customers’ local machines may be more powerful than what they have provisioned for their Cloud Functions. Customers may see these multiprocessing scenarios working locally and therefore assume their code will work in the same way in the Cloud Function instance.

For Python developers who require such async operations, we suggest using Cloud Tasks Service instead to schedule the background operation. See example below.

Using Cloud Tasks in a Python Cloud Function

The following Function demonstrates how you can use Cloud Tasks to schedule an async operation. This example shows a Cloud Function (named “create_task“) that creates a Cloud Task to invoke another Cloud Function that will run the background task. You can learn more about creating HTTP target tasks.

Read More  Building The Future Of Work With Google Workspace
"""Create a task for a given queue with an arbitrary payload."""
from flask import escape
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime
import json


def create_task(request):

   # Create a client.
   client = tasks_v2.CloudTasksClient()

   # TODO(developer): Uncomment these lines and replace with your values.
   project = '<PROJECT_ID>'
   queue = 'my-queue'
   location = '<your-region>'
   url = '<url-to-Cloud-Function-to-run-background-task>'
   payload = {'message': 'Hello from Cloud Tasks'}
   in_seconds = 60

   # Construct the fully qualified queue name.
   parent = client.queue_path(project, location, queue)

   # Construct the request body.
   task = {
       "http_request": {  # Specify the type of request.
           "http_method": tasks_v2.HttpMethod.POST,
           "url": url,  # The full url path that the task will be sent to.
           "oidc_token": {
               "service_account_email": "<your-service-account-with-function-invoker-role>@<PROJECT_ID>.iam.gserviceaccount.com",
               "audience": url
           }
       }
   }
   if payload is not None:
       if isinstance(payload, dict):
           # Convert dict to JSON string
           payload = json.dumps(payload)
           # specify http content-type to application/json
           task["http_request"]["headers"] = {
               "Content-type": "application/json"}

       # The API expects a payload of type bytes.
       converted_payload = payload.encode()

       # Add the payload to the request.
       task["http_request"]["body"] = converted_payload

   if in_seconds is not None:
       # Convert "seconds from now" into an rfc3339 datetime string.
       d = datetime.datetime.utcnow() + datetime.timedelta(seconds=in_seconds)

       # Create Timestamp protobuf.
       timestamp = timestamp_pb2.Timestamp()
       timestamp.FromDatetime(d)

       # Add the timestamp to the tasks.
       task["schedule_time"] = timestamp

   # Use the client to build and send the task.
   response = client.create_task(request={"parent": parent, "task": task})

   print("Created task {}".format(response.name))
   return 'Task Created!'

Other helpful tips

  • Although this tutorial is written for Node.js, it walks you through creating a Cloud Task queue and setting up a service account that will invoke the Function from Cloud Task. By specifying a service account for the Task, you can use an authenticated Function.
  • If you’re using a different service account to invoke the Function (rather than your Function’s identity), you need to verify that the service account has the Cloud Functions Invoker role `roles/cloudfunctions.invoker`.
  • If you’re using a different service account for your “create_task” Function’s identity than the default, you need to verify that the service account has permissions to create Tasks. It will need the Cloud Tasks Enqueuer role `roles/cloudtasks.enqueuer`.
Read More  A New Chapter For VMware: Spin-Off From Dell Technologies Completed

You can also read more about Cloud Tasks in our third blog post in this series on making outbound connections.

 

By: Sara Ford (Cloud Developer Advocate) and Martin Skoviera (Technical Solutions Engineer)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Cloud Functions
  • Cloud Tasks
  • Google Cloud
  • Python
You May Also Like
View Post
  • Computing
  • Multi-Cloud
  • Technology

Reliance on US tech providers is making IT leaders skittish

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Examine the 4 types of edge computing, with examples

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

AI and private cloud: 2 lessons from Dell Tech World 2025

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

TD Synnex named as UK distributor for Cohesity

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Weigh these 6 enterprise advantages of storage as a service

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Pulsant targets partner diversity with new IaaS solution

  • May 23, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Growing AI workloads are causing hybrid cloud headaches

  • May 23, 2025

Stay Connected!
LATEST
  • 1
    Just make it scale: An Aurora DSQL story
    • May 29, 2025
  • 2
    Reliance on US tech providers is making IT leaders skittish
    • May 28, 2025
  • Examine the 4 types of edge computing, with examples
    • May 28, 2025
  • AI and private cloud: 2 lessons from Dell Tech World 2025
    • May 28, 2025
  • 5
    TD Synnex named as UK distributor for Cohesity
    • May 28, 2025
  • Weigh these 6 enterprise advantages of storage as a service
    • May 28, 2025
  • 7
    Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more
    • May 28, 2025
  • 8
    Pulsant targets partner diversity with new IaaS solution
    • May 23, 2025
  • 9
    Growing AI workloads are causing hybrid cloud headaches
    • May 23, 2025
  • Gemma 3n 10
    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI
    • May 22, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Understand how Windows Server 2025 PAYG licensing works
    • May 20, 2025
  • By the numbers: How upskilling fills the IT skills gap
    • May 21, 2025
  • 3
    Cloud adoption isn’t all it’s cut out to be as enterprises report growing dissatisfaction
    • May 15, 2025
  • 4
    Hybrid cloud is complicated – Red Hat’s new AI assistant wants to solve that
    • May 20, 2025
  • 5
    Google is getting serious on cloud sovereignty
    • May 22, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.