aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Data
  • Engineering
  • Tech

Orchestrate Looker Data Transformations With Cloud Composer

  • aster.cloud
  • May 13, 2022
  • 3 minute read

We are announcing that Looker’s new Google Cloud operators for Apache Airflow are available in Cloud Composer, Google Cloud’s fully managed service for orchestrating workflows across cloud, hybrid, and multi-cloud environments. This integration gives users the ability to orchestrate Looker persistent derived tables (PDTs) alongside the rest of their data pipeline.

Looker PDTs are the materialized results of a query, written to a Looker scratch schema in the connected database and rebuilt on a defined schedule. Because they are defined within LookML, PDTs reduce friction and speed up time to value by putting the power to create robust data transformations in the hands of data modelers. But administration of these transformations can be difficult to scale. By leveraging this new integration, customers can now get greater visibility into and exercise more granular control over their data transformations.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

Using Looker with Cloud Composer enables customers to:

  • Know exactly when PDTs are going to rebuild by directly linking PDT regeneration jobs to the completion of other data transformation jobs. This insight ensures that PDTs are always up to date without using Looker datagroups to repeatedly query for changes in the underlying data and enables admins to closely control job timing and resource consumption.
  • Automatically kick off other tasks that leverage data from PDTs, like piping transformed data into a machine learning model or delivering transformed data to another tool or file store.
  • Quickly get alerted of errors that occur for more proactive troubleshooting and issue resolution.
  • Save time and resources by quickly identifying any points of failure within a chain of cascading PDTs and restarting the build process from there rather than from the beginning. Within Looker, there are only options to rebuild a specific PDT or to rebuild the entire chain.
  • Easily pick up any changes in your underlying database by forcing incremental PDTs to reload in full on a schedule or on an ad-hoc basis with the click of a button.
Read More  Building Cloud Into Your Data Strategy Delivers Higher Efficiency

Pairing Looker with Cloud Composer provides customers with a pathway for accomplishing key tasks like these, making it easier to manage and scale PDT usage.

What’s New

There are two new Looker operators available that can be used to manage PDT builds using Cloud Composer:

  • LookerStartPdtBuildOperator: initiates materialization for a PDT based on a specified model name and view name and returns the materialization ID.
  • LookerCheckPdtBuildSensor: checks the status of a PDT build based on a provided materialization ID for the PDT build job.

These operators can be used in Cloud Composer to create tasks inside of a Directed Acyclic Graph, or DAG, with each task representing a specific PDT build. These tasks can be organized based on relationships and dependencies across different PDTs and other data transformation jobs.

Getting Started

You can start using Looker and Cloud Composer together in a few steps:

  1. Within your connection settings in your Looker instance, turn on the Enable PDT API Control toggle. Make sure that this setting is enabled for any connection with PDTs that you’d like to manage using Cloud Composer.
  2. Set up a Looker connection in Cloud Composer. This connection can be done through Airflow directly, but for production use, we’d recommend that you use Cloud Composer’s Secret Manager.
  3. Create a DAG using Cloud Composer.
  4. Add tasks into your DAG for PDT builds.
  5. Define dependencies between tasks within your DAG.

To learn more about how to externally orchestrate your Looker data transformations, see this tutorial in the Looker Community.

Data Transformations at Scale

This integration between Looker and Cloud Composer pairs the speed and agility of PDTs with the added scalability and governance of Cloud Composer. By managing these Looker data transformations using Cloud Composer, customers can:

  • Define and manage build schedules to help ensure that resourcing is allocated efficiently across all ongoing processes
  • See the jobs that are running, have errored, or have completed, including Looker data transformations, in one place
  • Leverage the output of a PDT within other automated data transformations taking place outside of Looker
Read More  Meet AI’s Multitool: Vector Embeddings

Thanks to this integration with Cloud Composer, Looker is giving customers the ability to empower modelers and analysts to transform data at speed, while also tapping into a scalable governance model for transformation management and maintenance.  Looker operators for Cloud Composer are generally available to customers using an Airflow 2 environment. For more information, check out the Cloud Composer documentation or read this tutorial on setting up Looker with Apache Airflow.


Acknowledgements: Aleks Flexo, Product Manager

By: Aleksei Loginov (Software Engineer) and Maire Newton (Outbound Product Manager, Google Cloud)

Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Apache Airflow
  • Cloud Composer
  • Data Analytics
  • Google Cloud
  • Looker
You May Also Like
View Post
  • Engineering
  • Technology

Apple supercharges its tools and technologies for developers to foster creativity, innovation, and design

  • June 9, 2025
View Post
  • Engineering

Just make it scale: An Aurora DSQL story

  • May 29, 2025
Getting things done makes her feel amazing
View Post
  • Computing
  • Data
  • Featured
  • Learning
  • Tech
  • Technology

Nurturing Minds in the Digital Revolution

  • April 25, 2025
View Post
  • Engineering
  • Technology

Guide: Our top four AI Hypercomputer use cases, reference architectures and tutorials

  • March 9, 2025
View Post
  • Computing
  • Engineering

Why a decades old architecture decision is impeding the power of AI computing

  • February 19, 2025
View Post
  • Tech

Deep dive into AI with Google Cloud’s global generative AI roadshow

  • February 18, 2025
View Post
  • Engineering
  • Software Engineering

This Month in Julia World

  • January 17, 2025
View Post
  • Engineering
  • Software Engineering

Google Summer of Code 2025 is here!

  • January 17, 2025

Stay Connected!
LATEST
  • 1
    Pure Accelerate 2025: All the news and updates live from Las Vegas
    • June 18, 2025
  • 2
    ‘This was a very purposeful strategy’: Pure Storage unveils Enterprise Data Cloud in bid to unify data storage, management
    • June 18, 2025
  • What is cloud bursting?
    • June 18, 2025
  • 4
    There’s a ‘cloud reset’ underway, and VMware Cloud Foundation 9.0 is a chance for Broadcom to pounce on it
    • June 17, 2025
  • What is confidential computing?
    • June 17, 2025
  • Oracle adds xAI Grok models to OCI
    • June 17, 2025
  • Fine-tune your storage-as-a-service approach
    • June 16, 2025
  • 8
    Advanced audio dialog and generation with Gemini 2.5
    • June 15, 2025
  • 9
    A Father’s Day Gift for Every Pop and Papa
    • June 13, 2025
  • 10
    Global cloud spending might be booming, but AWS is trailing Microsoft and Google
    • June 13, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Google Cloud, Cloudflare struck by widespread outages
    • June 12, 2025
  • What is PC as a service (PCaaS)?
    • June 12, 2025
  • 3
    Crayon targets mid-market gains with expanded Google Cloud partnership
    • June 10, 2025
  • By the numbers: Use AI to fill the IT skills gap
    • June 11, 2025
  • 5
    Apple services deliver powerful features and intelligent updates to users this autumn
    • June 11, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.