aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
Automation
  • Automation
  • Platforms
  • Tools

Automate Your Data Warehouse Migration To BigQuery With New Data Migration Tool

  • aster.cloud
  • August 24, 2023
  • 3 minute read

As customers embark on their digital transformation journey, migrating on-premises data warehouses to the cloud accounts for a significant percentage of their overall transformation efforts.

Today, to accelerate data warehouse migrations to BigQuery, we are announcing additional services in the form of a data migration tool that automates the migration of on-premises data warehouses to BigQuery.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

Currently in preview, the data migration tool supports key phases of a data warehouse migration, from schema migration and historical data migration to SQL translation and validation and is built on Google Cloud. This tool works in close coordination with migration accelerators like Data Validation Tool and BigQuery migration services to automate the end-to-end data warehouse migration. Migrations from Teradata, Hive, Redshift, and Oracle are currently supported with support for more data warehouses in the future.

This is a fully configurable open-source solution that can be deployed via Terraform and triggered with a simple JSON configuration file. You can scale compute up or down based on data volume or complexity. The key benefits include:

  • A proven approach to enterprise data warehouse migration with reduced risk of errors, bugs, and data inconsistencies
  • Automated orchestration of data migration, SQL translation and validation
  • Reporting dashboard with results of data validation, job runs, and SQL translation in Looker
  • Reduced time to initial landing of data and SQL artifacts into Google Cloud
  • Lower total cost of migration
  • Limited need for an additional customer development team
  • A low learning curve

For example, here’s how the data migration tool facilitates a migration from Teradata to BigQuery:

  1. A user uploads the relevant input files (table list csv, SQL/DDL files) and a JSON configuration file to a Cloud Storage bucket, triggering the data migration tool.
  2. The data migration tool triggers Data Transfer Service to convert table schema, and transfer table data from the on-prem source data warehouse to BigQuery tables.
  3. After the data transfer is completed, the data migration tool triggers Data Transfer Service to compare source and target tables on aggregated column values, and outputs the validation result to BigQuery reporting tables.
  4. The data migration tool triggers the BigQuery SQL Translation Service to translate SQL queries (uploaded to a Cloud Storage bucket) from source syntax to BigQuery SQL, and output the translated SQL to a different bucket. The data migration tool writes translation statistics to BigQuery reporting tables.
  5. The tool then triggers a data validation tool to compare source and target SQL on their result set.
  6. The data migration tool writes validation statistics to BigQuery reporting tables.
  7. User reviews the Migration Dashboard in Looker Studio to inspect the results of the data migration, SQL translation, and data validation.
Read More  InfluxData Launches InfluxDB University

And that’s it! In a few short steps, the data migration tool orchestrates what can otherwise be a time-consuming, error-prone process. This simple and proven migration approach can help you reduce the risk and accelerate the migration process, freeing up time and resources to focus on other aspects of your digital transformation journey. Better yet, there are no license fees to use the data migration tool on Google Cloud; the only costs are storage and compute for the services that you consume. For more on the data migration tool, check out the project on Github.

By: Abdullateef Abdulsalam (Data & Analytics Cloud Consultant) and Preethi Igoor (Data & Analytics Cloud Consultant)
Originally published at: Google Cloud Blog

Source: cyberpogo.com


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Automation
  • BigQuery
  • Data
  • Data Analytics
  • Data migration
  • Data Warehouse
  • Google Cloud
  • Tutorial
You May Also Like
notta-ai-header
View Post
  • Featured
  • Tools

Notta vs Fireflies: Which AI Transcription Tool Deserves Your Attention in 2025?

  • May 16, 2025
Google Cloud and Smart Communications
View Post
  • Platforms
  • Technology

Smart Communications, Inc. Dials into Google Cloud AI to Help Personalize Digital Services for Filipinos

  • October 25, 2024
zedreviews-Apple-iPhone-16-Pro-finish-lineup-240909
View Post
  • Featured
  • Gears
  • Tech
  • Technology
  • Tools

Apple debuts iPhone 16 Pro and iPhone 16 Pro Max

  • September 10, 2024
zedreviews-Apple-AirPods-Active-Noise-Cancellation-240909
View Post
  • Featured
  • Gears
  • Tech
  • Technology
  • Tools

Apple introduces AirPods 4 and the world’s first all-in-one hearing health experience with AirPods Pro 2

  • September 10, 2024
View Post
  • Platforms
  • Public Cloud

Empowering builders with the new AWS Asia Pacific (Malaysia) Region

  • August 30, 2024
Red Hat and Globe Telecoms
View Post
  • Platforms
  • Technology

Globe Collaborates with Red Hat Open Innovation Labs to Modernize IT Infrastructure for Greater Agility and Scalability

  • August 19, 2024
Huawei Cloud Cairo Region Goes Live
View Post
  • Cloud-Native
  • Computing
  • Platforms

Huawei Cloud Goes Live in Egypt

  • May 24, 2024
Asteroid
View Post
  • Computing
  • Platforms
  • Technology

Asteroid Institute And Google Cloud Identify 27,500 New Asteroids, Revolutionizing Minor Planet Discovery With Cloud Technology

  • April 30, 2024

Stay Connected!
LATEST
  • 1
    Building secure, scalable AI in the cloud with Microsoft Azure
    • July 5, 2025
  • 2
    Turns out OpenAI is the customer behind Oracle’s mysterious $30 billion cloud deal
    • July 3, 2025
  • aster-cloud-erp-bill_of_materials_2 3
    What is an SBOM (software bill of materials)?
    • July 2, 2025
  • aster-cloud-sms-pexels-tim-samuel-6697306 4
    Send SMS texts with Amazon’s SNS simple notification service
    • July 1, 2025
  • Camping 5
    The Summer Adventures : Camping Essentials
    • June 27, 2025
  • aster-cloud-website-pexels-goumbik-574069 6
    Host a static website on AWS with Amazon S3 and Route 53
    • June 27, 2025
  • 7
    A looming hyperscaler exodus? UK IT leaders are thinking of ditching US cloud providers – here’s why
    • June 26, 2025
  • Prioritize security from the edge to the cloud
    • June 25, 2025
  • 6 edge monitoring best practices in the cloud
    • June 25, 2025
  • Genome 10
    AlphaGenome: AI for better understanding the genome
    • June 25, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • 1
    There’s a ‘cloud reset’ underway, and VMware Cloud Foundation 9.0 is a chance for Broadcom to pounce on it
    • June 17, 2025
  • 2
    ‘This was a very purposeful strategy’: Pure Storage unveils Enterprise Data Cloud in bid to unify data storage, management
    • June 18, 2025
  • 3
    Pure Accelerate 2025: All the news and updates live from Las Vegas
    • June 18, 2025
  • Oracle adds xAI Grok models to OCI
    • June 17, 2025
  • What is cloud bursting?
    • June 18, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.