aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
Automation
  • Automation
  • Platforms
  • Tools

Automate Your Data Warehouse Migration To BigQuery With New Data Migration Tool

  • aster_cloud
  • August 24, 2023
  • 3 minute read

As customers embark on their digital transformation journey, migrating on-premises data warehouses to the cloud accounts for a significant percentage of their overall transformation efforts.

Today, to accelerate data warehouse migrations to BigQuery, we are announcing additional services in the form of a data migration tool that automates the migration of on-premises data warehouses to BigQuery.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

Currently in preview, the data migration tool supports key phases of a data warehouse migration, from schema migration and historical data migration to SQL translation and validation and is built on Google Cloud. This tool works in close coordination with migration accelerators like Data Validation Tool and BigQuery migration services to automate the end-to-end data warehouse migration. Migrations from Teradata, Hive, Redshift, and Oracle are currently supported with support for more data warehouses in the future.

This is a fully configurable open-source solution that can be deployed via Terraform and triggered with a simple JSON configuration file. You can scale compute up or down based on data volume or complexity. The key benefits include:

  • A proven approach to enterprise data warehouse migration with reduced risk of errors, bugs, and data inconsistencies
  • Automated orchestration of data migration, SQL translation and validation
  • Reporting dashboard with results of data validation, job runs, and SQL translation in Looker
  • Reduced time to initial landing of data and SQL artifacts into Google Cloud
  • Lower total cost of migration
  • Limited need for an additional customer development team
  • A low learning curve

For example, here’s how the data migration tool facilitates a migration from Teradata to BigQuery:

  1. A user uploads the relevant input files (table list csv, SQL/DDL files) and a JSON configuration file to a Cloud Storage bucket, triggering the data migration tool.
  2. The data migration tool triggers Data Transfer Service to convert table schema, and transfer table data from the on-prem source data warehouse to BigQuery tables.
  3. After the data transfer is completed, the data migration tool triggers Data Transfer Service to compare source and target tables on aggregated column values, and outputs the validation result to BigQuery reporting tables.
  4. The data migration tool triggers the BigQuery SQL Translation Service to translate SQL queries (uploaded to a Cloud Storage bucket) from source syntax to BigQuery SQL, and output the translated SQL to a different bucket. The data migration tool writes translation statistics to BigQuery reporting tables.
  5. The tool then triggers a data validation tool to compare source and target SQL on their result set.
  6. The data migration tool writes validation statistics to BigQuery reporting tables.
  7. User reviews the Migration Dashboard in Looker Studio to inspect the results of the data migration, SQL translation, and data validation.
Read More  Koverse, A SAIC company, Introduces Industry’s Only Zero Trust Data Platform

And that’s it! In a few short steps, the data migration tool orchestrates what can otherwise be a time-consuming, error-prone process. This simple and proven migration approach can help you reduce the risk and accelerate the migration process, freeing up time and resources to focus on other aspects of your digital transformation journey. Better yet, there are no license fees to use the data migration tool on Google Cloud; the only costs are storage and compute for the services that you consume. For more on the data migration tool, check out the project on Github.

By: Abdullateef Abdulsalam (Data & Analytics Cloud Consultant) and Preethi Igoor (Data & Analytics Cloud Consultant)
Originally published at: Google Cloud Blog

Source: cyberpogo.com


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster_cloud

Related Topics
  • Automation
  • BigQuery
  • Data
  • Data Analytics
  • Data migration
  • Data Warehouse
  • Google Cloud
  • Tutorial
You May Also Like
View Post
  • Data
  • Engineering
  • Platforms
  • Solutions

How ‘Anything Is Possible’ Automated Data Pipelines With BigQuery And Windsor.ai

  • September 27, 2023
View Post
  • Computing
  • Platforms

Oracle CloudWorld 2023: 6 Key Takeaways From The Big Annual Event

  • September 25, 2023
View Post
  • Platforms

Combining AI With A Trusted Data Approach On IBM Power To Fuel Business Outcomes

  • September 21, 2023
View Post
  • Computing
  • Platforms

Start Your Ubuntu Confidential VM With Intel® TDX On Google Cloud

  • September 20, 2023
Microsoft and Adobe
View Post
  • Platforms
  • Solutions

Microsoft And Adobe Partner To Deliver Cost Savings And Business Benefits

  • September 20, 2023
View Post
  • Platforms
  • Technology

Huawei Releases Data Center 2030, Leading Innovation and Development of New Data Centers

  • September 20, 2023
View Post
  • Multi-Cloud
  • Platforms
  • Public Cloud

Oracle Expands Distributed Cloud Offerings to Help Organizations Innovate Anywhere

  • September 20, 2023
Volkswagen
View Post
  • Multi-Cloud
  • Platforms
  • Technology

Volkswagen Races Toward Next-Gen Automotive Manufacturing Leadership With Google Cloud And T-Systems

  • September 19, 2023

Stay Connected!
LATEST
  • 1
    How ‘Anything Is Possible’ Automated Data Pipelines With BigQuery And Windsor.ai
    • September 27, 2023
  • 2
    Oracle CloudWorld 2023: 6 Key Takeaways From The Big Annual Event
    • September 25, 2023
  • 3
    Nvidia H100 Tensor Core GPUs Come To Oracle Cloud
    • September 24, 2023
  • 4
    Combining AI With A Trusted Data Approach On IBM Power To Fuel Business Outcomes
    • September 21, 2023
  • 5
    Start Your Ubuntu Confidential VM With Intel® TDX On Google Cloud
    • September 20, 2023
  • Microsoft and Adobe 6
    Microsoft And Adobe Partner To Deliver Cost Savings And Business Benefits
    • September 20, 2023
  • Coffee | Laptop | Notebook | Work 7
    First HP Work Relationship Index Shows Majority of People Worldwide Have an Unhealthy Relationship with Work
    • September 20, 2023
  • 8
    Oracle Expands Distributed Cloud Offerings to Help Organizations Innovate Anywhere
    • September 20, 2023
  • 9
    Huawei Connect 2023: Accelerating Intelligence For Shared Success
    • September 20, 2023
  • 10
    Huawei Releases Data Center 2030, Leading Innovation and Development of New Data Centers
    • September 20, 2023
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Penguin 1
    How To Find And Fix Broken Packages On Linux
    • September 19, 2023
  • Volkswagen 2
    Volkswagen Races Toward Next-Gen Automotive Manufacturing Leadership With Google Cloud And T-Systems
    • September 19, 2023
  • 3
    VMware Scales Multi-Cloud Security With Workforce Identity Federation
    • September 18, 2023
  • Intel Innovation 4
    Intel Innovation 2023
    • September 15, 2023
  • 5
    Microsoft And Oracle Expand Partnership To Deliver Oracle Database Services On Oracle Cloud Infrastructure In Microsoft Azure
    • September 14, 2023
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.