aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
  • Tools
  • About
  • Data
  • Engineering

Break Down Data Silos With The New Cross-Cloud Transfer Feature Of BigQuery Omni

  • aster_cloud
  • December 30, 2022
  • 4 minute read

To help customers break down data silos, we launched BigQuery Omni in 2021. Organizations globally are using BigQuery Omni to analyze data across cloud environments. Now, we are excited to launch the next big evolution for multi cloud analytics: cross-cloud analytics. Cross-cloud analytics tools help analysts and data scientists easily, securely, and cost effectively distribute data between clouds to leverage the analytics tools they need. In April 2022, we previewed a SQL supported LOAD statement that allowed AWS/Azure blob data to be brought into BigQuery as a managed table for advanced analysis. We’ve learned a lot in this preview period. A few learnings stand out:

  1. Cross-cloud operations need to meet analysts where they are. In order for analysts to work with distributed data, workspaces should not be siloed. As soon as analysts are asked to leave their SQL workspaces to copy data, set up permissions, or grant permission, workflows break down and insights are lost. Same SQL can be used to periodically copy data using BigQuery scheduled queries. The more of the workflow that can be managed by SQL, the better.
  2. Networking is an implementation detail, latency should be too. The longer an analyst needs to wait for an operation to complete, the less likely a complete workflow is to be completed end-to-end. BigQuery users expect high performance for a single operation, even if those operations are managed across multiple data centers.
  3. Democratizing data shouldn’t come at the cost of security. In order for data admins to empower data analysts and engineers, they need to be assured there isn’t additional risk in doing so. By default, data admins and security teams are increasingly looking for solutions that don’t persist user credentials between cloud boundaries.
  4. Cost control comes with cost transparency. Data transfer costs can get costly, and we hear frequently this is the number 1 concern for multi-cloud data organizations. Providing transparency into single operations and invoices in a consolidated way is critical to driving success for cross-cloud operations. Allowing administrators to cap costs for budgeting is a must.
Read More  CISO Perspectives: June 2022

This feedback is why we’ve spent much of this year improving our cross-cloud transfer product to optimize releases around these core tenants:


Partner with aster.cloud
for your next big idea.
Let us know here.


cyberpogo
  • Usability: The LOAD SQL experience allows for data filtering and loading within the same editor across clouds. LOAD SQL supports data formats like JSON, CSV, AVRO, ORC and PARQUET. With semantics for both appending and truncating tables, LOAD supports both periodic syncs and refreshing the complete table semantics. We’ve also added SQL support for data lake standards like Hive partitioning and JSON data type.
  • Security: With a federated identity model, users don’t have to share or store credentials between cloud providers to access and copy their data. We also now support CMEK support for the destination table to help secure data as it’s written in BigQuery and VPC-SC boundaries to mitigate data exfiltration risks.
  • Latency: With data movement managed by BigQuery Write API, users can effortlessly move just the relevant data without having to wait for complex pipes. We’ve improved job latency significantly for the most common load jobs and are seeing performance improvements with each passing day.
  • Cost auditability: From one invoice, you can see all your compute and transfer costs for LOADs across clouds. Each job comes with statistics to help admins manage budgets.

During our preview period, we saw good proof points on how cross-cloud transfer can be used to accelerate time to insight and deliver value to data teams.

Getting started with a cross-cloud architecture can be daunting, but cross-cloud transfer has been used to help customers jumpstart proof of concepts because it enables the migration of subsets of data without committing to a full migration. Kargo used cross-cloud transfer to accelerate a performance test of BigQuery. “We tested Cross-Cloud Transfer to assist with a proof of concept on BigQuery earlier this year. We found the usability and performance useful during the POC,” saidDinesh Anchan, Manager of Engineering at Kargo.

Read More  Pro Tools For Pros: Industry Leading Observability Capabilities For Dataflow

Other customers, like ActionIQ, see Cross-Cloud Transfer potential to complement differentiated offerings in their customer-centric platforms. “ActionIQ is a composable CDP technology that helps enterprises tap into customer data to deliver personalized customer experiences. We’re investing heavily in allowing customers to query data wherever it is, and we’re excited Cross Cloud Transfer is embracing that philosophy,” said Justin DeBrabant, Senior Vice President of Product at ActionIQ. “Democratizing data should not come at the cost of security, cost, or latency of data, and we’re excited to provide joint solutions with BigQuery that extend that philosophy.”

We also saw this product being used to combine key datasets across clouds. A common challenge for customers is to manage cross-cloud billing data. CCT is being used to tie files together which have evolving schema on delivery for blob storage. “We liked the experience of using Cross-Cloud transfer to help consolidate our billing files across GCP, AWS, and Azure. CCT was a nice solution because we could use SQL statements to load our billing files into BigQuery,” said the engineering lead of a large research institution.

We’re excited to release the first of many cross-cloud features through BigQuery Omni. Check out the Google Cloud Next session to learn about more upcoming launches in the multicloud analytics space including support for Omni tables and local transformations to help supercharge these experiences for analysts and data scientists. We’re investing in cross-cloud because cloud boundaries shouldn’t slow innovation. Watch this space.

Availability and pricing

Cross-Cloud Transfer is now available in all BigQuery Omni regions. Check the BigQuery Omni pricing page for data transfer costs.

Read More  Google Classroom Is Streamlining Virtual Training For Iowa Employees

Getting Started

It has never been easier for analysts to move data between clouds. Check out our getting started (AWS/Azure) page to try out this SQL experience. For a limited trial, BigQuery customers can explore BigQuery Omni at no charge using on-demand byte scans from September 15, 2022 to March 31, 2023 (the “trial period”) for data scans on AWS/Azure. Note: data transfer fees for Cross-Cloud Transfer will still apply.

 

By: Joe Malone (Product Manager, Data Analytics) and Rushabh Desai (Technical Lead)
Source: Google Cloud Blog


Our humans need coffee too! Your support is highly appreciated, thank you!

aster_cloud

Related Topics
  • AWS
  • Azure
  • BigQuery Omni
  • BigQuery;
  • Cross-Cloud Transfer
  • Data Analytics
  • Google Cloud
  • SQL
You May Also Like
View Post
  • Computing
  • Design
  • Engineering
  • Multi-Cloud

Amazing Federated Multicloud Apps

  • June 2, 2023
View Post
  • Engineering

Resolving Deployment Issues With Ts-node And Azure Development Pipelines

  • June 1, 2023
View Post
  • Design
  • Engineering

Four Steps To Managing Your Cloud Logging Costs On A Budget

  • May 31, 2023
View Post
  • Engineering
  • Tools

Red Hat Puts Podman Container Management On The Desktop

  • May 30, 2023
View Post
  • Automation
  • Computing
  • Data

IBM To Help Automate Cloud Data Protection With Acquisition Of Polar Security

  • May 25, 2023
View Post
  • Engineering
  • Practices
  • Tools

Tricentis Launches Quality Engineering Community ShiftSync

  • May 23, 2023
View Post
  • Data
  • Public Cloud

Cloud Data Loss Prevention’s Sensitive Data Intelligence Service Is Now Available In Security Command Center

  • May 18, 2023
View Post
  • Design
  • Engineering
  • Public Cloud

Kubernetes Costs Less, But Less Than What?

  • May 18, 2023

Stay Connected!
LATEST
  • 1
    Amazing Federated Multicloud Apps
    • June 2, 2023
  • 2
    What’s The Future Of DevOps? You Tell Us. Take The 2023 Accelerate State Of DevOps Survey
    • June 2, 2023
  • 3
    Resolving Deployment Issues With Ts-node And Azure Development Pipelines
    • June 1, 2023
  • 4
    What To Expect From Apple’s WWDC 2023
    • June 1, 2023
  • 5
    What Is Platform Engineering And Why Adopt It In Your Company?
    • June 1, 2023
  • 6
    Four Steps To Managing Your Cloud Logging Costs On A Budget
    • May 31, 2023
  • 7
    Red Hat Puts Podman Container Management On The Desktop
    • May 30, 2023
  • 8
    The Agile Mindset: A Path to Personal Fulfillment and Growth
    • May 30, 2023
  • 9
    G7 2023: The Real Threat To The World Order Is Hypocrisy.
    • May 27, 2023
  • 10
    Build Next-Generation, AI-Powered Applications On Microsoft Azure
    • May 26, 2023
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • 1
    Huawei OceanStor Pacific Scale-Out Storage Tops IO500 Rankings
    • May 26, 2023
  • 2
    MongoDB And Alibaba Cloud Extend Global Partnership
    • May 25, 2023
  • 3
    Tricentis Launches Quality Engineering Community ShiftSync
    • May 23, 2023
  • 4
    Oracle Cloud Infrastructure Adds To Growing List Of Government Approved Cloud Services
    • May 22, 2023
  • 5
    Huawei And Partners Announce Yucatan Wildlife Conservation Findings
    • May 18, 2023
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.