aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Data
  • Engineering

Announcing Apache Iceberg Support For BigLake

  • aster.cloud
  • November 17, 2022
  • 4 minute read
Apache Iceberg is a popular open source table format for customers looking to build data lakes. It provides many features found in enterprise data warehouses, such as transactional DML, time travel, schema evolution, and advanced metadata that unlocks performance optimization. Iceberg’s open specification allows customers to run multiple query engines on a single copy of data stored in an object store. Backed by a growing community of contributors, Apache Iceberg is becoming the de facto open standard for data lakes, bringing interoperability across clouds for hybrid analytical workloads and systems to exchange data.Earlier this year, we announced BigLake, a storage engine that enables customers to store data in open file formats (such as Parquet) on Google Cloud Storage and run GCP and open source query engines on it in a secure, governed, and performant manner. BigLake unifies data warehouses and lakes by enabling BigQuery and open source frameworks like Spark to access data with fine-grained access control. Today, we are excited to announce that this support now extends to the Apache Iceberg format, enabling customers to take advantage of Iceberg’s capabilities to build an open format data lake while benefiting from native GCP integration using BigLake.“Besides BigQuery, a large segment of our data is stored on GCS. Our Datalake leveraged Iceberg to tap into this data in an efficient and scalable way on top of incredibly large datasets. BigLake integration makes this even easier by making this data available to our large BigQuery user base and leverage its powerful UI. Our users now have the ability to realize most BigQuery benefits on GCS data as if this was stored natively.” — Bo Chen, Sr. Manager of Data and Insights at Snap Inc.

Read More  Updates Coming For Authorized Networks And Cloud Run/Functions On GKE

Build a secure and governed Iceberg data lake with BigLake’s fine-grained security model

BigLake enables multi-compute architecture: Iceberg tables created in supported open source analytics engines can be read using BigQuery.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

# Creation of table using Iceberg format with Dataproc Spark 

CREATE TABLE catalog.db.table (col1 type1, col2 type2) USING iceberg TBLPROPERTIES(bq_table='{bigquery_table}', bq_connection='{bigquery_connection}');

Once the table has been created in Spark, easily query using BigQuery:
# Query table using the BigQuery console 

SELECT COL1, COL2 FROM bigquery_table LIMIT 10;

Apache Spark already has rich support for Iceberg, allowing customers to use Iceberg’s core capabilities, such as DML, transactions, and schema evolution, to carry out large-scale transformation and data processing. Customers can run Spark using Dataproc (managed clusters or serverless), or use built-in support for Apache Spark in BigQuery (stored procedures) to process Iceberg tables hosted on Google Cloud Storage. Regardless of your choice of Spark, BigLake automatically makes those Iceberg tables available for end users to query.Administrators can now use Iceberg tables, similar to BigLake tables, and don’t need to provide end users access to the underlying GCS bucket. The end user access is delegated through BigLake, simplifying access management and governance. Administrators can further secure Iceberg tables using fine-grained access policies, such as row, column level access control, or data masking, extending the existing BigLake governance framework to Iceberg tables. BigQuery utilizes Iceberg’s metadata for query execution, providing a performant query experience to end users.
This set of capabilities enables customers to store a single copy of data on object stores using Iceberg and run BigQuery as well as Dataproc workloads on it in a secure, governed, and performant manner, eliminating the need to duplicate data or write custom infrastructure. For GCP customers who store their data on BigQuery Storage and Google Cloud Storage, BigLake now further unifies data lake and warehouse workloads. Customers can directly query, join, secure, and govern data across BigQuery storage and Iceberg tables on Google Cloud Storage. In the coming months, we will extend Apache Iceberg to Amazon S3 and Azure data lake Gen 2, enabling customers to build multi-cloud Iceberg data lakes.

Read More  Creating Custom Notifications With Cloud Monitoring And Cloud Run

Differentiate your Iceberg workloads with native BigQuery and GCP integration

The benefits of running Iceberg on Google Cloud extend beyond realizing Iceberg’s core capabilities and BigLake’s fine-grained security model. Customers can use native BigQuery and GCP integration to use BigQuery’s differentiated services on Iceberg tables created over Google Cloud Storage data. Some key integrations most relevant in the context of Iceberg are:

  • Securely exchange Iceberg data using Analytics Hub – Iceberg as an open standard provides interoperability between various storage systems and query engines to exchange data. On Google Cloud, customers use Analytics hub to share BigQuery & BigLake tables with their partners, customers, and suppliers without needing to copy data. Similar to BigQuery tables, data providers can now create shared datasets to share Iceberg tables on Google Cloud storage. Consumers of the shared data can use any Iceberg compatible supported query engine to consume the data, providing an open and governed model of sharing and consuming data.
  • Run data science workloads on Iceberg using BigQueryML – Customers can now use BigQueryML to extend their machine learning workloads to Iceberg tables stored on Google cloud storage, enabling customers to realize AI value on data stored outside of BigQuery.
  • Discover, detect and protect PII data on Iceberg using Cloud DLP – Customers can now use Cloud DLP to identify, discover and secure PII data elements contained in Iceberg tables, and secure sensitive data using BigLake’s fine-grained security model to meet workload compliance.

Get Started

Learn more about BigLake support for Apache Iceberg by watching this demo video, and a panel discussion of customers building using BigLake with Iceberg. Apache Iceberg support for BigLake is currently in preview, sign up to get started. Contact a Google sales representative to learn how Apache Iceberg can help evolve your data architecture.

Read More  COVID-19 Public Dataset Program: Making Data Freely Accessible For Better Public Outcomes

Special mention to the engineering leadership of Micah Kornfield, Anoop Johnson, Garrett Casto, Justin Levandoski and team to make this launch possible.

 

By: Gaurav Saxena (Sr. Product Manager, Google Cloud) and Yuri Volobuev (Principal Engineer, Google Cloud)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Apache Iceberg
  • BigLake
  • Data Analyst
  • Data Lake
  • Google Cloud
You May Also Like
View Post
  • Engineering

Just make it scale: An Aurora DSQL story

  • May 29, 2025
Getting things done makes her feel amazing
View Post
  • Computing
  • Data
  • Featured
  • Learning
  • Tech
  • Technology

Nurturing Minds in the Digital Revolution

  • April 25, 2025
View Post
  • Engineering
  • Technology

Guide: Our top four AI Hypercomputer use cases, reference architectures and tutorials

  • March 9, 2025
View Post
  • Computing
  • Engineering

Why a decades old architecture decision is impeding the power of AI computing

  • February 19, 2025
View Post
  • Engineering
  • Software Engineering

This Month in Julia World

  • January 17, 2025
View Post
  • Engineering
  • Software Engineering

Google Summer of Code 2025 is here!

  • January 17, 2025
View Post
  • Data
  • Engineering

Hiding in Plain Site: Attackers Sneaking Malware into Images on Websites

  • January 16, 2025
View Post
  • Computing
  • Design
  • Engineering
  • Technology

Here’s why it’s important to build long-term cryptographic resilience

  • December 24, 2024

Stay Connected!
LATEST
  • 1
    Just make it scale: An Aurora DSQL story
    • May 29, 2025
  • 2
    Reliance on US tech providers is making IT leaders skittish
    • May 28, 2025
  • Examine the 4 types of edge computing, with examples
    • May 28, 2025
  • AI and private cloud: 2 lessons from Dell Tech World 2025
    • May 28, 2025
  • 5
    TD Synnex named as UK distributor for Cohesity
    • May 28, 2025
  • Weigh these 6 enterprise advantages of storage as a service
    • May 28, 2025
  • 7
    Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more
    • May 28, 2025
  • 8
    Pulsant targets partner diversity with new IaaS solution
    • May 23, 2025
  • 9
    Growing AI workloads are causing hybrid cloud headaches
    • May 23, 2025
  • Gemma 3n 10
    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI
    • May 22, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Understand how Windows Server 2025 PAYG licensing works
    • May 20, 2025
  • By the numbers: How upskilling fills the IT skills gap
    • May 21, 2025
  • 3
    Cloud adoption isn’t all it’s cut out to be as enterprises report growing dissatisfaction
    • May 15, 2025
  • 4
    Hybrid cloud is complicated – Red Hat’s new AI assistant wants to solve that
    • May 20, 2025
  • 5
    Google is getting serious on cloud sovereignty
    • May 22, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.