aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Data
  • Technology

Introducing Serverless Spark For Interactive Development

  • aster.cloud
  • April 22, 2022
  • 3 minute read

At Google Cloud, we’re committed to helping you build an open and integrated data platform that meets your specific business needs. We believe in order to build the world you want, you need to be able to use the tools you want, on a powerful and unified platform.

To that end, we’re making Apache Spark even more powerful on Google Cloud, and creating a simplified experience across the platform. Data engineers, data analysts, and data scientists like Spark for its high speed data querying, analysis, and transformation with large data sets; but they want to use it from their interface of choice, without the need for custom integrations. And if you’ve used Spark, you know someone has to manage tuning, provisioning, and security, and make sure you’re using what you pay for. Our customers have made it clear- they want that someone to be us. They want auto-tuned capabilities, easy provisioning, and an efficient cost basis without having to manually reconfigure every time load fluctuates.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

Earlier this year, we made Serverless Spark generally available, creating the industry’s first autoscaling serverless Spark. Today, we announced at Data Cloud Summit 2022 we’re taking it a step further and making Serverless Spark even more powerful, by enabling serverless interactive development through Jupyter notebooks, natively integrated with Vertex AI Workbench. Now, a data scientist will not need to provision any infrastructure in advance; they can start developing on-demand.

Now you can use Spark to conduct data processing and machine learning without any custom integrations: it is available natively in BigQuery, Vertex AI, Dataplex, and Dataproc. We have integrated a PySpark editor into the BigQuery Console, powered by the Serverless Spark backend. Users can write and submit PySpark code from BigQuery, see the results in the BigQuery console, and get the same serverless, auto-scaling experience they are used to with SQL.

Read More  Nordic Choice Hotels Bounces Back From Ransomware Attack With ChromeOS Flex

From the Vertex AI notebook, you can choose the Kernel that represents serverless Spark. Without any resources provisioned beforehand, clicking the Kernel will start a serverless Spark session, and you can start development right away. Once finished, submit the notebook as a Dataproc job for production or publish it for live inference in Vertex AI.

You can also use the Jupyter notebook inside the Serverless Spark session, if you choose to use it through gCloud CLI or the Dataproc API.

Once you have explored your data and developed your code through a notebook, you’ll want to take your work to production. To do so, you typically need to define ML pipelines that can be scheduled or triggered when specific events happen. We’re making this easier,with Dataproc components for Vertex AI Pipelines. This will simplify MLOps for Spark, Spark SQL, PySpark and Spark jobs.

Just as in Serverless Spark for ETL pipelines, you won’t be charged for starting up or shutting down the underlying infrastructure. You only pay for the time the Spark application is live.

We also know that some people want a bit more control. We’ve got good news for them too, with the general availability of Dataproc on GKE for Spark. Now you can deploy a GKE cluster, leverage the scaling capabilities of GKE, and use Spark with all security and compliance features of Dataproc. This adds the power of Spark to the advanced compute management and resource sharing capabilities you already rely on from your Kubernetes environment.

Now you can effortlessly power ETL, data science, data analytics and AI pipelines with Spark with exactly the amount of control you want; use serverless for no-ops automation, control where you need to with compute clusters, and standardize Kubernetes workloads when you run workloads across multiple environments using GKE.

Read More  Cloud Logging Pricing For Cloud Admins: How To Approach It & Save Cost

We all know how much of a difference it makes when you can use the exact right tool for a job. But when all those tools play nice together too? That’s the power you need to tackle your biggest challenges.

Get started with Serverless Spark today by connecting to it with your tool of choice, and request allowlist access to the private preview of BigQuery and Vertex AI integrations here.

 

 

By: Abhishek Kashyap (Sr. Product Manager, Google Cloud)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Apache Spark
  • Data Analytics
  • Google Cloud
  • Severless
You May Also Like
View Post
  • Computing
  • Multi-Cloud
  • Technology

Reliance on US tech providers is making IT leaders skittish

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Examine the 4 types of edge computing, with examples

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

AI and private cloud: 2 lessons from Dell Tech World 2025

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

TD Synnex named as UK distributor for Cohesity

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Weigh these 6 enterprise advantages of storage as a service

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Pulsant targets partner diversity with new IaaS solution

  • May 23, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Growing AI workloads are causing hybrid cloud headaches

  • May 23, 2025

Stay Connected!
LATEST
  • 1
    Just make it scale: An Aurora DSQL story
    • May 29, 2025
  • 2
    Reliance on US tech providers is making IT leaders skittish
    • May 28, 2025
  • Examine the 4 types of edge computing, with examples
    • May 28, 2025
  • AI and private cloud: 2 lessons from Dell Tech World 2025
    • May 28, 2025
  • 5
    TD Synnex named as UK distributor for Cohesity
    • May 28, 2025
  • Weigh these 6 enterprise advantages of storage as a service
    • May 28, 2025
  • 7
    Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more
    • May 28, 2025
  • 8
    Pulsant targets partner diversity with new IaaS solution
    • May 23, 2025
  • 9
    Growing AI workloads are causing hybrid cloud headaches
    • May 23, 2025
  • Gemma 3n 10
    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI
    • May 22, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Understand how Windows Server 2025 PAYG licensing works
    • May 20, 2025
  • By the numbers: How upskilling fills the IT skills gap
    • May 21, 2025
  • 3
    Cloud adoption isn’t all it’s cut out to be as enterprises report growing dissatisfaction
    • May 15, 2025
  • 4
    Hybrid cloud is complicated – Red Hat’s new AI assistant wants to solve that
    • May 20, 2025
  • 5
    Google is getting serious on cloud sovereignty
    • May 22, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.