aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Technology
  • Tools

Extending Network Reachability Of Vertex AI Pipelines

  • aster.cloud
  • March 17, 2022
  • 5 minute read

Vertex AI Pipelines and jobs use the Service Networking API to configure their networking. As such, they will run in a Google-owned VPC network in tenant project that is peered to the network in your project. This peering is subject to the following standard VPC Peering networking constraints;

 


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

  • The Google-owned network has its own default route and will not import your network’s default route to the Internet
  • Peered networks can be configured to export static and dynamic routes to each other. However, transitive peering is not allowed. Thus, Vertex AI jobs in the tenant project will not, by default, be able to reach other networks peered to your network.

 

Given these, it initially appears that you are unable to, for instance, apply firewall rules to your public endpoints that allow access to just your Vertex AI jobs. Additionally, these jobs are not immediately able to route through your network to reach other services that are peered with your network.

This article suggests configurations that will allow the Vertex AI Pipeline job, that is in a Google-managed VPC, to complete the following connections;

 

  • Access endpoints on the Internet using a public source IPs that you control
  • Access other peered endpoints by forwarding connections through your network

 

As a data scientist, this may be used as a reference to drive conversations about your networking requirements with your organization’s network administrator

Controlling the public outbound source IP

Some use cases call for allowing the pipeline job to connect to endpoints over the Internet. In order to configure firewall rules that allow these connections, an administrator needs to know the public source IP used by the pipeline.

Since the pipeline job runs in a Google-owned project that you have no access to, any outbound connections from there directly to the Internet are beyond your control.

While we recommend using IAM authentication to protect these endpoints, some users also need to deploy firewall rules that protect against Denial-of-Service attacks and comply with their organization’s security requirements.

Read More  Introducing GPT-4 In Azure OpenAI Service

The following workflows offer a way to control the outbound source IP for these connections.

Access Through NAT Instance

This configuration allows the pipeline to access endpoints on the Internet using the source IP of a NAT instance that you control.

 

Workflow;

 

  • Create a NAT instance with one interface in your peered network and another in an outbound network. You may select any image that meets your requirements; however we will use a standard Debian instance to drive this discussion. NOTE: Since User Network1 refers to the NAT instance as the next hop to 8.8.8.8, we should have the NAT instance forward traffic to 8.8.8.8  to it’s other interface in User Network2 in order to prevent a routing loop in User Network1
  • Ensure you enable IP forwarding on this instance and it has a public IP
  • Visit the Go to the VPC Routes page for your project and add a static route to the target with the instance as the next hop. The example below uses 8.8.8.8 as the target
  • Also visit the peering configuration and enable`Export custom routes` to ensure that this static route is exported to the tenant project. Here we see that the static route to 8.8.8.8 has been exported to service networking.

 

  • Visit the UI for the instance and configure the instance to translate the source IPs on forwarded traffic to its own outbound interface. In the example below, interface en4 on the Debian instance is in the peered network and en5 is the outbound network.

 

sudo sysctl net.ipv4.conf.all.forwarding=1
sudo iptables --table nat --append POSTROUTING --out-interface ens5 -j MASQUERADE

 

  • Also configure the instance with a static route to the endpoint with the outbound interface and gateway in User Network2 as the next hop

 

sudo ip route add 8.8.8.8/32 via 10.1.1.1 dev ens5

 

The following packet capture from the instance confirms that outbound connections from the pipeline’s 10.100.1.2 address are translated to the instance’s 10.1.1.2 address

Read More  Accelerate Google Cloud Vendor Due Diligence By Leveraging Third Party Risk Management Providers

 

 

 

Access Through Proxy

If you need to control the source IP the Vertex uses when connecting to a particular service you may also configure a service proxy in your network.

 

The Vertex AI job can reach the proxy in the directly peered network. Requests from the pipeline to external endpoints are relayed through the proxy and appear to originate from the Proxy’s public IP address

Similarly, connections from the Vertex AI job will appear to the public endpoint to originate from the proxy’s public IP

Access to other Peered Services

On occasion, the pipeline needs to access another service that consumes service networking.

We will use Memorystore for Redis to drive this discussion. Recalling that transitive peering is note supported, in the example below, connections from Vertex network cannot, by default, be forwarded through your network to the Redis network.

 

Collocation of Peered Services

The preferred option is to deploy all services that need to communicate with each other  in the same Service Networking reservation. In the example below the Vertex AI job can reach Redis directly.

 

Please refer to the Reserving IP Ranges for Vertex AI discussion for guidance on extending this range to accommodate other services. Many services offer the option to either reserve a new range or consume an existing range. Please refer to the respective service documentation for guidance on how to use existing ranges

NAT Instance

The workflows discussed above for assigning Vertex AI Pipelines public IPs by going through either a NAT instance or a proxy are easily adapted for access to this Redis example.

In the diagram below, User Network1 has a static route to Redis with the NAT instance as a next hop and exports this to the Vertex network. Connections from Vertex AI appear to Redis as if they originate from the NAT instance’s interface in User Network2.

Read More  Gordon Moore, Intel Co-Founder, Dies At 94

This configuration might reduce the number of routes to manage in case you  add new Service Networking IP ranges.

 

Routing Instance

You can also replace the NAT instance with a routing instance between User Network1 and User Network2.

With this configuration, you need a static route to the Redis IP range in User Network1 as well as one to the Vertex AI IP range in User Network2. Both the static routes point to the routing instance as the next hop.

This configuration might be easier to troubleshoot than using a NAT instance. How you will have to work with your network administrator to update the routing tables in both networks if you bring up new services to maintain bidirectional connectivity

Cloud VPN

Rather than a routing instance, you can also configure Cloud VPN between User Network1 and User Network2. Here again you will need to export static routes both ways so that the respective service networks know to route through your network to reach each other. While you will still have to maintain the static route, this option relieves you of having to manage a routing instance yourself.

 

 

Proxy Instance

Similarly, if Vertex goes through a proxy to access Redis, those connections appear to originate from the proxy’s IP. The Redis instance can respond without having to know how to reach the Vertex AI network.

 

Summary

Vertex AI is one of a number of services that use the Service Networking API to connect the Google owned tenant project to your consumer project. Deployment guides for these services discuss how to connect these services to endpoints in your project.

This document extends the networking configuration discussion and presents options for connecting Vertex AI to other Service Networking consumers or endpoints in the Internet.

 

 

By: Bernie Ongewe (Technical Solutions Engineer)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Google Cloud
  • Tutorial
  • Vertex AI
  • Vertex AI Pipelines
You May Also Like
View Post
  • Computing
  • Multi-Cloud
  • Technology

Reliance on US tech providers is making IT leaders skittish

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Examine the 4 types of edge computing, with examples

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

AI and private cloud: 2 lessons from Dell Tech World 2025

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

TD Synnex named as UK distributor for Cohesity

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Weigh these 6 enterprise advantages of storage as a service

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Pulsant targets partner diversity with new IaaS solution

  • May 23, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Growing AI workloads are causing hybrid cloud headaches

  • May 23, 2025

Stay Connected!
LATEST
  • 1
    Just make it scale: An Aurora DSQL story
    • May 29, 2025
  • 2
    Reliance on US tech providers is making IT leaders skittish
    • May 28, 2025
  • Examine the 4 types of edge computing, with examples
    • May 28, 2025
  • AI and private cloud: 2 lessons from Dell Tech World 2025
    • May 28, 2025
  • 5
    TD Synnex named as UK distributor for Cohesity
    • May 28, 2025
  • Weigh these 6 enterprise advantages of storage as a service
    • May 28, 2025
  • 7
    Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more
    • May 28, 2025
  • 8
    Pulsant targets partner diversity with new IaaS solution
    • May 23, 2025
  • 9
    Growing AI workloads are causing hybrid cloud headaches
    • May 23, 2025
  • Gemma 3n 10
    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI
    • May 22, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Understand how Windows Server 2025 PAYG licensing works
    • May 20, 2025
  • By the numbers: How upskilling fills the IT skills gap
    • May 21, 2025
  • 3
    Cloud adoption isn’t all it’s cut out to be as enterprises report growing dissatisfaction
    • May 15, 2025
  • 4
    Hybrid cloud is complicated – Red Hat’s new AI assistant wants to solve that
    • May 20, 2025
  • 5
    Google is getting serious on cloud sovereignty
    • May 22, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.