aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Design
  • Engineering

Access Control Set Up On Storage Transfer Service

  • aster.cloud
  • February 17, 2023
  • 3 minute read

Google Cloud Storage Transfer Service (STS) enables you to move or backup your data to a Cloud Storage bucket or POSIX file system. It offers flexibility and allows horizontal scaling, encryption and checksum verification. In most cases, storage administrators use this to migrate the whole file system including all files with different access rights.

It can be challenging to migrate data in a multi-tenant storage environment when users need to make sure POSIX access is observed. Transfer tool has to be able to transfer only what is allowed. Although the STS operates under root privileges, there are ways to apply access control to the transfer agent(s). Users / Storage administrators can transfer data with the peace of mind that POSIX access rights are being enforced during the transfer operation


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

Without the mount / permission setup, the transfer agent(2) can transfer any data within the mount. In the setup shown in the diagram, an agent mounted at /mnt/ can transfer data from all the underlying directories – datasetA, datasetB, datasetC, and datasetD to Cloud Storage.

If you need to set up the transfer service for a specific group of users in a multi-tenant NAS / NFS storage environment, you need to limit the access to data which does not belong to the group by using specific mount / POSIX user account permissions. Multiple set up can be done with different POSIX user account permissions for different groups.

With permission and / or mount setup shown in the diagram, specific agent(s) can only transfer data from its specific mount and the underlying directories to Cloud Storage. All agent(s) in Agent Pool A can only access data from /mnt/dataset A. They do not have access to other dataset B C and D. Also, All agent(s) in Agent Pool B can only access data from /mnt/dataset B. They do not have access to dataset A, C and D. 

Read More  Announcing Open Innovations For A New Era Of Systems Design

Access control methods overview:

Set up the container(s) as transfer agent(s)  

There are two methods to limit transfer agent(s) access right to data. Both access control methods can be used solely or combined. 

  • Mount
  • POSIX user account permission

Mount – By specifying the directory in the mount option at the container, transfer service 

CANNOT access data outside of the mount location. Even if the transfer node / VM has access to the directory, data cannot be accessed if it is not mounted to the docker container / agent. 

Advantage: In a large file system, projects are most likely being organized in directory structure. By mounting a specific directory to the agent, the agent pool can target specific project data.

POSIX user account permission – Although agent(s) may have access to the top of the directory, access is limited based on the POSIX user account permission. 

Advantage: In a multi-tenant environment, administrators can limit the user access with POSIX permissions. 

Agent Setup Commands:

Here is the example command to setup the the transfer agent with Mount / POSIX user account permissions:

Setup the service account json key

gcloud iam service-accounts keys create service_account.json --iam-account=<service account name>

Setup the transfer agent:

docker run --ulimit memlock=64000000 -d --rm -v <directory to be transferred>:<directory to be transferred> -v <path to the service account key>/service_account.json:<path to the service account key>service_account.json --user 1000:1001 gcr.io/cloud-ingest/tsop-agent:latest --project-id=thomashk-migration1 --creds-file=<path to the service account key>service_account.json --hostname=$(hostname) --agent-pool=<name of the agent pool>

Mount: using the “–user uid:gid”, this will setup the POSIX user permission.

POSIX user account permission: Using the -v <directory to be transferred>:<directory to be transferred> 

Read More  Introducing Data Studio As Our Newest Google Cloud Service

Error message shown if transfers directory without permission: 

The following error will show up: 

Error message:
202x-11-23T10:10:10Z transferOperations/xxxx_transferJobs/OPI476239788276320xxx_000000000166923xxxx FIND FAILED PERMISSION_FAILURE PERMISSION_DENIED failed to list the contents of directory “/etc/ssl/private/”. open /etc/ssl/private/: permission denied

You can set up the multiple transfer agents with access control in a multi-tenant NAS environment:

Google Cloud Storage Transfer Service does not only offer you the ability to move or backup data to the Cloud Storage bucket. The common use cases include transfer between Cloud Storage Buckets, Move Cloud Storage data to another location and Transfer data between file systems. Please refer to the link for future reference and documentation.

By: Thomas Leung (HPC Technical Solution Consultant, Professional Service Organization, Google Cloud)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Google Cloud
  • Google Cloud Storage
  • Storage
  • Tutorials
You May Also Like
View Post
  • Engineering

Just make it scale: An Aurora DSQL story

  • May 29, 2025
View Post
  • Engineering
  • Technology

Guide: Our top four AI Hypercomputer use cases, reference architectures and tutorials

  • March 9, 2025
View Post
  • Computing
  • Engineering

Why a decades old architecture decision is impeding the power of AI computing

  • February 19, 2025
View Post
  • Engineering
  • Software Engineering

This Month in Julia World

  • January 17, 2025
View Post
  • Engineering
  • Software Engineering

Google Summer of Code 2025 is here!

  • January 17, 2025
View Post
  • Data
  • Engineering

Hiding in Plain Site: Attackers Sneaking Malware into Images on Websites

  • January 16, 2025
View Post
  • Computing
  • Design
  • Engineering
  • Technology

Here’s why it’s important to build long-term cryptographic resilience

  • December 24, 2024
IBM and Ferrari Premium Partner
View Post
  • Data
  • Engineering

IBM Selected as Official Fan Engagement and Data Analytics Partner for Scuderia Ferrari HP

  • November 7, 2024

Stay Connected!
LATEST
  • 1
    Just make it scale: An Aurora DSQL story
    • May 29, 2025
  • 2
    Reliance on US tech providers is making IT leaders skittish
    • May 28, 2025
  • Examine the 4 types of edge computing, with examples
    • May 28, 2025
  • AI and private cloud: 2 lessons from Dell Tech World 2025
    • May 28, 2025
  • 5
    TD Synnex named as UK distributor for Cohesity
    • May 28, 2025
  • Weigh these 6 enterprise advantages of storage as a service
    • May 28, 2025
  • 7
    Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more
    • May 28, 2025
  • 8
    Pulsant targets partner diversity with new IaaS solution
    • May 23, 2025
  • 9
    Growing AI workloads are causing hybrid cloud headaches
    • May 23, 2025
  • Gemma 3n 10
    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI
    • May 22, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Understand how Windows Server 2025 PAYG licensing works
    • May 20, 2025
  • By the numbers: How upskilling fills the IT skills gap
    • May 21, 2025
  • 3
    Cloud adoption isn’t all it’s cut out to be as enterprises report growing dissatisfaction
    • May 15, 2025
  • 4
    Hybrid cloud is complicated – Red Hat’s new AI assistant wants to solve that
    • May 20, 2025
  • 5
    Google is getting serious on cloud sovereignty
    • May 22, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.