aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Data
  • Engineering
  • Technology

Data Mesh Vs Data Fabric: A Tale Of Two New Data Paradigms

  • aster.cloud
  • January 8, 2023
  • 5 minute read

Data is one of the most critical components of any business, as it allows us to personalize and customize our products for potential consumers. As important as data is, studies have shown that about 50‑70% of data collected by organizations goes unused and becomes what Gartner calls Dark Data. We can attribute this large amount of unused data to the inefficiencies in the systems that manage them.

This post discusses how methods like Data Meshes and Data Fabrics, which have emerged in the past decade, can help mitigate the problems associated with data management.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

At the end of this post, you should understand what Data Meshes and Data Fabrics are, their differences, and why one may overtake the other.

What is a Data Mesh?

According to IBM, a Data Mesh is a decentralized data architecture that organizes data by a specific business domain, providing more ownership to the producers of a given dataset. By decentralizing data, a Data Mesh offers an alternative to the central data lake and team culture that has been present in companies for decades. It is important to note that Data Meshes are language‑agnostic and technology‑agnostic as it is an approach that focuses more on organizational changes.

Principles of a Data Mesh

Data Meshes are built on four fundamental principles, which are discussed in the paragraphs below:

  • The domain ownership principle: domain ownership enables the decentralization of data, meaning that domains that need a particular brand of data are tasked with gathering, cleaning, and managing the ingestion of that data. This principle requires domains to take responsibility for their data.
  • The data as a product principle: this Data Mesh principle explains that there are consumers for the data besides the domain responsible for the data. This principle requires that the data be considered and taken care of as an actual product.
  • The self‑serve data infrastructure platform principle: this principle requires that the technical complexities in the infrastructure for creating data should be abstracted. This abstraction is done because is complex and challenging to replicate the infrastructure required to build, execute, and monitor a data product in each domain that needs the data. This principle allows data consumers in other domains to focus on using the data instead of recreating the infrastructure.
  • The federated governance principle: this Data Mesh principle enables data standardization across the organization. As multiple domains can use the data produced by one domain, an organization must standardize formatting, governance, and other data features to enable collaboration and understanding.
Read More  Efficient GRC With Cybersecurity Tooling

What is a Data Fabric?

As defined by IBM, a Data Fabric is an architecture that facilitates the end‑to‑end integration of various data pipelines and cloud environments through intelligent and automated systems. It is adaptive, flexible, secure, and ensures a consistent user experience across all integrated environments.

With Data Fabric, we can monitor and manage our data applications regardless of where they live.

At the center of the Data Fabric is rich metadata that enables automation, which is designed to automate data integration, engineering, and governance between data providers and consumers.

Responsibilities of a Data Fabric

Alongside automation, the Data Fabric is tasked with the following responsibilities.

  • Accessing the data
    The Data Fabric architecture is tasked with aggregating data from various sources. It is important to note that the Data Fabric supplies a virtualization layer that allows us to collect the data without copying or moving it.
    Coupled with the virtualization layer, a Data Fabric architecture should boast robust data integration and extract, transform, and load (ETL) tools to move the data when necessary.
  • Managing the lifecycle of the data
    After collecting data from different sources, the Data Fabric ensures privacy and data compliance with regulations.

    • Governance and Privacy
      Data Fabrics ensure that the right people assess the correct data. Data Fabrics use active metadata to automate policy enforcement to achieve this level of privacy.
      These Data Fabric policies govern that certain aspects of data should be masked and accessed on a role‑based method. The Data Fabric policy also requires that we provide rich lineage information for the data, which means the data source, transformations made to the data, etc., should all be provided. Rich lineage information helps us fact‑check the data and optimize for quality.
    • Compliance
      The Data Fabric architecture ensures that the data complies with the regulations set by governing organizations like the General Data Protection Regulation (GDPR), Fair Credit Reporting Act, etc.
  • Exposing data
    Next, the Data Fabric is tasked with exposing the data to different data consumers through other enterprise search catalogs.
Read More  Enforce Ingress Best Practices Using OPA

What are the differences between Data Meshes and data and Data Fabrics?

As both data paradigms are created to aid data gathering, governance, and distribution, it is easy to notice similarities between them. However, the differences are also apparent and should be considered before an organization chooses a paradigm.

This section discusses the differences between the Data Mesh and Data Fabric paradigms.

  • Decentralized vs. centralized data storage
    In a Data Mesh, data is distributed in domains, with no single necessary control point.
    In a Data Fabric, data access is centralized with high‑speed servers for network and high‑performance resource sharing.
  • Automation vs. human Inclusion approach
    Data Meshes treat data as a product and rely on domain owners to drive the requirements for the data product.
    Data Fabric relies on automation for discovering, governing, suggesting, and delivering data to data consumers. This automation is based on a rich metadata foundation.
  • Architecture
    The Data Mesh paradigm is language- and technology‑agnostic and focuses more on organizational changes. The Data Mesh architecture follows a domain‑driven design and product thinking to overcome data challenges.
    Data Fabric is a more technical data integration solution. The Data Fabric architecture is also more compatible with technical, business, and operational data.

Which paradigm to choose?

It is conceivable to see Data Fabrics take the lead in the coming years regarding efficient data management. Data Fabrics connect the entire organization’s data and facilitate frictionless data sharing.

Because Data Fabrics center on automation, we can optimize data management and send real‑time insights and analytics to data users. Moreover, Data Fabrics offer increased security: the virtualization layer ensures that the data is not unnecessarily moved. Data Fabrics are also cost‑efficient.

Read More  Why We Advocate For Workload-Centric Over Infrastructure-Centric Development

However, Data Meshes and Fabrics are not mutually exclusive. Data Fabrics can enable Data Mesh implementation by automating repetitive tasks using Data Fabrics’ metadata insights. With a Data Fabric, data owners in the Data Mesh paradigm can achieve the capabilities to create data products.

Conclusion

This article discusses the Data Mesh and Fabric paradigms, their differences, and, more importantly, what data management method is expected to take the lead in the coming years.

Mia‑Platform Fast Data is a perfect example of the cohabitation between the paradigms, and it can help you shift from one to the other if needed. To understand more about Mia‑Platform Fast Data, check out this article.

Guest post originally published on Mia-Platform’s blog
Source: CNCF Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • Cloud Native Computing Foundation
  • Data Fabric
  • Data Mesh
You May Also Like
View Post
  • Engineering

Just make it scale: An Aurora DSQL story

  • May 29, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Reliance on US tech providers is making IT leaders skittish

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Examine the 4 types of edge computing, with examples

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

AI and private cloud: 2 lessons from Dell Tech World 2025

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

TD Synnex named as UK distributor for Cohesity

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Weigh these 6 enterprise advantages of storage as a service

  • May 28, 2025
View Post
  • Computing
  • Multi-Cloud
  • Technology

Pulsant targets partner diversity with new IaaS solution

  • May 23, 2025

Stay Connected!
LATEST
  • 1
    Just make it scale: An Aurora DSQL story
    • May 29, 2025
  • 2
    Reliance on US tech providers is making IT leaders skittish
    • May 28, 2025
  • Examine the 4 types of edge computing, with examples
    • May 28, 2025
  • AI and private cloud: 2 lessons from Dell Tech World 2025
    • May 28, 2025
  • 5
    TD Synnex named as UK distributor for Cohesity
    • May 28, 2025
  • Weigh these 6 enterprise advantages of storage as a service
    • May 28, 2025
  • 7
    Broadcom’s ‘harsh’ VMware contracts are costing customers up to 1,500% more
    • May 28, 2025
  • 8
    Pulsant targets partner diversity with new IaaS solution
    • May 23, 2025
  • 9
    Growing AI workloads are causing hybrid cloud headaches
    • May 23, 2025
  • Gemma 3n 10
    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI
    • May 22, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Understand how Windows Server 2025 PAYG licensing works
    • May 20, 2025
  • By the numbers: How upskilling fills the IT skills gap
    • May 21, 2025
  • 3
    Cloud adoption isn’t all it’s cut out to be as enterprises report growing dissatisfaction
    • May 15, 2025
  • 4
    Hybrid cloud is complicated – Red Hat’s new AI assistant wants to solve that
    • May 20, 2025
  • 5
    Google is getting serious on cloud sovereignty
    • May 22, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.