Data governance includes people, processes, and technology. Together, these principles enable organizations to validate and manage across dimensions such as:

  1. Data management, including data and pipelines lifecycle management and master data management.
  2. Data protection, spanning data access management, data masking and encryption, along with audit and compliance.
  3. Data discoverability, including data cataloging, data quality assurance, and data lineage registration and administration.
  4. Data accountability, with data user identification and policies management requirements.

While prioritizing investment in their people to achieve the desired cultural transformation and processes to increase operational effectiveness and efficiency will help enterprises, the technology pillar is the critical enabler for people to interact with data and for organizations to truly govern their data initiatives.

Financial services organizations are faced with particularly stringent data governance requirements regarding security, regulatory compliance, and general robustness. Once people are aligned and processes are defined, the challenge for technology comes to the picture: solutions should be flexible enough to complement existing governance processes and be cohesive across data assets to help make data management simpler.

In the following sections, starting with standard requirements for data governance implementations in financial services, we will cover how these correspond to Google Cloud services, open-source resources, and third-party offerings. We will share an architecture capable of supporting the entire data lifecycle, based on our experience implementing data governance solutions with world-class financial services organizations.

Data management

Looking first at the data management dimension, we have compiled some of the most common requirements, along with the relevant Google Cloud services and capabilities from the technology perspective.

 

 

Data Management Requirements Services & Capabilities
Data and pipelines lifecycle management Batch ingestions: Data pipelines management, scheduling, and data pipelines processing logging

Streaming Pipelines: Metadata

Data lifecycle management

 

Operational metadata including both state and statistical metadata

A comprehensive end-to-end data platform

GCS Object Lifecycle

BigQuery data lifecycle

Data Fusion pipeline lifecycle management, orchestration, coordination, and metadata management

Dataplex Intelligent automation data lifecycle management

Cloud Logging, Cloud Monitoring

Informatica Axon Data Governance

Compliance Facilitate regulatory compliance requirements Easily expandable to help comply with CCPA, HIPAA, PCI, SOX, and GDPR, through security controls implementation using IAM, CMEKs, BQ column-level access control, BQ Table ACL, Data Masking, Authorized views, DLP PII data 

Identification, and Policy tags.

DCAM data and analytics assessment framework

CDMC best practice assessment and certification

Master Data Management Duplicate Suspect Processing rules

Solution and department scope

Enterprise Knowledge Graph

KG Entity Resolution/reconciliation and Financial Crime Record matching MDM + ML

Tamr Cloud-Native Master Data Management

Site Reliability Data Pipelines SLA

Data at Rest SLA

SLAs applied to data pipeline

SLAs applied to services managing data

DR strategies for data

 

Registering, creating, and scheduling data pipelines is a recurring challenge that organizations face. Similarly, data lifecycle management is a key part of a comprehensive data governance strategy.

This is where Google Cloud can help, offering multiple data processing engines and data storage options tailored for each need, but that are integrated and make orchestration and cataloging easy.

Data protection

Financial organizations demand world-class data protection services and capabilities to support their defined internal processes and help meet regulatory compliance requirements.

 

Data Protection Requirements Services & Capabilities
Data Access Management Definition of access policies

 

Multi-cloud approval workflow integration*

Access Approvals

Hierarchical resources & policies

  • Users, Authentication, Security (2FA), Authorization
  • Resources, Separation boundaries, Organization policies, Billing and quota, Networking, Monitoring

Event Threat Detection

Multi-cloud Approval workflow by 3rd Party – Collibra*

Data Audit & Compliance Operational metadata logs capture

Failing process alerting and root cause identification

Cloud Audit Logs

Security Command Center

Access Transparency & Access Approval

StackDriver Logging

Collibra Audit Logging

Security Health Data vulnerabilities identification

Security health checks

Security Health Analytics

Security Health Analytics

Data Masking and Encryption Storage-level encryption metadata

Application-level encryption metadata

PII data identification and tagging

Encryption at rest, Encryption in transit, KMS

 

Cloud DLP Transformations, De-identification

 

Access management, along with data and pipeline audit, is a common requirement that should be managed across the board for all data assets. These security requirements are usually supported by security health checks and automatic remediation processes.

Specifically on data protection, capabilities like data masking, data encryption, or PII data management should be available as an integral part for processing pipelines, and be defined and managed as policies.

Data discoverability

Data describes what an organization does, how it relates to its users, competitors, and regulatory institutions. This is why data discoverability capabilities are crucial for financial organizations.

 

Data Discoverability Requirements Services & Capabilities
Data Cataloging Data catalog storage

Metadata tags association with fields

Data classification metadata registration

 

Schema Versions control

Schema definition before data loading

Data Catalog

Column level tags

Dataplex logical aggregations (Lakes,  Zones and Assets)

DLP

Collibra Catalog

Collibra Asset version control

Collibra Asset Type creation and Asset pre-registration

Alation Data Catalog

Informatica Enterprise Data Catalog

Data Quality On ingestion data quality rules definition (like regex validations for each column)

 

Issues remediation lifecycle management

BigQuery DQ

Dataplex

Data quality with Dataprep

Collibra DQ

Alation Data Quality

CloudDQ declarative Data Quality validation (CLI)*

Informatica Data Quality

Data Lineage Storage and Attribute level Data Lineage

 

Multi-cloud/on-premises  lineage

Cloud Data Fusion Data Lineage

  • Understand the flow
  • Granular visibility into flow of data
  • Operational View
  • Openess or share lineage

Data Catalog & BigQuery

Collibra lineage

  • multi-cloud/on-premises management

Alation Data Lineage

Data Classification Data Discovery and Data Classification metadata registration DLP Discovery and classification

A data catalog is the foundation on which a large part of a data governance strategy is built. You need automatic classification options and data lineage registration and administration capabilities to make data discoverable. Dataplex is a fully managed data discovery and metadata management service that offers unified data discovery of all data assets, spread across multiple storage targets. Dataplex empowers users to annotate business metadata, providing necessary data governance foundation within Google Cloud, and providing metadata that can be integrated later with external metadata by a multi-cloud or enterprise-level catalog. The Collibra Catalog is an example of an enterprise data catalog on Google Cloud that complements Dataplex by providing enterprise functionality such as an operating model that includes the business and logical layer of governance, federation and the ability to catalog across multi-cloud and on-premises environments.

Data quality assurance and automation is the second foundation of data discoverability. To help with that effort Dataprep is another tool for assessing, remediating, and validating processes, and can be used in conjunction with customized data quality libraries like Cloud Data Quality Engine, a declarative and scalable data quality validation command-line Interface. Collibra DQ is another data quality assurance tool, and uses machine learning to identify data quality issues, recommend data quality rules and allow for enhanced discoverability.

Data accountability

Identifying data owners, controllers, stewards, or users, and effectively managing the related metadata, provides organizations with a way to ensure trusted and secure use of the data. Here we have the most commonly identified data accountability requirements and some tools and services you can use to meet them.

 

Data Accountability Requirements Services & Capabilities
Data User Identification Data owner and dataset linked registration

Data steward and dataset linked registration

Users role based data usage logging

Dataplex

Data Catalog

Analytics Hub

Collibra Data Stewardship

Alation Data Stewardship

Policies Management  Domain based policies management

 

Column level policies management

Cloud DLP

Dataplex

Policy Tags

BigQuery Column Level Security

Collibra Policy Management

Domain Based Accountability Governed data sharing IAM and ACL role based access

Analytics Hub

 

Having a centralized identity and access management solution across the data landscape is a key accelerator to defining a data security strategy. Core capabilities should include user identification, role- and domain-based access policy management, and a policy-managed data access authorization workflows.

Data governance building blocks to meet industry standards

Given these capabilities, we provide a reference architecture for a multi-cloud and centralized governance environment that enables a financial services organization to meet its requirements. While here we focus on the technology pillar of data governance, it is essential that people and processes are also aligned and well-defined.

The following architecture does not intend to cover each and every requirement presented above, but provides core building blocks for data governance implementation to meet industry standards as far as the technology pillar is concerned at the time of writing this blog.

 

1. Data cataloging is a central piece in any data governance technology journey. Finance enterprises often need to deal with several storage systems residing in multiple cloud providers and also on-premises. As such, an enterprise-level catalog, a “catalog of catalogs”, that centralizes and makes discoverable all the data assets in the organization, is a helpful capability to helping the business get the most from its data, wherever it sits.

Even when Google Data Catalog supports non-Google Cloud data assets through open-source connectors, a third-party cataloging solution (such as Collibra) may be well-suited to help with this, providing connection capabilities to several storage systems and additional layers of metadata administration. For example, this could enable having the ability to pre-register data assets even before they are available in storage, and to integrate those once actual tables or filesets are created, including schema evolution tracking.

2. From a Google cloud perspective, data to be discovered, cataloged, or protected can reside in a data lake or a landing zone in Cloud Storage, an enterprise data warehouse in BigQuery, a high-throughput low-latency datastore like BigTable, or even in relational or NoSQL databases supported by Spanner, CloudSQL or Firestore, for example.

Gathering Cloud Data Catalog metadata such as tags is a multi-step process. Financial enterprises should standardize and automate as much as possible to have reliable and complete metadata. To populate the Data Catalog with labels, the Cloud Data Loss Prevention API (DLP) is a key player. DLP inspection templates and inspection jobs can be used to standardize tagging, sampling, and discovering data, and finally to tag tables and filesets.

Security and access control is another big concern for finance organizations given the sensitivity of the data they handle. Several encryption and masking layers are usually applied to the data. In these scenarios, sampling and reading data to determine which labels to add is a slightly more complex process, requiring decryption along the way.

In order to be able to do things like apply column-level policy tags to BigQuery, the DLP inspection job findings need to be published to an intermediate storage location accessible to a tagging job using Cloud Data Catalog. In these contexts, a Dataflow job could help handle the required decryption and tagging. There is a step by step community tutorial on that here.

Ensuring the right people accessing the right data across numerous datasets can be challenging. Policy Taxonomy tags, in conjunction with IAM access management, covers that need.

Google Cloud’s Dataplex service (discussed more below) will also help to automate data discovery and classification using dynamic schema detection, such that metadata can be automatically registered in a Dataproc Metastore or in BigQuery before finally being used by Data Catalog.

3. To understand the origin, movement, and transformation of data over time, data lineage systems are fundamental. These allow users to store and access lineage records and provide reliable traceability to identify data pipeline errors. Given the large volume of data in a finance enterprise data warehouse environment, an automated data lineage recording system can simplify data governance for users.

Finance organizations have to meet compliance and auditability standards, enforce access policies, and perform root cause analysis on poor data or failing pipelines. To do that, Cloud Data Catalog Lineage and Cloud Data Fusion Lineage provide traceability capabilities that can help.

4. Dataplex is a fundamental part of Google Cloud’s vision for data governance. Dataplex is an intelligent data fabric that unifies and automates data management and allows easy and graphical control for analytics processing jobs. This helps financial organizations meet the complex requirements for data and pipeline lifecycle management.

Dataplex also provides a way to organize data into logical aggregations called lakes, zones and assets. Assets are directly related to Cloud Storage files or tables in BigQuery. Those assets are logically grouped into zones. Zones can be typical data lake implementation zones like raw zones, refined zones, or analytics zones, or can be based on business domains like sales or finance. On top of that logical organization, users can define security policies across your data assets, including granular access control. This way, data owners can grant permissions while data managers can monitor and audit the access granted.

Build a data governance strategy in the cloud

For financial data governance implementations to have trust in their data, and meet regulatory compliance requirements, they must have a solid and flexible technology pillar from which to build processes and align people. Google Cloud can help build that comprehensive data governance strategy, while allowing you to add third-party capabilities to meet specific industry needs.

To learn more:

 

By: Daryus Medora (Strategic Cloud Engineer, Data & Analytics) and Oscar Pulido (Strategic Cloud Engineer, Data & Analytics)
Source: Google Cloud Blog

Previous Spanner On A Modern Columnar Storage Engine
Next Introducing More Ways To Protect Corporate Applications With BeyondCorp Enterprise