aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Design
  • Solutions

Google’s Open-Source Solution To DFDL Processing

  • aster_cloud
  • May 27, 2022
  • 4 minute read

The cloud has become the choice for extending and modernizing applications, but there are some situations where the transition is not straightforward, such as migrating applications that access data from a mainframe environment.  Migrating the data and the applications at certain points can be outsync.  Mechanisms need to be in place during the transition to support interoperability with legacy workloads and  access data out of the mainframe.  For the latter, the Data Format Description Language  (DFDL) which is an open standard modeling language from the Open Grid Forum (OGF), has been used to access data from a mainframe, e.g. IBM Integration Bus.

DFDL uses a model or schema that allows text or binary data to be parsed from its native format and to be presented as an information set out of the mainframe (i.e., logical representation of the data contents, independent of the physical format).


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

DFDL Processing with IBM App Connect

If we talk about solutions for parsing and processing data described by DFDL, one of the options in the past has been IBM App Connect which allows development of custom solutions via IBM DFDL. The following diagram represents a high-level architecture of DFDL Solution implementation on IBM App Connect:

 

IBM App Connect brings stable integration to the table at an enterprise level cost. According to IBM’s sticker pricing as of May 2022, IBM App Connect charges $500 and above per month for using the App Connect with IBM Cloud services. These prices are excluding the cost of storing and maintaining DFDL Definitions in the Mainframe. With the introduction of Tailored Fit Pricing on IBMz15, cost of maintaining the mainframe can range from $4900 to $9300 per month over the span of 5 years, which may be costly for a small/medium business only wanting to process data defined by DFDL.

Read More  Introducing New Google Cloud Manufacturing Solutions: Smart Factories, Smarter Workers

Introducing Google Open-Source DFDL Processor with Google Cloud

At Google our mission is to build for everyone, everywhere. With this commitment in mind, the Google Cloud team has developed and open-sourced the solution for DFDL Processor which can be easily accessible and customizable for organizations to  use it.

We understand that mainframes can be expensive to maintain and use, which is why we have integrated Cloud Firestore and Bigtable as the databases to store the DFDL definitions. Firestore can provide 100K reads, 25K writes, 100K deletes, and 1TB of storage per month for approximately $186 per month. While on the other hand Bigtable provides a fast, scalable database solution for storing terabytes, or even petabytes of data at a relatively lower cost too. This move away from the mainframe and adopting cloud-native database solutions can save organizations thousands of dollars every month.

Next, we have substituted App Connect with a combination of our open-source DFDL processor, Cloud Pub/Sub service and open-source Apache Daffodil Library. Pub/Sub provides the connection between the mainframe and the processor, and from the processor to the downstream applications. The Daffodil Library helps in compiling schemas, and outputting infosets for the given DFDL definition and message. The total cost of employing the Pub/Sub service and the Daffodil Library comes out to be approximately $117 per month, which means an organization can save a minimum of $380 per month by using this solution.

The table below shows a summary of the cost difference breakdown between the solutions as discussed above:

 

How it works

The data described by the DFDL usually needs to be available in widely used formats such as JSON, in order to be consumed by downstream applications which might  have already been migrated to a cloud native environment. To achieve the consumption of the data, cloud native applications/services can be implemented in conjunction with Google Cloud Services, which accepts the textual or binary data as input from the mainframe , fetches corresponding DFDL from a database, and finally compiles and outputs the equivalent JSON for the downstreaming applications to consume.

Read More  A More Sustainable Future Should Be A More Open Future

The following diagram describes a high level architecture to be presented

 

An application can be built to process the information being received from the mainframe, e.g a DFDL Processor Service, leveraging the Daffodil API to parse the data against a corresponding DFDL schema and output the JSON.

DFDL schema definitions can be potentially migrated and stored in Firestore or Bigtable. Since these definitions rarely change and they can be stored in a key-value pair format, the storage of preference is a non-relational managed database.

Google Cloud Pub/Sub, can leverage an eventing mechanism that receives the binary/textual message from a Data Source, i.e. the mainframe, in a Pub/Sub topic.  This feature will  allow the DFDL Processor to access the data, to retrieve the corresponding DFDL definition from Firestore or Bigtable and finally pass both on to the Daffodil API to compile and output the JSON result. The JSON result is finally published into a resulting Pub/Sub topic for any downstream application to consume. It is recommended to follow CloudEvent schema specification which allows to describe events in common formats, providing interoperability across services platforms and systems.

You can find examples of the implementation in Github:

  • Firestore Example
  • Bigtable Example

Conclusion

In this post, we have discussed different pipelines used to process data defined by DFDL, and cost comparisons of these pipelines. Additionally, we have demonstrated how to use Cloud Pub/Sub, Firestore, and Bigtable to create a service which is capable of listening to binary event messages,  extract the corresponding DFDL definition from a  managed database, and process it to output a JSON which can then be consumed by downstream applications using well-established technologies and libraries.

Read More  Telia Company Partners With VMware To Accelerate Its Cloud-Native Journey

1. Price comparison analysis as of May 2022 and subject to change based on usage

 

 

By: Carolina Hernandez (Strategic Cloud Engineer) and Usama Ijaz (Cloud Technical Resident)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster_cloud

Related Topics
  • Application Modernization
  • Data Format Description Language
  • Google Cloud
  • IBM
  • IBM App Connect
  • Open Source
You May Also Like
Brush, Color, and Sketch pad
View Post
  • Cloud-Native
  • Design
  • Engineering

6 Security Best Practices For Cloud-Native Applications

  • November 17, 2023
Cloud
View Post
  • Design
  • Engineering
  • Public Cloud

The Impact Of Public Cloud Price Hikes

  • November 8, 2023
View Post
  • Design
  • Software Engineering

Organizing “spaghetti” Software So It Can Be Easily Modified

  • October 17, 2023
View Post
  • Data
  • Engineering
  • Platforms
  • Solutions

How ‘Anything Is Possible’ Automated Data Pipelines With BigQuery And Windsor.ai

  • September 27, 2023
Microsoft and Adobe
View Post
  • Platforms
  • Solutions

Microsoft And Adobe Partner To Deliver Cost Savings And Business Benefits

  • September 20, 2023
View Post
  • Design
  • Engineering

4 Ways To Improve Long Term Kubernetes Capacity And Cloud Costs In Kubernetes

  • September 9, 2023
Street Fighter Rockman
View Post
  • Platforms
  • Solutions

CAPCOM And Google Cloud Team Up To Strengthen The Reliability Of Live Service Game Launches For Millions Of Players Globally

  • August 28, 2023
Emission
View Post
  • Solutions
  • Technology

Microsoft And Accenture Partner To Tackle Methane Emissions With AI Technology

  • August 27, 2023

Stay Connected!
LATEST
  • Web 1
    Mastering the Art of Load Testing for Web Applications
    • November 29, 2023
  • Data center. Servers. 2
    Intel Granulate Optimizes Databricks Data Management Operations
    • November 27, 2023
  • Ubuntu. Chiselled containers. 3
    Canonical Announces The General Availability Of Chiselled Ubuntu Containers
    • November 25, 2023
  • Cyber Monday Sale. Guzz. Ideals collection. 4
    Decode Workweek Style with guzz
    • November 23, 2023
  • Guzz. Black Friday Specials. 5
    Art Meets Algorithm In Our Exclusive Shirt Collection!
    • November 23, 2023
  • Presents. Gifts. 6
    25 Besties Bargain Bags Below $100 This Black Friday 2023
    • November 22, 2023
  • Electronics 7
    Top 10+1 You Can’t Do Without For The Holidays: Electronics Edition.
    • November 20, 2023
  • Microsoft. Windows 8
    Ousted Sam Altman To Lead New Microsoft AI Team
    • November 20, 2023
  • Sale. Deals. Discount. 9
    The 50 Best Electronic Deals To Get On Amazon Before Cyber Monday 2023
    • November 20, 2023
  • Portrait of Rosalynn Carter, 1993 10
    Former First Lady Rosalynn Carter Passes Away at Age 96
    • November 19, 2023
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • Oracle | Microsoft 1
    Oracle Cloud Infrastructure Utilized by Microsoft for Bing Conversational Search
    • November 7, 2023
  • Riyadh Air and IBM 2
    Riyadh Air And IBM Sign Collaboration Agreement To Establish Technology Foundation Of The Digitally Led Airline
    • November 6, 2023
  • Ingrasys 3
    Ingrasys Unveils Next-Gen AI And Cooling Solutions At Supercomputing 2023
    • November 15, 2023
  • Cloud 4
    DigitalOcean Currents Report Finds That Adoption Of AI/ML, And Investments In Cybersecurity And Multi-Cloud Strategies Are On The Rise At Small Businesses
    • November 9, 2023
  • OpenAI 5
    OpenAI Announces Leadership Transition
    • November 18, 2023
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.