Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
ATZ-Fachkongress

Chassis.tech plus 2024


4 Kongresse in einer Veranstaltung

Treffen Sie in München die Fachleute von Chassis, Lenkung, Bremsen sowie Reifen/Räder.
Erweiterte Suche
Anmelden
nach oben

2023 | Buch

Evolution of Data Architecture Kapitel lesen Erstes Kapitel lesen

Data Fabric and Data Mesh Approaches with AI

A Guide to AI-based Data Cataloging, Governance, Integration, Orchestration, and Consumption

verfasst von: Eberhard Hechler, Maryela Weihrauch  , Yan (Catherine) Wu

Verlag: Apress

Enthalten in: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Einloggen, um Zugang zu erhalten
insite
SUCHEN

Über dieses Buch

Understand modern data fabric and data mesh concepts using AI-based self-service data discovery and delivery capabilities, a range of intelligent data integration styles, and automated unified data governance—all designed to deliver "data as a product" within hybrid cloud landscapes.

This book teaches you how to successfully deploy state-of-the-art data mesh solutions and gain a comprehensive overview on how a data fabric architecture uses artificial intelligence (AI) and machine learning (ML) for automated metadata management and self-service data discovery and consumption. You will learn how data fabric and data mesh relate to other concepts such as data DataOps, MLOps, AIDevOps, and more. Many examples are included to demonstrate how to modernize the consumption of data to enable a shopping-for-data (data as a product) experience.

By the end of this book, you will understand the data fabric concept and architecture as it relates to themes such as automated unified data governance and compliance, enterprise information architecture, AI and hybrid cloud landscapes, and intelligent cataloging and metadata management.

What You Will Learn

Discover best practices and methods to successfully implement a data fabric architecture and data mesh solutionUnderstand key data fabric capabilities, e.g., self-service data discovery, intelligent data integration techniques, intelligent cataloging and metadata management, and trustworthy AIRecognize the importance of data fabric to accelerate digital transformation and democratize data accessDive into important data fabric topics, addressing current data fabric challengesConceive data fabric and data mesh concepts holistically within an enterprise contextBecome acquainted with the business benefits of data fabric and data mesh

Who This Book Is For

Anyone who is interested in deploying modern data fabric architectures and data mesh solutions within an enterprise, including IT and business leaders, data governance and data office professionals, data stewards and engineers, data scientists, and information and data architects. Readers should have a basic understanding of enterprise information architecture.

Inhaltsverzeichnis

Frontmatter

Data Fabric and Data Mesh Foundation

Frontmatter
Chapter 1. Evolution of Data Architecture
Abstract
This chapter introduces the motivation for looking into data architectures. It shares an overview about data architecture evolution transitioning from traditional data warehouses to big data and data lakes and their main characteristics, values, and challenges. It outlines industry requirements in a data-driven world that ultimately led to the concept of a Data Fabric.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 2. Terminology: Data Fabric and Data Mesh
Abstract
This chapter explains the key terms that will be used throughout this book, the terms Data Fabric and Data Mesh, and how these two terms relate to each other. We introduce the term data-as-a-product or shopping-for-data and provide a high-level introduction into AI-infused Data Fabric capabilities. The chapter concludes with a description of a data product as a key concept of a Data Mesh.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 3. Data Fabric and Data Mesh Use Case Scenarios
Abstract
This chapter walks through several use cases for implementing a Data Fabric and Data Mesh that also represent business-relevant entry points. Data governance and privacy initiatives are ongoing in almost every organization, enabling access to enterprise data and AI artefacts across platforms to the people who have a business need. Other use cases are driven by hybrid cloud data integration; the need for a comprehensive view on customers, vendors, and other parties for better business outcome; and development and integration of trustworthy AI into business processes.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 4. Data Fabric and Data Mesh Business Benefits
Abstract
This chapter dives into business drivers and pain points that we hear in our conversations with enterprises. The business benefits of creating a Data Fabric architecture and also implementing a Data Mesh solution are discussed from the perspective of the technical, primarily data engineering team as well as the business teams consuming the data. We discuss a needed cultural shift related to managing data in an organization that looks at holistic data ownership of data source owners, data engineers, and data consumers.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu

Key Data Fabric and Data Mesh Capabilities and Concepts

Frontmatter
Chapter 5. Key Data Fabric and Data Mesh Capabilities
Abstract
A state-of-the-art Data Fabric architecture and Data Mesh solution is unquestionably linked to the knowledge catalog as one of its prime components. What differentiates a modern knowledge catalog from traditional ones are AI-infused capabilities to automate tasks and to provide self-service capabilities. AI is without dispute an inevitable domain that characterizes a modern Data Fabric and Data Mesh. Infusing AI generates additional added value specifically for business users, such as delivering trustworthy AI.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 6. Relevant ML and DL Concepts
Abstract
At the heart of a Data Fabric and Data Mesh is the use of artificial intelligence (AI) and machine learning (ML) technologies to automate complex data tasks to the greatest extent possible. Therefore, understanding the concepts of AI and ML is the foundation for implementing both concepts in an enterprise. If you are already an AI/ML practitioner, you might skip this chapter. If you are not sure, please take a quick assessment by answering the following questions, and please continue with this chapter should you need more clarity:
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 7. AI and ML for a Data Fabric and Data Mesh
Abstract
This chapter provides a deep dive into the exploitation of AI and ML for various Data Fabric and Data Mash topics and tasks, such as data discovery, data access, and data profiling, analyzing the “digital exhaust” of Data Fabric and Data Mesh process steps, ML-based entity matching, automated data quality assessments, and semantic enrichment of the underlying data.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 8. AI for Entity Resolution
Abstract
It is widely accepted that an organization's success is increasingly dependent on its ability to derive value from the data it has. However, many organizations are still stuck on the first step – understanding the data – especially as the volume and complexity of data continue to grow. Think about a simple question: How many customers does your enterprise have? For many businesses, providing an accurate answer to this question remains difficult.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 9. Data Fabric and Data Mesh for the AI Lifecycle
Abstract
With the development of AI in technology and business, AI is no longer an experiment limited to a select few data scientists. It will penetrate all aspects of enterprise business operations and continue to innovate and optimize for new business scenarios. Now the focus shifts from the competition of AI algorithms to how to combine the strength of expert teams and AI technology for the actual needs of the enterprise and industry to generate business value.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu

Deploying Data Fabric and Data Mesh in Context

Frontmatter
Chapter 10. Data Fabric Architecture Patterns
Abstract
A specific Data Fabric architecture is determined by its business and IT context and intent, meaning that not every implementation is identical. A Data Fabric could for instance serve different data consumption patterns, such as real-time transactional inference of AI-based insights, trustworthy AI scenarios, or AI governance purposes. A specific implementation of a Data Fabric also depends on concrete solution requirements, such as the ones associated with a Data Mesh solution (e.g., data-as-a-product) and whether the Data Fabric should serve certain technologies, such as IoT, edge computing, or 5G. Finally, intelligent information integration can be underpinned with different and complementary methods, such as data virtualization, replication, streaming, etc., which has an impact on the underlying Data Fabric architecture. Integration challenges within a hybrid cloud landscape leveraging public cloud services may differ from integration needs within a private cloud and on-premises landscape.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 11. Data Fabric Within an Enterprise Architecture
Abstract
Any data architecture, and therefore also our Data Fabric architecture, needs to be looked at in conjunction with the implemented application architecture in an existing enterprise landscape. Many organizations are in the process to modernize and digitalize their application and data landscape. Applications have different requirements with respect to data characteristics, which may recommend a particular data architecture implementation over another, for example, characterized by data access based on data virtualization or data replication and transformation.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 12. Data Fabric and Data Mesh in a Hybrid Cloud Landscape
Abstract
In this chapter, we investigate additional facets of both concepts that arise specifically from hybrid cloud deployments. We briefly review the term hybrid cloud and how it relates to on-premises. Although cloud services are rated very highly in importance, they nevertheless create new challenges regarding access, integration, and consumption of data and AI assets across the organization. What does Data Fabric architecture and Data Mesh solution mean in the context of a hybrid cloud landscape, and what are the key challenges, for instance, related to addressing data and AI governance and trustworthy AI?
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 13. Intelligent Cataloging and Metadata Management
Abstract
Suppose you are a business analyst and you need to find the customer purchase records of a certain market in the last quarter from 1.3 million tables and billions of records to make a predictive analysis of the consumption trend in this region for the next quarter. How are you going to do it? This task is like looking for a needle in a haystack. What is even more frustrating is that when you finally find the relevant data after spending weeks on source data exploration, it is out of date, and new essential data is available. This example illustrates that it is not sufficient for companies to make data accessible; they also need to make it discoverable, understandable, and consumable in near real time to gain timely and relevant insights from the data.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 14. Automated Data Fabric and Data Mesh Aspects
Abstract
The vigilant reader of this book has certainly noticed the distinguished focus that we have put on applying AI with automation and intelligent augmentation and optimization to nearly all aspects of a Data Fabric architecture and Data Mesh solution. There are indeed numerous areas of both concepts, which are increasingly optimized with AI-infused automation, such as automated workload performance prediction and runtime adjustment, automated capacity planning and resource demand estimation (e.g., CPU capacity, network bandwidth, memory sizes, etc.), automated query generation, intelligent information integration, automated data curation, and automated creation of data products.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 15. Data Governance in the Context of Data Fabric and Data Mesh
Abstract
Companies generate and manage large amounts of sensitive information about their employees, customers, and business during their operational and analytical activities. This information gives companies a competitive advantage and at the same time brings great risks. The exposure of sensitive information can lead to serious consequences, such as lawsuits. Therefore, companies need to implement a purposeful and well-planned data governance platform.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu

Current Offerings and Future Aspects

Frontmatter
Chapter 16. Sample Vendor Offerings
Abstract
As we have mentioned in Chapter 2, Gartner named Data Fabric as one of the top ten technology trends in data and analytics for 2019 and 2021 and tipped it as one of the top ten emerging technology trends for 2022. Meanwhile, Forrester said that of the 25,000 reports the company published last year, reports on Data Fabric ranked in the top ten downloads for 2020.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 17. Data Fabric and Data Mesh Research Areas
Abstract
Imagine a hyper-automated Data Fabric or Data Mesh that is self-acting, self-improving, and self-optimizing, meaning that it can operationalize intelligence and AI insight in real time without human intervention or significantly reducing human-driven data and AI management tasks. Such a modern Data Fabric architecture or Data Mesh solution is not only infused with AI to gain more relevant insight, discover assets, or activate the digital exhaust; it is action-oriented and capable to auto-tune and auto-correct operations of both concepts. Applying this to AI governance, imagine a new international law or regulation, which affects your company in terms of establishing new or adjusting existing data-related processes, for example, guaranteeing bias-free AI models. A modern Data Fabric or Data Mesh should be able to apply ontology-based semantic searches to identify relevant data and AI assets that are affected by the law or regulation and autonomously infuse these assets to be processed by the trustworthy AI component of your Data Fabric architecture.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Chapter 18. In Summary and Onward
Abstract
Reaching the end of the book, it must have become obvious that both a Data Fabric architecture and Data Mesh solution are inevitably associated with applying AI, intelligent knowledge, and automation. Indeed, infusing AI is required to enable intelligent cataloging, to generate active metadata, to build semantic knowledge graphs, and to gain necessary and holistic insight to improve, optimize, and automate tasks and to enable self-service generation of data products that are ready for consumption. These capabilities are enabled via a knowledge catalog that stores active metadata and data product specifications as well.
Eberhard Hechler, Maryela Weihrauch, Yan (Catherine) Wu
Backmatter
Metadaten
Titel
Data Fabric and Data Mesh Approaches with AI
verfasst von
Eberhard Hechler
Maryela Weihrauch
Yan (Catherine) Wu
Copyright-Jahr
2023
Verlag
Apress
Electronic ISBN
978-1-4842-9253-2
Print ISBN
978-1-4842-9252-5
DOI
https://doi.org/10.1007/978-1-4842-9253-2

Premium Partner

    Bildnachweise