کتاب ساخت معماری‌های مدالیون

تکامل معماری داده

What Is a Medallion Architecture? • A Brief History of Data Warehouse Architecture • OLTP Systems • Data Warehouses • The Staging Area • Inmon Methodology • Kimball Methodology • Key Takeaways from Traditional Data Warehouses • A Brief History of Data Lakes • Hadoop’s Distributed File System • MapReduce • Apache Hive • Spark Project • Moving Forward with Data Lakes • A Brief History of Lakehouse Architecture • Founders of Spark • Emergence of Open Table Formats • The Rise of Lakehouse Architectures • Medallion Architecture and Its Practical Challenges • Conclusion

آماده‌سازی زمینه

Foundational Preconditions • Extra Landing Zones • Raw Data • Batch Processing • Real-Time Data Processing • Spark Structured Streaming • Change Data Feed • Change Data Capture • Considerations and Learning Resources • ETL and Orchestration Tools • Managing Delta Tables • Z-Ordering • V-Ordering • Table Partitioning • Liquid Clustering • Compaction and Optimized Writes • DeltaLog • Conclusion

رمزگشایی از معماری مدالیون

The Three-Layered Design • Bronze Layer • Processing Hierarchy • Processing Full Data Loads • Processing Incremental Data Loads • Data Historization Within the Bronze Layer • Schema Evolution and Management • MergeSchema and Schema Enforcement • Technical Validation Checks • Usage and Governance • The Bronze Layer in Practice • Silver Layer • Cleaning Data Activities • Designing the Silver Layer’s Data Model • Harmonization with Other Sources • 3NF and Data Vault • Operational Querying and Machine Learning • Managing Overlapping Requirements • Automation Tasks • The Silver Layer in Practice • Gold Layer • Star Schema • Star Schema Design Nuances • Curated, Semantic, and Platinum Layers • One-Big-Table Design • Serving Layer • The Gold Layer in Practice • Conclusion

ساخت پایه مدالیون با Microsoft Fabric

Our Case Study: Oceanic Airlines • Introducing Microsoft Fabric • Domains • Workspaces and Capacities • OneLake • Data Engineering with Spark • Data Warehousing with T-SQL • Other Fabric Workload Types • Setting Up the Foundation • Setting up Capacities • Setting up Domains • Setting up Workspaces • Creating Lakehouses • Capacity Considerations • Domain Considerations • Workspace Considerations • Lakehouse Entities Considerations • Storage Account Considerations • Conclusion

ساخت لایه برنز

Building the Data Pipeline • Deploying the AdventureWorks Sample Database • Set Up an Azure SQL Database Connection • Creating a New Data Pipeline • Additional Considerations • Implementation of Lakehouse Tables • Traverse Parquet Files to Managed Delta Tables • Using External Tables • Updating Tables with MERGE Operations • Spark Structured Streaming • Using Change Data Capture • Navigating Data Handling Techniques • Schema Management • Create Tables Without Defining Schemas • Define Schemas with the DataFrame API • SQL DDL Statements • YAML or JSON Configurations • Metadata-Driven Approach • Databricks Auto Loader • Third-Party Tools • Handling Schema Evolution • Conclusion

ساخت لایه نقره‌ای

Quick Recap • Implementation of a Metadata-Driven Approach • Implementation of the Metadata Store • Implementation of Dynamic Data Validations • Improvement Areas • Data Cleansing • Implementation of Data Cleansing Tasks • Data Cleansing Considerations • Data Transformation Frameworks and Data Quality Tools • Optimization of Query Performance with Denormalization • Lightweight Enrichments • Data Historization • Optimization Jobs • Orchestration with Apache AirFlow • Final Recommendations • Silver-Layer Data as a Product • Conclusion

بهینه‌سازی لایه طلایی

Design of the Gold Layer • Transform Data Using a Star Schema • Creation of the Semantic Model • Creation of the First Power BI Report • Creation of Task Flows • Enhancements for Gold-Layer Design • Microsoft Fabric in Practice • Data Products • Data Governance with Microsoft Purview • Microsoft Purview Design Considerations • Guidance for Medallion Architectures • Conclusion

مطالعه موردی: داده، تحلیل و استراتژی کسب‌وکار در AP Pension

Medallion Architecture • Other Considerations • Final Recommendations

مطالعه موردی: Amadeus، یک رهبر فناوری در صنعت سفر

Medallion Architecture • FinOps • Data Models • Data Contracts • Data Governance

مطالعه موردی: تحول استراتژیک داده در ZEISS

Data Platform Evolution • Medallion Architecture • Data Products and Sharing • Recommendations and Best Practices

مقیاس‌پذیری معماری مدالیون

Decentralization of Data Management • Flexibility in Federation • Medallion Mesh • Number of Medallion Architectures • Medallion Inner Architecture Variations • Separate Data Product Layers • Tailored Medallions Architectures • Adaptability of the Bronze Layer • Silver Layer Variations • Gold Layer Variations • Enterprise Data Models • Master Data Management • Reference Data Management • Conclusion

حاکمیت و امنیت مدالیون

Data Governance • Governance Within a Medallion Architecture • Unity Catalog • Medallion Architecture with Unity Catalog • Data Contracts • Contracts Within a Catalog • Contracts Within a Metastore • Data Contracts Using YAML Files and GitOps • Other Data Contract Specifications • Data Security and Access Management • Conclusion

معماری‌های مدالیون آینده با هوش مصنوعی مولد

Unstructured Data Processing • Retrieval-Augmented Generation • Bronze Layer • Silver Layer • Gold Layer • Integration of LLMs and Medallion Architectures • Role of Agents • Training and Fine-Tuning LLMs • Future of Medallion Architectures • Conclusion