کتاب دلتا لیک: راهنمای جامع

مقدمه‌ای بر فرمت دریاچه‌واره دلتا لیک

The Genesis of Delta Lake • Data Warehousing, Data Lakes, and Data Lakehouses • Project Tahoe to Delta Lake: The Early Years Months • What Is Delta Lake? • Common Use Cases • Key Features • Anatomy of a Delta Lake Table • Delta Transaction Protocol • Understanding the Delta Lake Transaction Log at the File Level • The Single Source of Truth • The Relationship Between Metadata and Data • Multiversion Concurrency Control (MVCC) File and Data Observations • Observing the Interaction Between the Metadata and Data • Table Features • Delta Kernel • Delta UniForm • Conclusion

نصب دلتا لیک

Delta Lake Docker Image • Delta Lake for Python • PySpark Shell • JupyterLab Notebook • Scala Shell • Delta Rust API • ROAPI • Native Delta Lake Libraries • Multiple Bindings Available • Installing the Delta Lake Python Package • Apache Spark with Delta Lake • Setting Up Delta Lake with Apache Spark • Prerequisite: Set Up Java • Setting Up an Interactive Shell • PySpark Declarative API • Databricks Community Edition • Create a Cluster with Databricks Runtime • Importing Notebooks • Attaching Notebooks • Conclusion

عملیات ضروری دلتا لیک

Create • Creating a Delta Lake Table • Loading Data into a Delta Lake Table • The Transaction Log • Read • Querying Data from a Delta Lake Table • Reading with Time Travel • Update • Delete • Deleting Data from a Delta Lake Table • Overwriting Data in a Delta Lake Table • Merge • Other Useful Actions • Parquet Conversions • Delta Lake Metadata and History • Conclusion

غوطه‌وری در اکوسیستم دلتا لیک

Connectors • Apache Flink • Flink DataStream Connector • Installing the Connector • DeltaSource API • DeltaSink API • End-to-End Example • Kafka Delta Ingest • Install Rust • Build the Project • Run the Ingestion Flow • Trino • Getting Started • Configuring and Using the Trino Connector • Using Show Catalogs • Creating a Schema • Show Schemas • Working with Tables • Table Operations • Conclusion

نگهداری دلتا لیک

Using Delta Lake Table Properties • Delta Lake Table Properties Reference • Create an Empty Table with Properties • Populate the Table • Evolve the Table Schema • Add or Modify Table Properties • Remove Table Properties • Delta Lake Table Optimization • The Problem with Big Tables and Small Files • Using OPTIMIZE to Fix the Small File Problem • Table Tuning and Management • Partitioning Your Tables • Defining Partitions on Table Creation • Migrating from a Nonpartitioned to a Partitioned Table • Repairing, Restoring, and Replacing Table Data • Recovering and Replacing Tables • Deleting Data and Removing Partitions • The Life Cycle of a Delta Lake Table • Restoring Your Table • Cleaning Up • Conclusion

ساخت برنامه‌های بومی با دلتا لیک

Getting Started • Python • Rust • Building a Lambda • What’s Next

جریان‌سازی داده به درون و بیرون دلتا لیک

Streaming and Delta Lake • Streaming Versus Batch Processing • Delta as Source • Delta as Sink • Delta Streaming Options • Limit the Input Rate • Ignore Updates or Deletes • Initial Processing Position • Initial Snapshot with withEventTimeOrder • Advanced Usage with Apache Spark • Idempotent Stream Writes • Delta Lake Performance Metrics • Auto Loader and Delta Live Tables • Auto Loader • Delta Live Tables • Change Data Feed • Using Change Data Feed • Schema • Conclusion

ویژگی‌های پیشرفته

Generated Columns, Keys, and IDs • Comments and Constraints • Comments • Delta Table Constraints • Deletion Vectors • Merge-on-Read • Stepping Through Deletion Vectors • Conclusion

معماری دریاچه‌واره

The Lakehouse Architecture • What Is a Lakehouse? • Learning from Data Warehouses • Learning from Data Lakes • The Dual-Tier Data Architecture • Lakehouse Architecture • Foundations with Delta Lake • Open Source on Open Standards in an Open Ecosystem • Transaction Support • Schema Enforcement and Governance • The Medallion Architecture • Exploring the Bronze Layer • Exploring the Silver Layer • Exploring the Gold Layer • Streaming Medallion Architecture • Conclusion

تنظیم عملکرد: بهینه‌سازی خطوط لوله داده با دلتا لیک

Performance Objectives • Maximizing Read Performance • Maximizing Write Performance • Performance Considerations • Partitioning • Table Utilities • Table Statistics • Cluster By • Bloom Filter Index • Conclusion

الگوهای طراحی موفق

Slashing Compute Costs • High-Speed Solutions • Smart Device Integration • Efficient Streaming Ingestion • Streaming Ingestion • The Inception of Delta Rust • The Evolution of Ingestion • Coordinating Complex Systems • Combining Operational Data Stores at DoorDash • Change Data Capture • Delta and Flink in Harmony • Conclusion

مبانی حاکمیت و امنیت دریاچه‌واره

Lakehouse Governance • The Emergence of Data Governance • Data Products and Their Relationship to Data Assets • Data Products in the Lakehouse • Maintaining High Trust • Data Assets and Access • The Data Asset Model • Unifying Governance Between Data Warehouses and Lakes • Permissions Management • Filesystem Permissions • Cloud Object Store Access Controls • Identity and Access Management • Data Security • Fine-Grained Access Controls for the Lakehouse • Conclusion

مدیریت متادیتا، جریان داده و تباری

Metadata Management • What Is Metadata Management? • Data Catalogs • Data Reliability, Stewards, and Permissions Management • Why the Metastore Matters • Unity Catalog • Data Flow and Lineage • Data Lineage • Data Sharing • Automating Data Life Cycles • Audit Logging • Monitoring and Alerting • What Is Data Discovery? • Conclusion

اشتراک‌گذاری داده با پروتکل دلتا شیرینگ

The Basics of Delta Sharing • Data Providers • Data Recipients • Delta Sharing Server • Using the REST APIs • Anatomy of the REST URI • List Shares • Get Share • List Schemas in Share • List All Tables in Share • Delta Sharing Clients • Delta Sharing with Apache Spark • Stream Processing with Delta Shares • Delta Sharing Community Connectors • Conclusion