کتاب اصول مشاهده‌پذیری داده‌ها

معرفی مشاهده‌پذیری داده‌ها

Scaling Data Teams • Challenges of Scaling Data Teams • Segregated Roles and Responsibilities and Organizational Complexity • Anatomy of Data Issues and Consequences • Impact of Data Issues on Data Team Dynamics • Scaling AI Roadblocks • Challenges with Current Data Management Practices • Effects of Data Governance at Scale • Data Observability to the Rescue • The Areas of Observability • How Data Teams Can Leverage Data Observability Now • Low Latency Data Issues Detection • Efficient Data Issues Troubleshooting • Preventing Data Issues • Decentralized Data Quality Management • Complementing Existing Data Governance Capabilities • The Future and Beyond • Conclusion

اجزای مشاهده‌پذیری داده‌ها

Channels of Data Observability Information • Logs • Traces • Metrics • Observations Model • Physical Space • Server • User • Static Space • Dynamic Space • Expectations • Rules • Automatic Anomaly Detection • Prevent Garbage In, Garbage Out • Conclusion

نقش‌های مشاهده‌پذیری داده‌ها در یک سازمان داده

Data Architecture • Where Does Data Observability Fit in a Data Architecture? • Data Architecture with Data Observability • How Data Observability Helps with Data Engineering Undercurrents • Security • Data Management • Support for Data Mesh’s Data as Products • Conclusion

تولید مشاهدات داده

At the Source • Generating Data Observations at the Source • Low-Level API in Python • Description of the Data Pipeline • Definition of the Status of the Data Pipeline • Data Observations for the Data Pipeline • Generate Contextual Data Observations • Generate Data-Related Observations • Generate Lineage-Related Data Observations • Wrap-Up: The Data-Observable Data Pipeline • Using Data Observations to Address Failures of the Data Pipeline • Conclusion

خودکارسازی تولید مشاهدات داده

Abstraction Strategies • Event Listeners • Aspect-Oriented Programming • High-Level Applications • No-Code Applications • Low-Code Applications • Differences Among Monitoring Alternatives • Conclusion

پیاده‌سازی انتظارات

Introducing Expectations • Shift-Left Data Quality • Corner Cases Discovery • Lifting Service Level Indicators • Using Data Profilers • Maintaining Expectations • Overarching Practices • Fail Fast and Fail Safe • Simplify Tests and Extend CI/CD • Conclusion

ادغام مشاهده‌پذیری داده‌ها در پشته داده شما

Ingestion Stage • Ingestion Stage Data Observability Recipes • Airbyte Agent • Transformation • Transformation Stage Data Observability Recipes • Apache Spark • dbt Agent • Serving • Recipes • BigQuery in Python • Orchestrated SQL with Airflow • Analytics • Machine Learning Recipes • Business Intelligence Recipes • Conclusion

شفاف‌سازی سیستم‌های کدر

Data Translucence • Opaque Systems • SaaS • Don’t Touch It; It (Kinda) Works • Inherited Systems • Strategies for Data Translucence • Strategies • The Data Observability Connector • Example: Building a dbt Data Observability Connector (SaaS) • Conclusion