Delta Lake: Up and Running

With the rapid growth of big data and AI, organizations are quickly building data products and solutions in an ad-hoc manner. But as these data organizations mature, it’s apparent that their analysis and machine learning models are only as reliable as the data they’re built upon. The solution? Delta Lake, an open-source format that enables building a lakehouse architecture on top of existing storage systems such as S3, ADLS, and GCS.

In this practical book, author Bennie Haelen shows data engineers, data scientists, and data analysts how to get Delta Lake and its unique features up and running. The ultimate goal of building data pipelines and applications is to query processed data and gain insights from it. You’ll learn how the choice of storage solution determines the robustness and performance of the data pipeline, from raw data to insights. With this book, you will:

Use modern data management and data engineering techniques
Understand how ACID transactions bring reliability to data lakes at scale
Run streaming and batch jobs against your data lake concurrently
Execute update, delete, and merge commands against your data lake
Use time travel to roll back and examine previous versions of your data
Build a streaming data quality pipeline following the medallion architecture

Author Biography

Bennie is a principal architect with Insight Digital Innovation-a Microsoft and Databricks partner. As Principal architect with Insight, Bennie’s primary focus areas are Modern Data Warehousing, Machine learning, AI, and IoT on various commercial cloud platforms. Bennie has overseen many Data + AI projects in different application domains such as health care, the public sector, oil & gas, and financial applications. Bennie has architected and delivered real time streaming Data Lakehouse applications with Databricks, Spark Structured Streaming, Delta Lake, and Microsoft Power BI for various application domains. Bennie brings a wealth of practical experience in implementing secure, enterprise-scale Data Lakehouse-based solutions to support business intelligence, data science and machine learning. Bennie has also been a frequent speaker at Databricks events at Microsoft Technology Centers around the country, and was a speaker at the Data+AI 2021 summit. Dan Davis is a Cloud Data Architect with a decade of experience delivering analytic insights and business value from data. Using modern tools and technologies, Dan specializes in designing and delivering data platforms, frameworks, and process’ to support data integration and analytics for on-premises, hybrid, and cloud environments on an enterprise scale.