Databricks AI Summit 2023 Definitive Healthcare Session

Definitive Healthcare, an AI health company, has faced several challenges in managing and optimizing large-scale data. They shared their experiences and the solutions they found in a recent presentation.

Problems Faced:

Definitive Healthcare encountered a range of complex issues in its data management:

1. The path toward data optimization was complicated and nuanced despite a clear goal.
2. They had hundreds of different assets, each requiring additional ETL processing and moving through various zones from landing to curating.
3. Data governance and compliance need under HIPAA/GDPR necessitated data silos.
4. There were knowledge gaps among end users, particularly concerning query optimization.
5. Humans and machines were constantly querying a significant volume of data.

Data and Stakeholders:

Definitive Healthcare's data ecosystem was vast, with tables ranging from 111 records for Patient Transactions to 15M Expert Profiles for Scientific Affairs Data. The company had various stakeholders, including product teams, data science teams, custom predictive analytics teams, and data analysts. This meant managing a wide variety of needs, from ETL ingestion to custom predictive modeling.

Databricks Lakehouse as a Solution:

Databricks' Lakehouse proved to be the right solution for Definitive Healthcare, meeting all the key priorities:

1. Capability to handle robust data pipelines for large-scale daily ETL ingestions.
2. Support for complete database restatements.
3. Provision of an enterprise data warehouse to manage the hundreds of inbound data sources and 24/7 data aggregation.