Data Lineage Tracking: Why Coverage Stops at the System Boundary

By: Lakshay Nasa

09.24.26

Quick definition: What is data lineage tracking?

Data lineage tracking is the continuous capture of how data actually moves through your systems, taken from those systems as pipelines run. Data lineage is the path data takes from data sources through transformations to consumption. Lineage tracking is the practice that keeps a record of that path accurate as schemas, queries, and pipelines change.

An engineer proposes a schema change and cannot say what breaks. A number in a board deck looks wrong and nobody can trace where it came from. A migration stalls because no one can prove which tables are safe to retire.

In those moments, teams are rarely missing lineage altogether. They have what the warehouse exposes, what dbt declares, and what the BI tool infers. But what they lack is a record that reaches across all of it. The important question isn’t whether you have data lineage (most do)… It’s how far yours reaches, and whether you know where it stops.

Documented lineage vs. tracked lineage

In our view, the distinction that matters most is between documented lineage and tracked lineage.

Documented lineage describes what a pipeline was supposed to do. Someone captured it once, by hand, in a diagram, a wiki page, or a spreadsheet that still lives in a shared drive. It was accurate on the day it was written.
Tracked lineage records what the pipeline actually did. It comes from the systems themselves, and it changes when they change.

The difference matters most under pressure: A stale map is worse than no map at all, because people trust it. An engineer who checks a quarterly spreadsheet before shipping a change feels informed, and ships anyway.

(For the full case against hand-maintained maps, see why data lineage mapping fails at the moment you need it.)

Three things lineage tracking has to get right

These are what separate lineage tracking that survives production from lineage tracking that only demos well.

1. It has to connect across systems

Real pipelines cross platform boundaries constantly. Events land in Kafka, get processed in Spark, load into Snowflake, get transformed by dbt, and surface in Looker. Five systems, four handoffs, one number on a dashboard.

Lineage that stops at each boundary produces disconnected fragments. Worse, a single missing link in the middle of a chain hides everything downstream of it. The graph still returns an answer. It just stops before the part you needed.

Say the number on that dashboard looks wrong. Warehouse lineage tells you which tables feed the view, which is where most investigations stop, because the transformation that introduced the error was a dbt model and the aggregation that compounded it was defined in the BI layer. Neither appears in the warehouse graph. The investigation restarts in each tool, and the engineer reassembles the chain from memory.

2. It has to be continuous

Data estates change daily. New pipelines come online, columns get deprecated, models get refactored. Lineage captured as a project is accurate on the day it is captured and progressively wrong after that.

Tracking means capture that runs with the pipelines rather than alongside them. Nobody schedules a refresh, because nothing needs refreshing.

3. It has to resolve to columns

Table-level lineage tells you that Table A feeds Table B. That does not answer the question people actually ask, which is what breaks if this field changes.

A table might feed 40 downstream assets while the column you are changing feeds two. Table-level lineage makes every change look catastrophic, so every impact assessment becomes a manual exercise in ruling things out. Column-level lineage turns impact analysis from an estimate into an answer.

The problem with platform-native lineage tracking

Platform-native lineage has become genuinely good.

Snowflake‘s Horizon Catalog tracks how data flows from source to target objects, with column-level lineage where supported, task-level lineage, and a GET_LINEAGE function for querying relationships programmatically.
Databricks captures lineage in Unity Catalog automatically as queries run, and the graph inherits the platform’s permission model.

For work that happens inside those platforms, this is strong, current, and free of the stale-documentation problem. The pitch is architectural: Native tracking means fewer connectors to maintain, less metadata ingestion lag, and less reconciliation between what the lineage graph shows and what the platform is doing.

But read that pitch closely: Fewer connectors is a description of covering fewer systems.

However, every platform also offers a path to reach outside its own walls, and every one of those paths carries a maintenance cost:

Snowflake supports external lineage for objects outside the account
Databricks lets you register external assets in Unity Catalog and define their relationships by hand, which means someone creates each object, maps its columns, and maintains all of it as systems change
Native AWS lineage is scoped to AWS services, and reaching tools AWS does not operate depends on OpenLineage events you instrument yourself. The same boundary shows up in BigQuery and Azure.

But hand-maintained cross-platform lineage reintroduces exactly the drift that automated tracking was supposed to remove, and it reintroduces it at the boundary, which is the worst possible place to have a gap.

That boundary is where incidents happen. When a source schema change breaks an executive dashboard, the data engineers running the warehouse cannot see the dashboard, the team running the BI tool cannot see the source table, and the engineer who made the change had no way to know about either.

The gap shows up in the questions each scope can answer.

Question	Platform-scoped tracking	Cross-system tracking
What feeds this table inside the warehouse?	Answered	Answered
What breaks in the BI layer if I rename this column?	Not visible past the platform edge	Answered
Where did this dashboard number originate, three systems upstream?	Partial, stops at the first handoff	Answered
Which downstream assets inherit this PII classification?	Only assets inside the platform	Every asset on the path*
How is coverage outside the platform maintained?	Registered and maintained by hand	Captured automatically from each system

* Cross-platform column-level classification propagation is a DataHub Cloud capability

When tracking gaps start costing money

Gaps in coverage stay theoretical until something forces the question. In its March 2026 Business Value of DataHub Cloud study, IDC found that organizations using DataHub Cloud increased the share of datasets with mapped data lineage by 75% and resolved data-related outages 58% faster. Root cause analysis starts from a dependency path instead of a guess.

Three things usually drive the urgency:

1. Regulatory deadlines

Regulatory compliance treats lineage as evidence. An auditor asking where a regulated field originates, how it was transformed, and which reports it reaches wants a verifiable record, and that record needs to reflect production rather than the last review cycle.

Teams working from documented lineage answer that question by reconstructing it, which takes weeks and produces an artifact that starts decaying immediately. Teams working from tracked lineage generate the same artifact from live metadata, and it stays current between audits. The same graph also shows where a sensitive data classification such as PII needs to propagate, which is the practical form most lineage for compliance work takes.

2. Migration and platform consolidation

Migrations stall on the same question every time: Which tables can we retire, and what proves it?

DPG Media grew through acquisition, and every acquisition brought an inherited data landscape with its own operational systems and data technology. Exposing all of it in one structure is what made retirement decisions safe, and the company reduced annual data storage costs by more than 50%.

Due to mergers and acquisitions, we have a tremendous landscape of different operational systems, different data technology, and it’s key to have one single point of view.
Sven van EgmondHead of Data, DPG Media

Netflix reports a related outcome at a different scale. System-wide upgrades and ownership transfers that once required tracking across multiple spreadsheets now run with visibility across every affected entity, because its catalog models data, ML, and software assets in the same graph.

3. AI readiness

An agent answering questions over enterprise data is only as trustworthy as the path back to its sources. Lineage is what supplies that path, and IDC’s study found a 29% improvement in training datasets with complete lineage alongside 119% more AI and ML models reaching production.

Block runs more than 50 data platforms under financial services regulation. Its engineering team connected its open-source agent, Goose, to the DataHub Model Context Protocol (MCP) server, which gives agents governed access to schema, ownership, and lineage traversal. An engineer investigating a suspect table can now confirm the asset, trace its upstream dependencies, and identify owners in one conversation.

Something that might have taken hours, or days, or even weeks turns into just a few simple, short conversation messages.
Sam OsbornSenior Software Engineer, Block

No tracking covers everything

Tracked lineage is captured from the queries your systems are already running. A parser reads those queries at ingestion and extracts the dependencies, which is why the graph reflects what the data is doing rather than what someone wrote down.

DataHub’s parser does this with 97-99% accuracy in published benchmarks, and the mechanics of extracting column-level lineage from SQL are worth a read if you want the detail. Accuracy and coverage are different measurements, though, and the second is where teams get caught out. A parser can be highly accurate on the transformations it understands and still miss a category of them completely. Certain functions, nested data structures, and scripted multi-step transformations are hard for any parser to resolve, and the honest vendors publish which ones.

DataHub lists its parser’s limits in the docs, and every result carries a confidence score so a weak edge can be flagged rather than trusted quietly.

So, when you evaluate lineage tracking, ask where the gaps are. A specific answer is evidence that someone has measured coverage.

How DataHub tracks lineage

DataHub treats lineage as a property of the metadata platform rather than a documentation workflow running on top of it.

Automatic extraction: Table- and column-level lineage is detected from BigQuery, Snowflake, dbt, Looker, PowerBI, and 20+ other data tools, without manual annotation.
SQL parsing where native lineage is thin: The parser fills in for legacy systems, bespoke pipelines, and cross-system transformations that no individual tool reports.
Cross-platform stitching: Upstream and downstream relationships resolve into one graph, including the task relationships inside your data pipelines, so you can expand lineage across a boundary the same way you expand it within a system.
The workflows the graph exists to serve: Impact analysis surfaces every dependent dashboard, model, and consumer before a change ships. Metadata propagation carries tags, descriptions, and classifications downstream to every data asset on the path, so they stay attached to the data as it moves.

My favorite part about DataHub is the lineage because this is one really easy way of connecting the producers to the consumers. Now the producers know who is using their data. Consumers know where the data is coming from. And it is easier to have accountability mechanisms.
Sherin ThomasSoftware Engineer, Chime

Judge lineage tracking by reach. It needs to extend to the places where your changes break things, and you need to be able to name the places it misses. To see how far the graph reaches on a stack like yours, take the interactive lineage tour.

FAQs

Data lineage answers three questions data teams hit constantly: where did this number come from, what breaks if I change it, and can we prove it to an auditor. The practical benefits of data lineage follow from those answers. Engineers assess impact before shipping, analysts trace data errors back to the transformation that introduced them, and data scientists find the upstream cause when a model degrades. For the fuller list, see the benefits of data lineage.

Data lineage is important because it turns dependency questions into answers that can be checked. Without it, impact assessment is guesswork and audits become reconstruction projects. With it, data governance policies can be applied and verified against how data actually moves. Lineage is sometimes confused with data provenance, which records origin, ownership, and custody rather than movement and transformation.

It is enough for the work that happens inside the warehouse. Warehouse-native lineage captures automatically, stays current, and usually respects existing access controls. It stops at the platform edge. If your transformations run in dbt, your orchestration runs in Airflow, and your dashboards live in a BI tool, native lineage cannot connect those hops, which is where most production incidents originate.

Accuracy depends on the capture method. SQL parsing is the most precise approach in common use, and DataHub‘s parser generates lineage with 97-99% accuracy in published benchmarks. Pattern-based approaches that infer relationships from column-name similarity are less reliable. Ask any vendor for both numbers: parsing accuracy, and the list of SQL constructs the parser does not support.

Careful here, because the useful relationship runs the other way. DataHub captures lineage deterministically by parsing the SQL that ran, not by inferring relationships with a model, and that is the point. Inferred lineage is a guess presented as a dependency. Where AI changes the picture is in consumption: agents connected through an MCP server can traverse the lineage graph conversationally, so impact analysis becomes a question rather than an investigation.

Agents answering questions over enterprise data need a traceable path from any answer back to a verified source. Lineage supplies that path, and it helps you trace an upstream data quality problem when a model degrades. IDC’s March 2026 study of DataHub Cloud found a 29% improvement in training datasets with complete lineage and 119% more AI and ML models reaching production.

Yes, provided you can reach the queries. Legacy databases, bespoke transformation scripts, and internal tools often run SQL without reporting any dependency information. If a query log is available, DataHub‘s SQL queries connector builds column-level lineage and usage statistics from the log itself. For systems with no query log, lineage can be declared through the API, though anything declared by hand carries the same maintenance cost as documentation.

Coverage decides most of it. Ask which of your platforms are captured automatically, whether lineage resolves to columns or stops at tables, whether the graph updates from production metadata or on a schedule, and whether the vendor will tell you where its capture fails. A tool that covers three of your five systems perfectly leaves you reconstructing the other two by hand. For a fuller breakdown, see where data lineage tools sit in your stack.