Closed Bug 1384888 Opened 9 years ago Closed 5 years ago

Centralize tracking of datasets and data lineage/provenance

Categories

(Data Platform and Tools :: General, enhancement, P3)

Product:
Data Platform and Tools
Component:
General
Type:
enhancement
Points:
3

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: amiyaguchi, Unassigned)

References

Details

There are dozens of datasets and tables scattered across the data pipeline. A large number of these are scheduled via airflow. If every dataset included a small amount of metadata, we could track the flow of data from data lake to query. For example, an `ETL_ID` (current, parent) could aid in the construction of a graph to diagnose particular datasets. Some reading: https://en.wikipedia.org/wiki/Data_lineage https://www.timmitchell.net/post/2016/05/06/etl-data-lineage/ http://www.pachyderm.io/
This is related to the "data dictionary" efforts in Bug 1392726 (and others).
Points: --- → 3
Depends on: 1392726
Priority: -- → P3
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.