1362436 - A function that automatically fetches the latest version

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P3)

Description

[bugzilla]

In the interest of doing less hardcoding of paths, a "get_dataset" function that will return the latest version would be super useful

Comment 1

[Frank Bertsch [:frank]]

The metastore is where this lives. If you need to always have the latest version, instead of reading

> sqlContext.read.load("s3://path/to/dataset/v1", "parquet")

just do:

> sqlContext.sql("SELECT * FROM dataset")

Raw data should always be pointed to by sources.json, so the Dataset API will return the correct dataset (which we saw when we updated the infrastructure).

Is there another case we're missing?

Comment 2

[bugzilla]

Hmm, I think at the moment until we have the p2h operator working there's still that theoretical lag until the cron runs and loads any new partitions.

And, exactly for this kind of reason, we should abstract the mechanics of "get me the latest dataset"  away from the semantics, so we can start writing "get_dataset('dataset')" now and not have to change every script when the p2h operator is available and in use for all our datasets (or if we make any other changes to the infrastructure in the future.)