1152462 - Server-side versioning of pipeline data

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Mark Reid [:mreid]]

We need a mechanism for handling server-side data changes, including format/layout changes, locations, backprocessing, and migrating data.

I propose that we allocate a "meta" prefix within our production S3 bucket, and store mappings of the current location of each logical data source.

In other words, we have a "telemetry" data source, which could map to a bucket prefix of "/telemetry/v1", then if we need to modify the storage layout or data format, we could update to "/telemetry/v2".

Consumers of the data could consult the metadata to determine where the correct data is located (ideally this would be built in to client applications).

We could attach other metadata to data sources using this approach as well, such as the location of data specification documentation, the current storage layout, and so on.

Comment 1

[Wesley Dawson [:whd]]

https://github.com/mozilla-services/svcops/commit/3c3856b4ebbd8cd3cf7fa809775bf6baadb720f6
https://github.com/mozilla-services/svcops/commit/ff4fd5cc214c62888622e7268101c08c5651d65b

The production metadata bucket has been set up: s3://net-mozaws-prod-us-west-2-pipeline-metadata. It has RO access from new dev, and RW access via the production S3 credentials for the pipeline team. :mreid will populate the bucket with the requisite metadata.

We decided to use a separate bucket instead of an object prefix within the current bucket for flexibility.

Comment 2

[Mark Reid [:mreid]]

The meta bucket is there, I will write up some docs asap.

Comment 3

[Mark Reid [:mreid]]

I've added some documentation here:
https://wiki.mozilla.org/CloudServices/DataPipeline/Metadata