1116511 - [MozID] Automate generation of JSON data sources using Tableau data

Status: RESOLVED FIXED

(www.mozilla.org :: Bedrock, defect)

Description

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

# Overview

This bug is created to automate the process of creating the data sources for Moz ID.

# History

Bug 1087053 - temporary solution was put into place to manually create the JSON.
Bug 1081284 - access to the REST API was granted.

# Specifications

Source:

The data sources will be out of Tableau in 3 tabs:
https://dataviz.mozilla.org/views/Active_Contributor_Dashboard/ByTeam#1
https://dataviz.mozilla.org/views/Active_Contributor_Dashboard/BySource#1
https://dataviz.mozilla.org/views/Active_Contributor_Dashboard/Total#1

Final format:

The final format of the data will be in JSON files; one for each group.

Example JSON format and files - https://github.com/mozilla/bedrock/tree/master/media/js/styleguide/logo-prototype/tab

You can see the WIP prototype here - https://www.mozilla.org/en-US/styleguide/identity/mozilla/logo-prototype/

# Request

For this bug, we will need to

1. Investigate and confirm usability of the REST API identified in bug 1081284.
2. Build functionality to pull and convert this data to JSON.
3. Create CRON job to generate these files once per week (Monday 12:00 AM PST).

Comment 1

[Paul [:pmac] McLanahan]

Now unblocked and proceeding with the Tableau API.

Comment 2

[Paul [:pmac] McLanahan]

Nearly ready to submit a PR for this. A couple of questions first.

1. Currently you're getting the data for "sumo" and "reps" from the "Team" tab instead of "Source". I take that to mean that it's the numbers for Sumo and Reps contributions regardless of source, but it's potentially missing contributions to other teams (Firefox e.g.) via the "reps" or "sumo" source. Do you know which is more appropriate?

2. The data we're getting goes back to 1998, but in the current files we only go back to 2013. How much do you want included in the final data?

Comment 3

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

(In reply to Paul McLanahan [:pmac] from comment #2)
> 1. Currently you're getting the data for "sumo" and "reps" from the "Team"
> tab instead of "Source". I take that to mean that it's the numbers for Sumo
> and Reps contributions regardless of source, but it's potentially missing
> contributions to other teams (Firefox e.g.) via the "reps" or "sumo" source.
> Do you know which is more appropriate?
From what I can see, these numbers look to be the same in Tableau. It appears to be simply different views for the same data. Let's keep it as is.

> 2. The data we're getting goes back to 1998, but in the current files we
> only go back to 2013. How much do you want included in the final data?
It should go back 12 months from the time the data is queried.

Comment 4

[Sheeri Cabral [:sheeri]]

Currently the only sources for sumo and reps are their dbs, respectively. That could change in the future, so I'd recommend basing your data gathering on what you want - do you want it by source, or by team? (or both?)

Comment 5

[Paul [:pmac] McLanahan]

Attachment: Link to Github pull-request: https://github.com/mozilla/bedrock/pull/2717

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/b0eced3174a92d003b31730e210f50e58ea162ac
Bug 1116511: Add script to sync data from Tableau.

https://github.com/mozilla/bedrock/commit/acca7041ddf3d0085385b9ce54ebcbdd9643e520
Bug 1116511: Add view for serving JSON contributor data.

https://github.com/mozilla/bedrock/commit/b807b1a05dc21819b0671aad0b701dcdab962581
Bug 1116511: Add cron jobs for stage and prod tableau data.

https://github.com/mozilla/bedrock/commit/1ecd039a13614261ff5c88d7d6c34255c717875e
Merge pull request #2717 from pmclanahan/automate-mozid-data-tableau-1116511

Bug 1116511: Add tableau data sync and JSON view for MozID.

Comment 7

[Paul [:pmac] McLanahan]

This is done and working in production:

https://www.mozilla.org/contributor-data/all.json

Comment 8

[Jennifer Bertsch [:jbertsch]]

Whoo hoo!