1053149 - Simplify transfer of Baloo data from Tableau to Dashboard

Status: RESOLVED FIXED

(Mozilla Foundation :: Metrics, task)

Description

[Adam Lofting (:adamlofting)]

Baloo data is now available via a behind LDAP Tableau table. This includes an option to export the aggregate numbers as a CSV file.

To make the best use of this, we should modify the interim workflow which consisted of:

Google Docs Sheet:
http://doctodash.herokuapp.com/

Middleware Node App:
http://doctodash.herokuapp.com/
https://github.com/adamlofting/doctodash

The ideal workflow is a single paste of the CSV data from Tableau. We should avoid the need to manually copy individual cells.

We could reformat the Google Doc, or we can modify the d3 code on the dashboard to use the CSV file directly, and include this in the repo.

Comment 1

[Adam Lofting (:adamlofting)]

1) I've added a new worksheet to the google doc:

https://docs.google.com/a/mozillafoundation.org/spreadsheets/d/1NdHzBIDKduAu6-vvQx6iy-93zCzAqUKk4x_eanDxv1A/edit#gid=1758040737

This will make it easier to copy and paste from the Tableau export view.

2) But it will be even easier if we can make the three Tableau export sheets consistent in their export layout (which varies from the reporting view). 

Sheeri: is this feasible?
https://docs.google.com/presentation/d/1YhNUx7q7etf-ytm7jJZjZdgEGbGZwFLklL-ht9oyIBs/edit?usp=sharing

3) I've also added new code to the node app that transforms this for the dashboard. This is reading from the new and old sheet currently. Here's an example of the app transforming the data from the new worksheet: http://doctodash.herokuapp.com/tab/sumo

4) When we have all the data in that new worksheet we can switch the front-end at areweamillionyet.org to use the new end-point and then clean out the old version of the google doc content.

Comment 2

[Sheeri Cabral [:sheeri]]

2) It's sort of feasible. You can't change the order of columns in the "view data" part, but the column "Measure Values" was really just "Sum(Number of Records)", so the only change I could do was make nums_team match nums_source (now, both have is_new before # of records).

I've also fixed the nums_total to match the other 2. How's it look?

I think that's all I needed to do...let me know if it's not.

I've fixed it so the data on the graphs and data extracts start at 12/1/2013 and go forward until whatever date is latest. Can you confirm that you can see 8/11/2014 data on the graphs and in the data parts?

Comment 3

[Adam Lofting (:adamlofting)]

(In reply to Sheeri Cabral [:sheeri] from comment #2)
> 2) It's sort of feasible. You can't change the order of columns in the "view
> data" part, but the column "Measure Values" was really just "Sum(Number of
> Records)", so the only change I could do was make nums_team match
> nums_source (now, both have is_new before # of records).
> 
> I've also fixed the nums_total to match the other 2. How's it look?
> 
> I think that's all I needed to do...let me know if it's not.

That works great. Less than 2 mins to get all the data transferred. The only manual step was adding a name 'all' into the view column for the totals data, but that was only 10 seconds work. 

> I've fixed it so the data on the graphs and data extracts start at 12/1/2013
> and go forward until whatever date is latest. Can you confirm that you can
> see 8/11/2014 data on the graphs and in the data parts?

I'm seeing 8/1/2014 in the nums_total, but not in the team or source views currently.

Comment 4

[Sheeri Cabral [:sheeri]]

OK, try now. Should have from 12/1 to 8/11 on *all* the 6 tabs. (and thanks for looking!)

Comment 5

[Adam Lofting (:adamlofting)]

(In reply to Sheeri Cabral [:sheeri] from comment #4)
> OK, try now. Should have from 12/1 to 8/11 on *all* the 6 tabs. (and thanks
> for looking!)

That did it :)

Comment 6

[Adam Lofting (:adamlofting)]

(In reply to Adam Lofting (:adamlofting) from comment #1)

> 4) When we have all the data in that new worksheet we can switch the
> front-end at areweamillionyet.org to use the new end-point and then clean
> out the old version of the google doc content.

I forgot Step 4, in my haste to close the bug. I'll implement this ASAP.

Comment 7

[Adam Lofting (:adamlofting)]

http://areweamillionyet.org is now using the data from the new worksheet.