Simplify transfer of Baloo data from Tableau to Dashboard

RESOLVED FIXED

Status

Mozilla Foundation
Metrics
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: adamlofting, Assigned: adamlofting)

Tracking

Details

(Whiteboard: [metrics][Aug22][Baloo][Q3][data: integration])

(Assignee)

Description

3 years ago
Baloo data is now available via a behind LDAP Tableau table. This includes an option to export the aggregate numbers as a CSV file.

To make the best use of this, we should modify the interim workflow which consisted of:

Google Docs Sheet:
http://doctodash.herokuapp.com/

Middleware Node App:
http://doctodash.herokuapp.com/
https://github.com/adamlofting/doctodash

The ideal workflow is a single paste of the CSV data from Tableau. We should avoid the need to manually copy individual cells.

We could reformat the Google Doc, or we can modify the d3 code on the dashboard to use the CSV file directly, and include this in the repo.
(Assignee)

Comment 1

3 years ago
1) I've added a new worksheet to the google doc:

https://docs.google.com/a/mozillafoundation.org/spreadsheets/d/1NdHzBIDKduAu6-vvQx6iy-93zCzAqUKk4x_eanDxv1A/edit#gid=1758040737

This will make it easier to copy and paste from the Tableau export view.

2) But it will be even easier if we can make the three Tableau export sheets consistent in their export layout (which varies from the reporting view). 

Sheeri: is this feasible?
https://docs.google.com/presentation/d/1YhNUx7q7etf-ytm7jJZjZdgEGbGZwFLklL-ht9oyIBs/edit?usp=sharing

3) I've also added new code to the node app that transforms this for the dashboard. This is reading from the new and old sheet currently. Here's an example of the app transforming the data from the new worksheet: http://doctodash.herokuapp.com/tab/sumo

4) When we have all the data in that new worksheet we can switch the front-end at areweamillionyet.org to use the new end-point and then clean out the old version of the google doc content.
Flags: needinfo?(scabral)
Flags: needinfo?(pierros)
2) It's sort of feasible. You can't change the order of columns in the "view data" part, but the column "Measure Values" was really just "Sum(Number of Records)", so the only change I could do was make nums_team match nums_source (now, both have is_new before # of records).

I've also fixed the nums_total to match the other 2. How's it look?

I think that's all I needed to do...let me know if it's not.

I've fixed it so the data on the graphs and data extracts start at 12/1/2013 and go forward until whatever date is latest. Can you confirm that you can see 8/11/2014 data on the graphs and in the data parts?
Flags: needinfo?(scabral)
(Assignee)

Comment 3

3 years ago
(In reply to Sheeri Cabral [:sheeri] from comment #2)
> 2) It's sort of feasible. You can't change the order of columns in the "view
> data" part, but the column "Measure Values" was really just "Sum(Number of
> Records)", so the only change I could do was make nums_team match
> nums_source (now, both have is_new before # of records).
> 
> I've also fixed the nums_total to match the other 2. How's it look?
> 
> I think that's all I needed to do...let me know if it's not.

That works great. Less than 2 mins to get all the data transferred. The only manual step was adding a name 'all' into the view column for the totals data, but that was only 10 seconds work. 

> I've fixed it so the data on the graphs and data extracts start at 12/1/2013
> and go forward until whatever date is latest. Can you confirm that you can
> see 8/11/2014 data on the graphs and in the data parts?

I'm seeing 8/1/2014 in the nums_total, but not in the team or source views currently.
OK, try now. Should have from 12/1 to 8/11 on *all* the 6 tabs. (and thanks for looking!)
(Assignee)

Comment 5

3 years ago
(In reply to Sheeri Cabral [:sheeri] from comment #4)
> OK, try now. Should have from 12/1 to 8/11 on *all* the 6 tabs. (and thanks
> for looking!)

That did it :)
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Comment 6

3 years ago
(In reply to Adam Lofting (:adamlofting) from comment #1)

> 4) When we have all the data in that new worksheet we can switch the
> front-end at areweamillionyet.org to use the new end-point and then clean
> out the old version of the google doc content.

I forgot Step 4, in my haste to close the bug. I'll implement this ASAP.
Status: RESOLVED → REOPENED
Flags: needinfo?(adam)
Resolution: FIXED → ---
(Assignee)

Comment 7

3 years ago
http://areweamillionyet.org is now using the data from the new worksheet.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Flags: needinfo?(adam)
Resolution: --- → FIXED
Whiteboard: [metrics][Aug22][Baloo][Q3] → [metrics][Aug22][Baloo][Q3][integration]
Whiteboard: [metrics][Aug22][Baloo][Q3][integration] → [metrics][Aug22][Baloo][Q3][data: integration]
(Assignee)

Updated

3 years ago
Flags: needinfo?(pierros)
You need to log in before you can comment on or make changes to this bug.