1430177 - Re-activate BLP data query to support the ADI forecasting dashboard

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[John Jensen]

The data powering this Desktop ADI historical/forecast dashboard:

https://metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/adiDetails.html

... stopped in July 2016.

Could it be reactivated please?

Comment 1

[Mark Reid [:mreid]]

See also Bug 1304084 as a potential way to simplify access to ADI data.

Comment 2

[brendan c]

I believe the problem here is that I have an R script running on hala1.metrics.scl3.mozilla.com that queries the Vertica BLP table and processes that data, and which used to rsync a couple of small csv files over to dashboard1.metrics.scl3.mozilla.com (i'm pretty sure those are the right servers, it's been a couple years...). I believe that the script still runs on its cron job, but the rsync between servers did some sketchy stuff, and IT disabled it.

One solution would be to figure out a way to move the files from "hala1" to "dashboard1" every day. I did not pursue this with IT previously because I didn't know if anyone was using the dashboard, and also because it seemed like there might be some security wrinkles around sending files between servers that I didn't really understand.

A less good solution would be to dump the output of the query directly to dashboard1 as a csv -- hala1 was needed because it can run R, but the analysis is pretty light and if I had to I could re-implement it in client-side js.

Comment 3

[Thomas Huelbert]

Will meet with IT to figure out a solution in the next week or so.

Comment 4

[Thomas Huelbert]

quick update - we need access to the blp vertica data (https://bugzilla.mozilla.org/show_bug.cgi?id=1304084). As we need IT, and they are currently working on migrating the current infrastructure from SCL3 data center to AWS, we are reaching out to understand timeline for this.

Comment 5

[Mark Reid [:mreid]]

I propose that we do the following:

- Migrate the dashboard from dashboard1 over to metrics.services.mozilla.com (is the frontend code for this available in github somewhere?)
- Continue to run the analysis script on hala1
- Upload the resulting CSV data to S3 using a similar "pass through" to metrics.s.m.c as used by a few of the other dashboards on that host.

Comment 6

[Jason Thomas [:jason]]

(In reply to Mark Reid [:mreid] from comment #5)
> I propose that we do the following:
> 
> - Migrate the dashboard from dashboard1 over to metrics.services.mozilla.com
> (is the frontend code for this available in github somewhere?)

I think we should use https://strategy-and-insights.mozilla.com/ since we've been hosting these sort of dashboards for the S&I team here.

Comment 7

[Mark Reid [:mreid]]

The strategy-and-insights server sounds good to me.

A few of us met today and discussed the following:
- Saptarshi verified that the analysis job is still running fine on hala
- Brendan will publish the frontend code to a private github repo and give Jason access
- Saptarshi will add a cron job to upload the csv output from hala to S3 at the following location:
   s3://net-mozaws-data-us-west-2-data-sai/dailyUsageSignals/data/
- Jason will give Saptarshi's AWS account access to write data to the above bucket, and deploy the code and data to the S&I server.

That should restore the functioning ADI dashboard, though at a new URL.

Comment 8

[brendan c]

holy moly, i did make a repo for this... back in sept 2014... which i'd pushed to once as of this morning, and which was not up to date. but i pushed up all the more recent changes (erm, the most recent of which were from like nov 2014). So I guess i haven't worked on this thing for even longer than i'd thought... at least 3 years (!!!)

https://github.com/mozilla/firefox-adi-details-dashboard

i added a bunch of teams, lmk if i need to add any more.

also, let's not decommission the one at https://metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/adiDetails.html right away -- i'd like to double check that i've committed all the correct versions of all the files. I'm 95% sure that it's right (diffed a couple of the key files from the server and got matches) but i had some uncommitted changes, so there's an outside chance that i was working on some changes when i left off, and the there are some things in the changes i just committed that should not actually be in the "production" version in the master branch.

Comment 9

[Jason Thomas [:jason]]

I've created a new access key for Saptarshi and he was able to publish the data to the s3 bucket.

https://github.com/mozilla-services/cloudops-deployment/pull/1565
https://github.com/mozilla-services/cloudops-deployment/pull/1567

https://github.com/mozilla/firefox-adi-details-dashboard has been deployed to https://strategy-and-insights.mozilla.com/dailyUsageSignals/adiDetails.html. Changes to the repo will be auto deployed. 

Can someone please validate and make sure the dashboard is functioning as expected?

Comment 10

[Mark Reid [:mreid]]

Looks good!

Comment 11

[Jason Thomas [:jason]]

Note:

I've added a redirect from https://metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/adiDetails.html to https://strategy-and-insights.mozilla.com/dailyUsageSignals/adiDetails.html