Firefox Dashboard -- Deseasonalized ADI numbers are not being pushed to the dashboard

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: aalmossawi, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
The numbers at the data-source on dashboard1 are being updated, but aren't being pushed to S3, it would appear.

Also filed on GitHub: https://github.com/mozilla/firefox-dashboard/issues/10

Comment 1

3 years ago
Heather's looking into this...

Comment 2

3 years ago
As far as I can see, vertica database is updated with up to yesterday's data, so it's not a vertica issue. Is there a script that runs a file to create the dashboard? If so, what is the name of the file?
(Reporter)

Comment 3

3 years ago
The issue is with the pushing to S3. Sheeri owned these scripts. Who owns them in her absence? mpressman?
Flags: needinfo?(hcrince)
which file is this? There are no errors in the logs for the cron that sends files to s3?
It looks like the push_files_to_s3.sh script on dashboard1.metrics.scl3 is pushing blocklistDecomposition.csv to s3 with the rest of the files - Here is the log output of the push for what has run so far today:

INFO: beginning copy at Mon Feb 29 11:00:01 UTC 2016
upload: ../data/www/metrics.mozilla.com/protected/cchoi/confidence_intervals.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/confidence_intervals.csv
upload: ../data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/blocklistDecomposition.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_weekly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_monthly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_weekly_data.csv to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_last_updated.json to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_last_updated.json
upload: ../home/hulmer/tmp/dashboards/main/mau_data.csv to s3://net-mozaws-prod-metrics-data/dashboards-landing-page/mau_data.csv
upload: ../home/hulmer/tmp/dashboards/mofo/learning-network-cities.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/learning-network-cities.json
upload: ../home/hulmer/tmp/dashboards/mofo/webmaker-maus.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/webmaker-maus.json
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_weekly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_monthly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_last_updated.json to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_last_updated.json
INFO: end copy at Mon Feb 29 11:00:10 UTC 2016
INFO: beginning copy at Mon Feb 29 18:03:01 UTC 2016
upload: ../data/www/metrics.mozilla.com/protected/cchoi/confidence_intervals.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/confidence_intervals.csv
upload: ../data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/blocklistDecomposition.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_weekly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_monthly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_weekly_data.csv to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_last_updated.json to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_last_updated.json
upload: ../home/hulmer/tmp/dashboards/main/mau_data.csv to s3://net-mozaws-prod-metrics-data/dashboards-landing-page/mau_data.csv
upload: ../home/hulmer/tmp/dashboards/mofo/learning-network-cities.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/learning-network-cities.json
upload: ../home/hulmer/tmp/dashboards/mofo/webmaker-maus.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/webmaker-maus.json
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_weekly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_monthly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_last_updated.json to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_last_updated.json
INFO: end copy at Mon Feb 29 18:03:12 UTC 2016

I additionally looked at the contents of /data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv and it looks up to date with the max date being 2016-02-26
(Reporter)

Comment 7

3 years ago
"s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/blocklistDecomposition.csv"

Is that "firefox-weekly-dashboard" bucket the same one used by https://metrics.services.mozilla.com/firefox-dashboard?

If you compare the two csv files in my last comment, you'll see that the one on https://metrics.services.mozilla.com/firefox-dashboard has records up to Nov 10, 2015.
(Reporter)

Comment 8

3 years ago
Wes, is this, by chance, something you might be able to provide insight into?
Flags: needinfo?(whd)

Comment 9

3 years ago
This is probably where the issue is. The firefox-weekly-dashboard was renamed to firefox-dashboard and data being pushed there should updated, e.g.

s3://net-mozaws-prod-metrics-data/firefox-dashboard/blocklistDecomposition.csv
Flags: needinfo?(whd)
(Reporter)

Comment 10

3 years ago
Thanks, I can confirm that the issue is fixed.
I manually pushed to firefox-dashboard and I renamed the location in the cron job:
/usr/bin/aws s3 cp /data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv s3://net-mozaws-prod-metrics-data/firefox-dashboard/blocklistDecomposition.csv

Should the config be renamed to firefox-dashboard for the four other files that push to firefox-weekly-dashboard as well?

confidence_intervals.csv
firefox_weekly_data.csv
firefox_monthly_data.csv
firefox/last_updated.json
Flags: needinfo?(hcrince)
(In reply to Matt Pressman [:mpressman] from comment #11)
> Should the config be renamed to firefox-dashboard for the four other files
> that push to firefox-weekly-dashboard as well?

Yes.
I have updated the rest of the files that were pointing to firefox-weekly-dashboard. One file, last_updated.json is not landing. The file only contains a date. At https://metrics.services.mozilla.com/firefox-dashboard/data/last_updated.json the result is: 2015-11-16 00:00:00

The location the push_files_to_s3.sh script gets it from: /home/hulmer/tmp/dashboards/firefox/last_updated.json as well as the previous location at https://metrics.services.mozilla.com/firefox-weekly-dashboard/data/last_updated.json show the value 2016-01-11 00:00:00

Without access to the bucket, I cannot determine what the problem is, but I'm guessing this should be looked into as to why it's not able to go to s3://net-mozaws-prod-metrics-data/firefox-dashboard/last_updated.json when it is able to go to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/last_updated.json
Flags: needinfo?(whd)
Flags: needinfo?(aalmossawi)
(Reporter)

Comment 14

3 years ago
last_updated.json is no longer needed, so feel free to ignore it. Thank you.
Flags: needinfo?(aalmossawi)
This was a permissions issue from when the dashboard was cut over, which should be fixed now.
Flags: needinfo?(whd)
right on, thank you both!

Comment 17

3 years ago
This can be closed now, right?
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.