1239493 - Firefox Dashboard -- Deseasonalized ADI numbers are not being pushed to the dashboard

Status: RESOLVED FIXED

(Cloud Services :: Metrics: Dashboard, defect)

Description

[Ali Almossawi]

The numbers at the data-source on dashboard1 are being updated, but aren't being pushed to S3, it would appear.

Also filed on GitHub: https://github.com/mozilla/firefox-dashboard/issues/10

Comment 1

[John Jensen]

Heather's looking into this...

Comment 2

[[:josephine]]

As far as I can see, vertica database is updated with up to yesterday's data, so it's not a vertica issue. Is there a script that runs a file to create the dashboard? If so, what is the name of the file?

Comment 3

[Ali Almossawi]

The issue is with the pushing to S3. Sheeri owned these scripts. Who owns them in her absence? mpressman?

Comment 4

[Matt Pressman [:mpressman]]

which file is this? There are no errors in the logs for the cron that sends files to s3?

Comment 5

[Ali Almossawi]

blocklistDecomposition.csv

From (dashboard1): https://metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv

To: https://metrics.services.mozilla.com/firefox-dashboard/data/blocklistDecomposition.csv

Comment 6

[Matt Pressman [:mpressman]]

It looks like the push_files_to_s3.sh script on dashboard1.metrics.scl3 is pushing blocklistDecomposition.csv to s3 with the rest of the files - Here is the log output of the push for what has run so far today:

INFO: beginning copy at Mon Feb 29 11:00:01 UTC 2016
upload: ../data/www/metrics.mozilla.com/protected/cchoi/confidence_intervals.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/confidence_intervals.csv
upload: ../data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/blocklistDecomposition.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_weekly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_monthly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_weekly_data.csv to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_last_updated.json to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_last_updated.json
upload: ../home/hulmer/tmp/dashboards/main/mau_data.csv to s3://net-mozaws-prod-metrics-data/dashboards-landing-page/mau_data.csv
upload: ../home/hulmer/tmp/dashboards/mofo/learning-network-cities.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/learning-network-cities.json
upload: ../home/hulmer/tmp/dashboards/mofo/webmaker-maus.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/webmaker-maus.json
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_weekly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_monthly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_last_updated.json to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_last_updated.json
INFO: end copy at Mon Feb 29 11:00:10 UTC 2016
INFO: beginning copy at Mon Feb 29 18:03:01 UTC 2016
upload: ../data/www/metrics.mozilla.com/protected/cchoi/confidence_intervals.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/confidence_intervals.csv
upload: ../data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/blocklistDecomposition.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_weekly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/firefox/firefox_monthly_data.csv to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/firefox_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_weekly_data.csv to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/content-services/up_fhr_last_updated.json to s3://net-mozaws-prod-metrics-data/content-services-dashboard/up_fhr_last_updated.json
upload: ../home/hulmer/tmp/dashboards/main/mau_data.csv to s3://net-mozaws-prod-metrics-data/dashboards-landing-page/mau_data.csv
upload: ../home/hulmer/tmp/dashboards/mofo/learning-network-cities.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/learning-network-cities.json
upload: ../home/hulmer/tmp/dashboards/mofo/webmaker-maus.json to s3://net-mozaws-prod-metrics-data/mofo-kpis/webmaker-maus.json
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_weekly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_weekly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_monthly_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_monthly_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_data.csv to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_data.csv
upload: ../home/hulmer/tmp/dashboards/fennec/fennec_adi_last_updated.json to s3://net-mozaws-prod-metrics-data/fennec-dashboard/fennec_adi_last_updated.json
INFO: end copy at Mon Feb 29 18:03:12 UTC 2016

I additionally looked at the contents of /data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv and it looks up to date with the max date being 2016-02-26

Comment 7

[Ali Almossawi]

"s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/blocklistDecomposition.csv"

Is that "firefox-weekly-dashboard" bucket the same one used by https://metrics.services.mozilla.com/firefox-dashboard?

If you compare the two csv files in my last comment, you'll see that the one on https://metrics.services.mozilla.com/firefox-dashboard has records up to Nov 10, 2015.

Comment 8

[Ali Almossawi]

Wes, is this, by chance, something you might be able to provide insight into?

Comment 9

[Wesley Dawson [:whd]]

This is probably where the issue is. The firefox-weekly-dashboard was renamed to firefox-dashboard and data being pushed there should updated, e.g.

s3://net-mozaws-prod-metrics-data/firefox-dashboard/blocklistDecomposition.csv

Comment 10

[Ali Almossawi]

Thanks, I can confirm that the issue is fixed.

Comment 11

[Matt Pressman [:mpressman]]

I manually pushed to firefox-dashboard and I renamed the location in the cron job:
/usr/bin/aws s3 cp /data/www/metrics.mozilla.com/protected/bcolloran/dailyUsageSignals/data/blocklistDecomposition.csv s3://net-mozaws-prod-metrics-data/firefox-dashboard/blocklistDecomposition.csv

Should the config be renamed to firefox-dashboard for the four other files that push to firefox-weekly-dashboard as well?

confidence_intervals.csv
firefox_weekly_data.csv
firefox_monthly_data.csv
firefox/last_updated.json

Comment 12

[Wesley Dawson [:whd]]

(In reply to Matt Pressman [:mpressman] from comment #11)
> Should the config be renamed to firefox-dashboard for the four other files
> that push to firefox-weekly-dashboard as well?

Yes.

Comment 13

[Matt Pressman [:mpressman]]

I have updated the rest of the files that were pointing to firefox-weekly-dashboard. One file, last_updated.json is not landing. The file only contains a date. At https://metrics.services.mozilla.com/firefox-dashboard/data/last_updated.json the result is: 2015-11-16 00:00:00

The location the push_files_to_s3.sh script gets it from: /home/hulmer/tmp/dashboards/firefox/last_updated.json as well as the previous location at https://metrics.services.mozilla.com/firefox-weekly-dashboard/data/last_updated.json show the value 2016-01-11 00:00:00

Without access to the bucket, I cannot determine what the problem is, but I'm guessing this should be looked into as to why it's not able to go to s3://net-mozaws-prod-metrics-data/firefox-dashboard/last_updated.json when it is able to go to s3://net-mozaws-prod-metrics-data/firefox-weekly-dashboard/last_updated.json

Comment 14

[Ali Almossawi]

last_updated.json is no longer needed, so feel free to ignore it. Thank you.

Comment 15

[Wesley Dawson [:whd]]

This was a permissions issue from when the dashboard was cut over, which should be fixed now.

Comment 16

[Matt Pressman [:mpressman]]

right on, thank you both!

Comment 17

[John Jensen]

This can be closed now, right?