1284885 - Need to re-run ChromeHangsWeekly job (atmo) from 3/7 to 6/5

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[David Durst [:ddurst]]

The logs for the ChromeHangsWeekly ATMO job show a failure to symbolicate, and so the data is not present in https://telemetry.mozilla.org/chromehangs/ for... well, a lot of dates. It picks back up with successful runs starting 20160606 to 20160612.

Mark Reid tested a re-run on data in mid-May and said that it ran without issue, so perhaps this has all been a result of bustage on the symbolication server side (which has been patched since mozlondon).

Assuming that's the case, we should re-run the job for the following weeks:
05/30 - 06/05
05/23 - 05/29
05/16 - 05/22
05/09 - 05/15
05/02 - 05/08
04/25 - 05/01
04/18 - 04/24
04/11 - 04/17
04/04 - 04/10
03/28 - 04/03
03/21 - 03/27
03/14 - 03/20
03/07 - 03/13

This takes us back to the beginning of FF48 (which seems like a reasonable point, from an e10s POV).

If it continues to fail, we will have to look at something else in the chain.

I currently own this scheduled job (1009: s3://telemetry-analysis-code-2/jobs/ChromeHangsWeekly/chromehangs-0.4.tar.gz via ./run_public.sh) -- do I need to be the one to re-run it?

Comment 1

[Mark Reid [:mreid]]

Yeah, if you're able to re-run those periods, it's likely to get done much more quickly. I can offer advice / support though.

If that works for you, please go ahead and 'take' this bug.

Comment 2

[David Durst [:ddurst]]

I'ma start slow (via scheduled job).

Comment 3

[David Durst [:ddurst]]

Currently re-running, on the week of 4/11 (averaging processing ~1wk/day). Will update again with NI when I'm ready to move the resulting files to the proper location.

Comment 4

[David Durst [:ddurst]]

This back-filled data should be complete by noon tomorrow (7/22). I'll need some assistance in putting the output files in the right place.

Comment 5

[Mark Reid [:mreid]]

Some assistance sent via email.

Comment 6

[David Durst [:ddurst]]

The files have been moved to the proper bucket (as far as I can tell), but if you page back to the updated date ranges, I'm getting nothing.

I see that the request for the gz data itself is returning 200, but... nothing is displaying, nor is there any error in the console. So I'm at a loss for why this isn't now fixed.

Comment 7

[David Durst [:ddurst]]

Ah. I didn't realize what "version" meant. Data's there, just not for 50.