1300060 - SHA1 repacks break uptake monitoring by timing out in release promotion

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

In https://bugzilla.mozilla.org/show_bug.cgi?id=1290179#c12 we enabled SHA1 repacks and added them to the stuff we check as part of the uptake monitoring step in release promotion.

Because SHA1 repacks take a bit longer to finish up and currently the builder is not chained as a dependency to uptake monitoring builder, the latter one started and timed-out for 5 retries in a row, eventually giving up as failed.

I'm thinking of three possible solutions here:
1. Chain up just the sha1 repacks builder as a dependency to uptake monitoring builder.
2. Chain the entire release-{}-{}_partner_repacks_copy_to_releases as a dependency to uptake monitoring builder to cope with existing uptake monitoring dependencies (push-to-releases)
3. Bump the uptake monitoring sleep/pause times to cover a larger time window to allow sha1 repacks to finish. 

From https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm70-build1-build2.txt.gz

00:39:32     INFO - Current uptake for Firefox-49.0b9-sha1 is 0
00:39:32     INFO - Requesting https://bounceradmin.mozilla.com/api/uptake/?product=Firefox-49.0b9-stub&os=win from tuxedo
00:39:32     INFO - Starting new HTTPS connection (1): bounceradmin.mozilla.com
/builds/slave/rel-m-beta-fx_uptk_mntr-000000/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

Comment 1

[Rail Aliiev [:rail]]

(In reply to Mihai Tabara [:mtabara] from comment #0)
> 1. Chain up just the sha1 repacks builder as a dependency to uptake
> monitoring builder.


My main concern here is that we don't want to be hard blocked by partner repacks. They may fail for some reason, but this should not be a blocker, esp for betas.


> 2. Chain the entire release-{}-{}_partner_repacks_copy_to_releases as a
> dependency to uptake monitoring builder to cope with existing uptake
> monitoring dependencies (push-to-releases)

1) without 2) doesn't make too much sense, because we have to wait for 2) to make uptake monitoring work. The same concerns apply here.


> 3. Bump the uptake monitoring sleep/pause times to cover a larger time
> window to allow sha1 repacks to finish. 

I think 2) is better, because you still have to wait for it to make this work.

I have 2 more options:

4) Separate the sha1 repacks graph branch from partner repacks, create 3rd push to mirrors for win32-sha1, create separate uptake monitoring for this, create a separate postrelease task to update bouncer aliases and make them depend on each other (and postrelease on the main publish release task). Adds a lot complexity though.

5) Almost the same as 4), but without separate uptake monitoring - the main uptake monitoring should depend on the main push to mirrors and sha1 push to mirrors (or we can merge pushes together)

2 or 5, pick one! :)

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Ah, right.

1) is non-sense since the push-partner-repacks-to-releases is needed anyway before talking about any uptake monitoring
2) is indeed the easiest option to be altered but as you said, adds some heavyburden-blocking on the general work-flow of release that is not needed
3) yeah - that was my main thinking as well. Uptake monitoring doesn't even have to start and keep timing-out in a loop until we have that partner_to_push_to_releases ready anyway.

At second thoughts, I don't like any of these.
I'll go with 5 I guess, seems the cleanest way. Unless there is way to tweak 2 to avoid the "blocking" part?

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Will resume to my WIP this week as we're yet again at the beginning of a beta-cycle and we can test the things out.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Bleah, I need to address this soon, 50.0b2 brings this up yet again https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm94-build1-build5.txt.gz 

Most likely I'll run uptake monitoring manually or rerun once sha1 is done.

Comment 5

[Rail Aliiev [:rail]]

Sorry, hijacking! :)

Comment 6

[Rail Aliiev [:rail]]

Attachment: PR

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8799925 [details] [review]
PR

Somehow I imagined this would be more complicated, given https://bugzilla.mozilla.org/show_bug.cgi?id=1300060#c2 but other than that, looks awesome, r+.

Apologies for procrastinating and thanks for taking care of it.

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8799925 [details] [review]
PR

deployed

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Uptake monitoring worked like a charm this time in 50.0b7 => https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm70-build1-build9.txt.gz

...
09:26:55 <release-notifications>w8_4uEGvQbe4y85S81Z0KA: [beetmover] firefox mozilla-beta partner repacks push to releases has completed successfully! Yay!
09:26:55 <release-notifications>https://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-50.0b7/build1/[beetmover]_firefox_mozilla-beta_partner_repacks_push_to_releases-all-w8_4uEGvQbe4y85S81Z0KA-0
09:27:57 <release-notifications> bQTOqnaNQdufcyO6z5bptA: firefox mozilla-beta uptake monitoring has completed successfully! Yay!
release-notifications> https://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-50.0b7/build1/firefox_mozilla-beta_uptake_monitoring-all-bQTOqnaNQdufcyO6z5bptA-0
...