1465799 - Windows build bustages USE_STUB_INSTALLER is not specified in the environment when Gecko 62 merges to Beta on 14-06-2018

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Andreea Pavel [:apavel]]

Central as Beta simulation:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d2fab7033cf95755ef892c50da8c8dbe11ceebd7&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=181102749

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=181102749&repo=try&lineNumber=1180

12:03:13     INFO -  ERROR: STUB installer expected to be enabled but
12:03:13     INFO -  ERROR: USE_STUB_INSTALLER is not specified in the environment

This is a regression from Bug 1286092

Comment 1

[Justin Wood (:Callek)]

Hrm this is a tricky one, I didn't expect to break simulation pushes but this won't break on the merge to beta (because it keys off taskcluster project, which doesn't have it enabled on try...)

I'm not sure what the right solution is offhand right now.

Comment 2

[Phabricator Automation]

Attachment: Bug 1465799 - Merge-Day Stubbing. r=RyanVM

Comment 3

[Justin Wood (:Callek)]

Attachment: Enable stub installer for win32/opt

Ryan said he'd fold this into the uplift-sim patch if it works.

This should fix it for now (We may need to adjust in a bit)

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Andrei, please include this patch in your next round of uplift simulations to see if it helps with this issue. Thanks!

Comment 5

[Andrei Ciure[:aciure]]

OK, we will, thanks.

Comment 6

[Justin Wood (:Callek)]

Looks like my patch above fixed it https://treeherder.mozilla.org/#/jobs?repo=try&revision=94f89908021275cc63f98e3111249000ecf74c97

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Callek, there's a related issue on maple where the l10n nightly jobs fails,
 https://treeherder.mozilla.org/#/jobs?repo=maple&revision=0fdbae4678f3e71479c6956124723e48352937af&selectedJob=182463324
but not the en-US nightly. Is there another maple-specific hack we can land there ?

Comment 9

[Intermittent Failures Robot]

11 failures in 669 pushes (0.016 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 11

Platform breakdown:
* windows2012-32: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1465799&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 10

[Justin Wood (:Callek)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #8)
> Callek, there's a related issue on maple where the l10n nightly jobs fails,
>  https://treeherder.mozilla.org/#/
> jobs?repo=maple&revision=0fdbae4678f3e71479c6956124723e48352937af&selectedJob
> =182463324
> but not the en-US nightly. Is there another maple-specific hack we can land
> there ?

Yes, we can fix this on maple-specific with patches to:

https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/build/windows.yml#639
and
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/build/windows.yml#269

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

I thought that at first too, but when I looked more closely on maple
* win32 en-US nightly ends up with USE_STUB_INSTALLER=1 in the env, is green and produces a stub
   https://tools.taskcluster.net/groups/ao2WmlM3S5G2HMSq76-qWg/tasks/elY6M1vSSGmc9tGdlAZhSw/details
* win32 *l10n* nightly doesn't have the env var, and is red
   https://tools.taskcluster.net/groups/ZtKvjRK3Q7OuxTS4ipfqFw/tasks/DB7-mhsnTQeDARPydzXoZQ/details

AIUI your links are for the en-US nightlies, so I'm not sure why they don't hit the 'default: false' on maple and have bustage. We have 'project: maple' in parameters.yml for both the push and promote decision tasks. 

And it looks like release l10n doesn't inherit use-stub-installer from the nightly job like we do for m-c, so perhaps there really is an issue with release promotion which needs fixing for 62.

Comment 12

[Phabricator Automation]

Attachment: Bug 1465799 - Makes sure that we don't check for a truthy stub-installer value until after resolving keyed by r=dustin

Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1465799#c11 specifically the fact that USE_STUB_INSTALLER is leaked through other tasks even though its not actually specified for other tasks

Comment 13

[Intermittent Failures Robot]

2 failures in 408 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 2

Platform breakdown:
* windows2012-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1465799&startday=2018-06-11&endday=2018-06-17&tree=trunk

Comment 14

[Cosmin Sabou [:CosminS]]

The issue still persists with the patch applied: https://treeherder.mozilla.org/logviewer.html#?job_id=183584809&repo=try&lineNumber=1537

https://treeherder.mozilla.org/#/jobs?repo=try&revision=536226982a00fd69ebccf2834cea87ae152baed1&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-searchStr=200978e1058ca08fcd06b81dde0645758c43cab2&selectedJob=183584809

Comment 15

[Justin Wood (:Callek)]

(In reply to Cosmin Sabou [:CosminS] from comment #14)
> The issue still persists with the patch applied:
> https://treeherder.mozilla.org/logviewer.
> html#?job_id=183584809&repo=try&lineNumber=1537
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=536226982a00fd69ebccf2834cea87ae152baed1&filter-
> resultStatus=testfailed&filter-resultStatus=busted&filter-
> resultStatus=exception&filter-
> searchStr=200978e1058ca08fcd06b81dde0645758c43cab2&selectedJob=183584809

https://bugzilla.mozilla.org/attachment.cgi?id=8983176 is the patch that would have been necessary this other patch is for a different (but similar) issue.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

This is actively burning Win32 builds on Beta62 now.

Comment 17

[Phabricator Automation]

Attachment: Bug 1465799 - Set expectations for stub installer to true on beta/release because we always set update channel as such on these branches. r=aki

Comment 18

[Phabricator Automation]

Comment on attachment 8985686 [details]
Bug 1465799 - Makes sure that we don't check for a truthy stub-installer value until after resolving keyed by r=dustin

Dustin J. Mitchell [:dustin] pronoun: he has approved the revision.

https://phabricator.services.mozilla.com/D1671

Comment 19

[Phabricator Automation]

Comment on attachment 8986248 [details]
Bug 1465799 - Set expectations for stub installer to true on beta/release because we always set update channel as such on these branches. r=aki

Aki Sasaki [:aki] has approved the revision.

https://phabricator.services.mozilla.com/D1708

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f07e22575ae
Makes sure that we don't check for a truthy stub-installer value until after resolving keyed by r=dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/bf1d08ea54a8
Set expectations for stub installer to true on beta/release because we always set update channel as such on these branches. r=aki

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for breaking gecko decision task:

https://hg.mozilla.org/integration/mozilla-inbound/rev/544a7362dd8a7d5e7c6178403f6d097faa172c11

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=bf1d08ea54a8b185bc4917782a5badaeda1e6a21&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=183836924&repo=mozilla-inbound
KeyError: u'stub-installer'

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/778e2a00786e
Makes sure that we don't check for a truthy stub-installer value until after resolving keyed by r=dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/36b51dda5abe
Set expectations for stub installer to true on beta/release because we always set update channel as such on these branches. r=aki

Comment 23

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/778e2a00786e
https://hg.mozilla.org/mozilla-central/rev/36b51dda5abe

Comment 24

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed in recent beta simulations: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6ab490786fd29454fab7bc274a5248d9bd462dd6&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Comment 25

[Intermittent Failures Robot]

5 failures in 521 pushes (0.01 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 5

Platform breakdown:
* windows2012-32: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1465799&startday=2018-06-18&endday=2018-06-24&tree=trunk

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/projects/maple/rev/f6f8b14aa7097f314a06d9b7a86849e021ba54e5 is the maple hack.