1448070 - busted Linux and Windows opt builds when Gecko 61 merges to Beta on 2018-04-26

Status: VERIFIED FIXED

(Toolkit :: Add-ons Manager, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Like mentioned yesterday, the patches for bug 1446676 cause Linux and Windows opt builds of central-as-beta simulations to go bust. Bug 1446676 got backed out later yesterday for mass failing T-P-e10s for OS X nightlies. It relanded later and the build issue is back for the beta simulations.

Today's central-as-beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811e2ea438a88cb9e407f4ab66387bc2c18d0f35&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&group_state=expanded

Build log: https://treeherder.mozilla.org/logviewer.html#?job_id=169640968&repo=try

[task 2018-03-22T12:31:33.417Z] 12:31:33     INFO -  rm -f jarlog/en-US.log
[task 2018-03-22T12:31:33.417Z] 12:31:33     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-22T12:31:33.417Z] 12:31:33     INFO -  make[1]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-22T12:31:33.417Z] 12:31:33     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-22T12:31:33.417Z] 12:31:33     INFO -  make[1]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-22T12:31:33.417Z] 12:31:33     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-22T13:11:33.457Z] 13:11:33     INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'build', '-v']
[task 2018-03-22T13:11:33.479Z] 13:11:33    ERROR - timed out after 2400 seconds of no output
[task 2018-03-22T13:11:33.480Z] 13:11:33    ERROR - Return code: -15

This shall get fixed soon to prevent losing our beta 61 coverage.

FTR, yesterday's bisecting to identify the bug:

Last good: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1bca4bb60e9c074aeee4f1aefff196a76e2522cf
First bad: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff89401a888803eda8fee7d54cbc7461b6e0aef4

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This blocks being able to do useful uplift simulations, upping the severity.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

A backout of bug 1446676 didn't fix the issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fcada7dffac6f3f6ec64d86d00a565495d781996&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success

A bisection on central showed:

First bad: https://treeherder.mozilla.org/#/jobs?repo=try&revision=22956532b934f0bceee31dbab71528ee04d79c51&selectedJob=169862175
Treeherder: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=b41c7c1ff91f49b1500b087ff23c288dd88f1fde&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Last good: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3761f9bbdb75f5c082656a5cd61aec1147b12518&selectedJob=169863505

Bisection of that inbound to central merge on inbound always returned build failures for me.

Comment 3

[Kris Maglione [:kmag]]

When I try to do a PGO build locally, what actually times out is the last rustc stage that builds gkrust. I suspect that's what's happening here, too. It doesn't look like it ever gets to the point of running Firefox.

Comment 4

[Kris Maglione [:kmag]]

No, I take that back. It looks like I'm wrong, and the last version of the quitter extension was signed as a privileged extension.

It looks like aswan is the person who added that version, so I suppose he'll need to take care of signing the new one.

Comment 5

[Andrew Swan [:aswan]]

See bug 1393162.  I'll try to take care of it later today if nobody beats me to it before then.

Comment 6

[Andrew Swan [:aswan]]

Attachment: [e-mail]

I attached a recent copy of quitter.xpi, with a stray call to removeObserver() removed, I'll include that in the final patch with the updated signed xpi.

Jason, can you sign this with "Mozilla Extensions", just like in bug 1393162?

Comment 7

[:wezhou]

Attachment: signed.8962066.xpi

Signed file attached. Please test.

Hi Andrew, did you know "Mozilla Extensions" aka Internal Extensions signing is now self service? If you often need to have this type of add-ons signed, please consider request access to sign them using the process here
https://mana.mozilla.org/wiki/display/FIREFOX/Internal+Extension+Signing.

Thanks!

Comment 8

[Andrew Swan [:aswan]]

Attachment: Bug 1448070 Sign quitter.xpi as privileged

Review commit: https://reviewboard.mozilla.org/r/231292/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/231292/

Comment 9

[Andrew Swan [:aswan]]

Ryan, can you either point me to the latest simulated beta patch set or launch a test build with the attached patch?

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thank you for the patch. Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c401feb61d8af9028bb3d5b8102da97814e4ed03

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Scratch that, new Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9da1fca9a1c62606149e777f64f5c7025b1a3e2a

Comment 12

[Andrew Swan [:aswan]]

The bug summary doesn't say this but the issue is with PGO builds right?  That try job doesn't include pgo...

Comment 13

[Ryan VanderMeulen [:RyanVM]]

They're PGO by virtue of the in-tree config changes made by the simulation patches (same way it's done on "real" Beta/Release)

Comment 14

[Kris Maglione [:kmag]]

Comment on attachment 8962447 [details]
Bug 1448070 Sign quitter.xpi as privileged

https://reviewboard.mozilla.org/r/231292/#review236786

Thanks!

Comment 15

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s f031c0efc040003e8443945a4680a21d85c19eaf -d 226c8c740c34: rebasing 454085:f031c0efc040 "Bug 1448070 Sign quitter.xpi as privileged r=kmag" (tip)
merging tools/quitter/bootstrap.js
merging tools/quitter/[e-mail]
warning: tools/quitter/[e-mail] looks like a binary file.
warning: conflicts while merging tools/quitter/[e-mail]! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)

Comment 16

[Andrew Swan [:aswan]]

Comment on attachment 8962447 [details]
Bug 1448070 Sign quitter.xpi as privileged

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/231292/diff/1-2/

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cdc8f6ff1a88
Sign quitter.xpi as privileged r=kmag

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/cdc8f6ff1a88

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thank you, verified fixed in the latest central-as-beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=62fff771b6a6ffadea4f7383c0986f0e5f4f2cda&group_state=expanded&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable