1442036 - Linux and Windows opt builds are going to permafail when Gecko 60 merges to Beta on 2018-03-01

Status: VERIFIED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

central-as-beta uplift sim with broken Linux and Windows opt builds: https://treeherder.mozilla.org/#/jobs?repo=try&revision=47f90d887c656d192abaf8cb49f2558eb7af72a2&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=pending&filter-resultStatus=running

Bisecting indicates that it is from this push: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=5297541590781af40ff09e067646f3115960af75

Successful uplift simulation before that push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a1f8c30a86d22085280ddd163124431b79666c9&selectedJob=164962556

uplift simulation with that push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5dd66180805ca7fef8418ff29750a215f4eef891&selectedJob=164961242
The build job is still running but should already be done.

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=164887190&repo=try
End of log with issue:
[task 2018-02-28T16:35:36.104Z] 16:35:36     INFO -  /builds/worker/workspace/build/src/obj-firefox/_virtualenv/bin/python -m mozbuild.action.langpack_manifest --locales en-US --min-app-ver 60.0 --max-app-ver 60.* --app-name "Firefox" --l10n-basedir "/builds/worker/.mozbuild/l10n-central" --defines /builds/worker/workspace/build/src/toolkit/locales/en-US/defines.inc /builds/worker/workspace/build/src/browser/locales/en-US/defines.inc  --input ../../dist/xpi-stage/locale-en-US
[task 2018-02-28T16:35:36.104Z] 16:35:36     INFO -  make[6]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/browser/locales'
[task 2018-02-28T16:35:36.245Z] 16:35:36     INFO -  make[6]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/browser/locales'
[task 2018-02-28T16:35:36.245Z] 16:35:36     INFO -  /builds/worker/workspace/build/src/obj-firefox/_virtualenv/bin/python -m mozbuild.action.zip -C ../../dist/xpi-stage/locale-en-US -x **/*.manifest -x **/*.js -x **/*.ini /builds/worker/workspace/build/src/obj-firefox/dist/target.langpack.xpi chrome browser manifest.json
[task 2018-02-28T16:35:36.245Z] 16:35:36     INFO -  make[6]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/browser/locales'
[task 2018-02-28T16:35:36.246Z] 16:35:36     INFO -  make[4]: Nothing to be done for 'tools'.
[task 2018-02-28T16:35:36.246Z] 16:35:36     INFO -  make[1]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T16:35:36.246Z] 16:35:36     INFO -  rm -f jarlog/en-US.log
[task 2018-02-28T16:35:36.246Z] 16:35:36     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T16:35:36.246Z] 16:35:36     INFO -  make[1]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T16:35:36.247Z] 16:35:36     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T16:35:36.248Z] 16:35:36     INFO -  make[1]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T16:35:36.248Z] 16:35:36     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T17:15:36.281Z] 17:15:36     INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'build', '-v']

The successful log continues like this:
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  make[1]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  MOZ_PGO_INSTRUMENTED=1 JARLOG_FILE=jarlog/en-US.log /builds/worker/workspace/build/src/obj-firefox/_virtualenv/bin/python /builds/worker/workspace/build/src/build/pgo/profileserver.py 10
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  *** UTM:SVC TimerManager:registerTimer called after profile-before-change notification. Ignoring timer registration for id: telemetry_modules_ping
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  ###!!! [Parent][DispatchAsyncMessage] Error: PClientManager::Msg_Teardown Route error: message sent to unknown actor ID
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  JavaScript error: resource://gre/modules/osfile/osfile_async_front.jsm, line 408: Error: OS.File has been shut down. Rejecting post to DirectoryIterator_prototype_close
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  JavaScript error: resource://gre/modules/osfile/osfile_async_front.jsm, line 408: Error: OS.File has been shut down. Rejecting post to remove
[task 2018-02-28T20:54:52.655Z] 20:54:52     INFO -  JavaScript error: resource://gre/modules/osfile/osfile_async_front.jsm, line 408: Error: OS.File has been shut down. Rejecting post to remove
[task 2018-02-28T20:54:52.656Z] 20:54:52     INFO -  jarlog: /builds/worker/workspace/build/src/obj-firefox/jarlog/en-US.log
[task 2018-02-28T20:54:52.656Z] 20:54:52     INFO -  *** WIFI GEO: shutdown called
[task 2018-02-28T20:54:52.656Z] 20:54:52     INFO -  make[1]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'

Fallout from bug 1434729?

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Prime suspect bug 888600.

Good central:
Revision: https://hg.mozilla.org/mozilla-central/rev/eeb50080d6de 
Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=257e9803aced8e8dbf1ccb26b1820d5dcf640438 

Good inbound:
Revision: https://hg.mozilla.org/integration/mozilla-inbound/rev/724b51d7263f 
Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3c150a7fa658a581ed7de229b2fa91258dad7df6 

Bad:
Revision: https://hg.mozilla.org/integration/mozilla-inbound/rev/83c87140dc3d 
Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=aae2d84852c0de7426f0355cb3599919b22925ee

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

central-as-beta simulation with bug 888600 backed out succeeded: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5df390036cea740eabc0938b6d2102908af2977&selectedJob=165071975
The new Linux opt bustages are from a different bug.

Bug 888600 got backed out from central.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed in today's central-as-beta simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=af077909ea82311a0ef6d326bdda221148926b8b&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable