1481861 - Permafailing LTO opt make[1]: *** [automation/package] Error 2 when Gecko 63 merges to Beta on 2018-08-27

Status: VERIFIED FIXED

(Firefox Build System :: General, defect)

Description

[Natalia Csoregi [:nataliaCs]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1980582224a1da634a8d126f7e35b35fbe9afcf7&group_state=expanded&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&selectedJob=192769978

Failure log:
https://treeherder.mozilla.org/logviewer.html#?job_id=192769978&repo=try&lineNumber=39017

[task 2018-08-08T13:18:42.107Z] 13:18:42     INFO -  package> /builds/worker/workspace/build/src/browser/build.mk:6: recipe for target 'package' failed
[task 2018-08-08T13:18:42.107Z] 13:18:42     INFO -  package> make[2]: *** [package] Error 2
[task 2018-08-08T13:18:42.107Z] 13:18:42     INFO -  /builds/worker/workspace/build/src/build/moz-automation.mk:89: recipe for target 'automation/package' failed
[task 2018-08-08T13:18:42.107Z] 13:18:42     INFO -  make[1]: *** [automation/package] Error 2
[task 2018-08-08T13:18:42.108Z] 13:18:42     INFO -  make[1]: *** Waiting for unfinished jobs....
[task 2018-08-08T13:18:43.869Z] 13:18:43     INFO -  package-tests> make[3]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/src/tests'
[task 2018-08-08T13:18:43.869Z] 13:18:43     INFO -  package-tests> ../../../config/nsinstall -D ../../../dist/test-stage/jsreftest/tests
[task 2018-08-08T13:18:43.869Z] 13:18:43     INFO -  package-tests> (cd /builds/worker/workspace/build/src/js/src/tests && tar -chf - jsreftest.html shell.js browser.js js-test-driver-end.js user.js non262/ shell/ test/ test262/ ) | (cd ../../../dist/test-stage/jsreftest/tests && tar -xf -)
[task 2018-08-08T13:18:43.869Z] 13:18:43     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python /builds/worker/workspace/build/src/js/src/tests/jstests.py --make-manifests ../../../dist/test-stage/jsreftest/tests/
[task 2018-08-08T13:18:43.870Z] 13:18:43     INFO -  package-tests> make[3]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/js/src/tests'
[task 2018-08-08T13:18:44.026Z] 13:18:44     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:44.026Z] 13:18:44     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  awsy '/builds/worker/artifacts/target.awsy.tests.tar.gz'
[task 2018-08-08T13:18:44.026Z] 13:18:44     INFO -  package-tests> Wrote 17 files in 14902 bytes to target.awsy.tests.tar.gz in 0.04s
[task 2018-08-08T13:18:44.027Z] 13:18:44     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.060Z] 13:18:49     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.060Z] 13:18:49     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  gtest '/builds/worker/artifacts/target.gtest.tests.zip'
[task 2018-08-08T13:18:49.060Z] 13:18:49     INFO -  package-tests> Wrote 141 files in 56084012 bytes to target.gtest.tests.zip in 5.08s
[task 2018-08-08T13:18:49.060Z] 13:18:49     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.121Z] 13:18:49     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.121Z] 13:18:49     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  raptor '/builds/worker/artifacts/target.raptor.tests.tar.gz'
[task 2018-08-08T13:18:49.121Z] 13:18:49     INFO -  package-tests> Wrote 932 files in 7795402 bytes to target.raptor.tests.tar.gz in 5.15s
[task 2018-08-08T13:18:49.122Z] 13:18:49     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.137Z] 13:18:49     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.137Z] 13:18:49     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  cppunittest '/builds/worker/artifacts/target.cppunittest.tests.tar.gz'
[task 2018-08-08T13:18:49.137Z] 13:18:49     INFO -  package-tests> Wrote 63 files in 9362112 bytes to target.cppunittest.tests.tar.gz in 5.18s
[task 2018-08-08T13:18:49.137Z] 13:18:49     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.795Z] 13:18:49     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:49.795Z] 13:18:49     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  talos '/builds/worker/artifacts/target.talos.tests.tar.gz'
[task 2018-08-08T13:18:49.795Z] 13:18:49     INFO -  package-tests> Wrote 1380 files in 16417541 bytes to target.talos.tests.tar.gz in 5.85s
[task 2018-08-08T13:18:49.795Z] 13:18:49     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:50.192Z] 13:18:50     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:50.192Z] 13:18:50     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  xpcshell '/builds/worker/artifacts/target.xpcshell.tests.tar.gz'
[task 2018-08-08T13:18:50.192Z] 13:18:50     INFO -  package-tests> Wrote 5859 files in 10308296 bytes to target.xpcshell.tests.tar.gz in 6.20s
[task 2018-08-08T13:18:50.193Z] 13:18:50     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:58.604Z] 13:18:58     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:18:58.604Z] 13:18:58     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  reftest '/builds/worker/artifacts/target.reftest.tests.tar.gz'
[task 2018-08-08T13:18:58.604Z] 13:18:58     INFO -  package-tests> Wrote 30441 files in 47025643 bytes to target.reftest.tests.tar.gz in 14.60s
[task 2018-08-08T13:18:58.604Z] 13:18:58     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:02.211Z] 13:19:02     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:02.211Z] 13:19:02     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  common '/builds/worker/artifacts/target.common.tests.tar.gz'
[task 2018-08-08T13:19:02.211Z] 13:19:02     INFO -  package-tests> Wrote 44058 files in 27048643 bytes to target.common.tests.tar.gz in 18.22s
[task 2018-08-08T13:19:02.212Z] 13:19:02     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:02.580Z] 13:19:02     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:02.580Z] 13:19:02     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  mochitest '/builds/worker/artifacts/target.mochitest.tests.tar.gz'
[task 2018-08-08T13:19:02.580Z] 13:19:02     INFO -  package-tests> Wrote 29986 files in 55687462 bytes to target.mochitest.tests.tar.gz in 18.59s
[task 2018-08-08T13:19:02.580Z] 13:19:02     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:05.545Z] 13:19:05     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:05.546Z] 13:19:05     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  web-platform '/builds/worker/artifacts/target.web-platform.tests.tar.gz'
[task 2018-08-08T13:19:05.546Z] 13:19:05     INFO -  package-tests> Wrote 66548 files in 51137169 bytes to target.web-platform.tests.tar.gz in 21.54s
[task 2018-08-08T13:19:05.546Z] 13:19:05     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-08-08T13:19:05.546Z] 13:19:05     INFO -  client.mk:154: recipe for target 'build' failed
[task 2018-08-08T13:19:05.546Z] 13:19:05     INFO -  make: *** [build] Error 2
[task 2018-08-08T13:19:05.573Z] 13:19:05     INFO -  285 compiler warnings present.

Comment 1

[Mike Hommey [:glandium]]

Can you check after bug 1481727?

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Mike Hommey [:glandium] from comment #1)
> Can you check after bug 1481727?
Still fails: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c404fb67c1dc7926b7027fea3ea1ba0e8da7579&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=192975300

Comment 3

[Mike Hommey [:glandium]]

Attachment: libmozavutil.so causing the error

I can reproduce with this file taken from an interactive task from your job.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Move the eh_frame sections when they are before the first executable section.

In some rare cases, it is possible for one of the eh_frame sections'
original address to be larger than the address of the injected code
section, which is added before the first executable section. Namely,
this happens in the rare case where that eh_frame section is smaller
than the injected code section, and is adjacent to the first executable
section. We obviously want to move the eh_frame sections in that case,
since one of them is in the way.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b4eb54d89f0
Move the eh_frame sections when they are before the first executable section. r=froydnj

Comment 6

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/5b4eb54d89f0

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed with today's beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f2fbccfc0a525116d76c93a563bbdee252635e3&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable