1515282 - Perma Android 4.0 API16+ CCov debug bustage package> make[5]: *** [make-package-internal] Error 1

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Eliza Balazs [:ebalazs_]]

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=217832055&repo=mozilla-central&lineNumber=2626

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=be5e54edad6088b4179c8c1b617664db62150388&selectedJob=217831301&searchStr=android%2C4.0%2Capi16%2B%2Cccov%2Cdebug%2Cbuild-android-api-16-ccov%2Fdebug%2C%28b%29

package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-12-19T05:24:00.776Z] 05:24:00     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  reftest '/builds/worker/artifacts/target.reftest.tests.tar.gz'
[task 2018-12-19T05:24:00.776Z] 05:24:00     INFO -  package-tests> Wrote 30781 files in 52875646 bytes to target.reftest.tests.tar.gz in 23.12s
[task 2018-12-19T05:24:00.777Z] 05:24:00     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-12-19T05:24:00.780Z] 05:24:00     INFO -  client.mk:129: recipe for target 'build' failed
[task 2018-12-19T05:24:00.780Z] 05:24:00     INFO -  make: *** [build] Error 2
[task 2018-12-19T05:24:00.809Z] 05:24:00     INFO -  0 compiler warnings present.
[task 2018-12-19T05:24:00.837Z] 05:24:00    ERROR - Return code: 2
[task 2018-12-19T05:24:00.838Z] 05:24:00  WARNING - setting return code to 2
[task 2018-12-19T05:24:00.838Z] 05:24:00    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-12-19T05:24:00.838Z] 05:24:00    FATAL - Running post_fatal callback...
[task 2018-12-19T05:24:00.838Z] 05:24:00    FATAL - Exiting -1
[task 2018-12-19T05:24:00.838Z] 05:24:00     INFO - [mozharness: 2018-12-19 05:24:00.838730Z] Finished build step (failed)
[task 2018-12-19T05:24:00.838Z] 05:24:00     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2018-12-19T05:24:00.839Z] 05:24:00     INFO - Running post-run listener: _shutdown_sccache
[task 2018-12-19T05:24:00.839Z] 05:24:00     INFO - Running command: ['/builds/worker/workspace/build/src/sccache2/sccache', '--stop-server'] in /builds/worker/workspace/build/src
[task 2018-12-19T05:24:00.839Z] 05:24:00     INFO - Copy/paste: /builds/worker/workspace/build/src/sccache2/sccache --stop-server
[task 2018-12-19T05:24:00.844Z] 05:24:00     INFO -  Stopping sccache server...
[task 2018-12-19T05:24:00.844Z] 05:24:00     INFO -  error: couldn't connect to server
[task 2018-12-19T05:24:00.844Z] 05:24:00     INFO -  caused by: Connection refused (os error 111)
[task 2018-12-19T05:24:00.844Z] 05:24:00    ERROR - Return code: 2
[task 2018-12-19T05:24:00.844Z] 05:24:00     INFO - Running post-run listener: _summarize
[task 2018-12-19T05:24:00.845Z] 05:24:00    ERROR - # TBPL FAILURE #
[task 2018-12-19T05:24:00.845Z] 05:24:00     INFO - [mozharness: 2018-12-19 05:24:00.845282Z] FxDesktopBuild summary:
[task 2018-12-19T05:24:00.845Z] 05:24:00    ERROR - # TBPL FAILURE #
[taskcluster 2018-12-19 05:24:01.167Z] === Task Finished ===

This seems to be related to https://hg.mozilla.org/mozilla-central/rev/6fd3033d7cb7
:nalexander, Can you have a look at this?

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Presumably a regression from bug 1509573.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #1)
> Presumably a regression from bug 1509573.

Correct.  Leaving NI, 'cuz I still need to fix it, but:

This is because Bug 1509573 completely broke Android artifact builds.  The offending thing is that I pushed the `android_apks` target into the `export` tier... which has special handling that I did not know about here:

https://searchfox.org/mozilla-central/rev/13788edbabb04d004e4a1ceff41d4de68a8320a2/python/mozbuild/mozbuild/backend/recursivemake.py#595

That is, we never actually force the export tier in this situation.  Luckily I think the way to address this is straight-forward; patch inbound.

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1515282 - Fix mobile/android artifact build bustage.

This was fallout from Bug 1509573.  That ticket pushed the Android APKs step
into the export tier, where it is required; but since most things in export are
only required for compilation, the target is not itself built by default, and
in particular, not during an artifact build.  That's not right; this fixes it.

Comment 4

[Nick Alexander :nalexander [he/him]]

This is looking happy on try.  Waiting for build peer review.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0cd83e32e55e7c860ae3309f682f2a813318adda

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/58305b0665c9
Fix mobile/android artifact build bustage. r=froydnj

Comment 6

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/58305b0665c9

Comment 7

[Intermittent Failures Robot]

14 failures in 2648 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 14

Platform breakdown:
* android-4-0-armv7-api16-ccov: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1515282&startday=2018-12-17&endday=2018-12-23&tree=all