1448049 - Intermittent hang in Android builds

Status: RESOLVED WONTFIX

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Geoff Brown [:gbrown]]

Bug 1411358 collects intermittent "Task timeout after N seconds" failures -- from a variety of causes. Recently I have noticed several such failures for Android builds. Logs seem to show a long (over 1 hour) period of inactivity leading up to the task timeout.

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169549148&lineNumber=24744-24749

[task 2018-03-21T22:11:19.240Z] 22:11:19     INFO -  	"xpcshell/20EDC3A4E69FE763B8AAC40C3CF77C110/xpcshell.sym"
[task 2018-03-21T22:11:19.240Z] 22:11:19     INFO -  make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-21T22:11:20.349Z] 22:11:20     INFO -  make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-21T22:11:20.349Z] 22:11:20     INFO -  ./config/nsinstall -D dist/test-stage/cppunittest
[task 2018-03-21T22:11:20.363Z] 22:11:20     INFO -  /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestScopeExit dist/test-stage/cppunittest/TestScopeExit; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestMaybe dist/test-stage/cppunittest/TestMaybe; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestMathAlgorithms dist/test-stage/cppunittest/TestMathAlgorithms; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestEndian dist/test-stage/cppunittest/TestEndian; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestBinarySearch dist/test-stage/cppunittest/TestBinarySearch; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestNotNull dist/test-stage/cppunittest/TestNotNull; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestTemplateLib dist/test-stage/cppunittest/TestTemplateLib; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestTuple dist/test-stage/cppunittest/TestTuple; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestCountPopulation dist/test-stage/cppunittest/TestCountPopulation; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestThreadSafeWeakPtr dist/test-stage/cppunittest/TestThreadSafeWeakPtr; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSegmentedVector dist/test-stage/cppunittest/TestSegmentedVector; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestArray dist/test-stage/cppunittest/TestArray; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestDoublyLinkedList dist/test-stage/cppunittest/TestDoublyLinkedList; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestDefineEnum dist/test-stage/cppunittest/TestDefineEnum; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestWrappingOperations dist/test-stage/cppunittest/TestWrappingOperations; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestXorShift128PlusRNG dist/test-stage/cppunittest/TestXorShift128PlusRNG; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestRange dist/test-stage/cppunittest/TestRange; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestPoisonArea dist/test-stage/cppunittest/TestPoisonArea; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/ShowSSEConfig dist/test-stage/cppunittest/ShowSSEConfig; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestIntegerRange dist/test-stage/cppunittest/TestIntegerRange; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSaturate dist/test-stage/cppunittest/TestSaturate; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestIntegerPrintfMacros dist/test-stage/cppunittest/TestIntegerPrintfMacros; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestPrintf dist/test-stage/cppunittest/TestPrintf; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestRefPtr dist/test-stage/cppunittest/TestRefPtr; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestLinkedList dist/test-stage/cppunittest/TestLinkedList; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSplayTree dist/test-stage/cppunittest/TestSplayTree; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestEnumeratedArray dist/test-stage/cppunittest/TestEnumeratedArray; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestCeilingFloor dist/test-stage/cppunittest/TestCeilingFloor; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSmallPointerArray dist/test-stage/cppunittest/TestSmallPointerArray; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestFastBernoulliTrial dist/test-stage/cppunittest/TestFastBernoulliTrial; /builds/worker/wor[taskcluster 2018-03-21 23:52:12.466Z] === Task Finished ===
[taskcluster 2018-03-21 23:52:12.466Z] Unsuccessful task run with exit code: -1 completed in 7202.129 seconds


More succinctly...

[task 2018-03-21T22:11:20.363Z] 22:11:20 ...
[taskcluster 2018-03-21 23:52:12.466Z] === Task Finished ===
[taskcluster 2018-03-21 23:52:12.466Z] Unsuccessful task run with exit code: -1 completed in 7202.129 seconds

Comment 1

[Geoff Brown [:gbrown]]

Another example:

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169418192&lineNumber=31454-31459

[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182436 res/drawable-mdpi-v4/ic_pause_dark.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182604 res/drawable-mdpi-v4/ic_pause_light.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182784 res/drawable-mdpi-v4/ic_play_dark.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183076 res/drawable-mdpi-v4/ic_play_light.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183364 res/drawable-mdpi-v4/ic_speaker_dark.png (OK)[taskcluster 2018-03-21 13:56:00.630Z] === Task Finished ===
[taskcluster 2018-03-21 13:56:00.631Z] Unsuccessful task run with exit code: -1 completed in 7202.27 seconds

Comment 2

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169362207&lineNumber=18600-18606

[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/components/reflect'
[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  libtoolkit_components_reflect.a.desc
[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  rm -f libtoolkit_components_reflect.a
[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/components/reflect'
[task 2018-03-21T04:04:33.384Z] 04:04:33     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/components/backgroundhangmonitor'
[task 2018-03-21T04:04:33.391Z] 04:04:33     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_ackgroundhangmonitor0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DMOZ_ENABLE_BACKGROUND_HANG_MONITOR=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor -I/builds/worker/workspace/build/src/obj-firefox/toolkit/components/backgroundhangmonitor -I/builds/worker/workspace/build/src/caps -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worke[taskcluster 2018-03-21 05:45:34.476Z] === Task Finished ===
[taskcluster 2018-03-21 05:45:34.476Z] Unsuccessful task run with exit code: -1 completed in 7201.71 seconds

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Geoff Brown [:gbrown] from comment #1)
> Another example:
> 
> https://treeherder.mozilla.org/logviewer.
> html#?repo=autoland&job_id=169418192&lineNumber=31454-31459
> 
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182436
> res/drawable-mdpi-v4/ic_pause_dark.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182604
> res/drawable-mdpi-v4/ic_pause_light.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182784
> res/drawable-mdpi-v4/ic_play_dark.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183076
> res/drawable-mdpi-v4/ic_play_light.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183364
> res/drawable-mdpi-v4/ic_speaker_dark.png (OK)[taskcluster 2018-03-21
> 13:56:00.630Z] === Task Finished ===
> [taskcluster 2018-03-21 13:56:00.631Z] Unsuccessful task run with exit code:
> -1 completed in 7202.27 seconds

This one in particular seems like the worker just died.  That output is the middle of a packaging task; it's essentially infallible and neither CPU nor IO bound.

In the same way we see "cannot write to file", it looks we we see "cannot continue computation".  I see nothing to suggest this is Android specific, or even really specific to the build system.  It seems more likely to be an issue with the compute provider or the container/image/kernel combination.

Comment 4

[Geoff Brown [:gbrown]]

Test failures continue in bug 1411358 but I rarely see such build failures now. And yeah, maybe the cause is not specific to Android builds.