Closed Bug 1448049 Opened 3 years ago Closed 3 years ago

Intermittent hang in Android builds

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect)

defect
Not set
normal

Tracking

(firefox61 affected)

RESOLVED WONTFIX
Tracking Status
firefox61 --- affected

People

(Reporter: gbrown, Unassigned)

References

(Blocks 1 open bug)

Details

Bug 1411358 collects intermittent "Task timeout after N seconds" failures -- from a variety of causes. Recently I have noticed several such failures for Android builds. Logs seem to show a long (over 1 hour) period of inactivity leading up to the task timeout.

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169549148&lineNumber=24744-24749

[task 2018-03-21T22:11:19.240Z] 22:11:19     INFO -  	"xpcshell/20EDC3A4E69FE763B8AAC40C3CF77C110/xpcshell.sym"
[task 2018-03-21T22:11:19.240Z] 22:11:19     INFO -  make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-21T22:11:20.349Z] 22:11:20     INFO -  make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-21T22:11:20.349Z] 22:11:20     INFO -  ./config/nsinstall -D dist/test-stage/cppunittest
[task 2018-03-21T22:11:20.363Z] 22:11:20     INFO -  /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestScopeExit dist/test-stage/cppunittest/TestScopeExit; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestMaybe dist/test-stage/cppunittest/TestMaybe; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestMathAlgorithms dist/test-stage/cppunittest/TestMathAlgorithms; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestEndian dist/test-stage/cppunittest/TestEndian; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestBinarySearch dist/test-stage/cppunittest/TestBinarySearch; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestNotNull dist/test-stage/cppunittest/TestNotNull; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestTemplateLib dist/test-stage/cppunittest/TestTemplateLib; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestTuple dist/test-stage/cppunittest/TestTuple; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestCountPopulation dist/test-stage/cppunittest/TestCountPopulation; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestThreadSafeWeakPtr dist/test-stage/cppunittest/TestThreadSafeWeakPtr; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSegmentedVector dist/test-stage/cppunittest/TestSegmentedVector; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestArray dist/test-stage/cppunittest/TestArray; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestDoublyLinkedList dist/test-stage/cppunittest/TestDoublyLinkedList; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestDefineEnum dist/test-stage/cppunittest/TestDefineEnum; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestWrappingOperations dist/test-stage/cppunittest/TestWrappingOperations; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestXorShift128PlusRNG dist/test-stage/cppunittest/TestXorShift128PlusRNG; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestRange dist/test-stage/cppunittest/TestRange; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestPoisonArea dist/test-stage/cppunittest/TestPoisonArea; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/ShowSSEConfig dist/test-stage/cppunittest/ShowSSEConfig; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestIntegerRange dist/test-stage/cppunittest/TestIntegerRange; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSaturate dist/test-stage/cppunittest/TestSaturate; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestIntegerPrintfMacros dist/test-stage/cppunittest/TestIntegerPrintfMacros; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestPrintf dist/test-stage/cppunittest/TestPrintf; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestRefPtr dist/test-stage/cppunittest/TestRefPtr; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestLinkedList dist/test-stage/cppunittest/TestLinkedList; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSplayTree dist/test-stage/cppunittest/TestSplayTree; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestEnumeratedArray dist/test-stage/cppunittest/TestEnumeratedArray; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestCeilingFloor dist/test-stage/cppunittest/TestCeilingFloor; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestSmallPointerArray dist/test-stage/cppunittest/TestSmallPointerArray; /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-objcopy --strip-unneeded dist/cppunittests/TestFastBernoulliTrial dist/test-stage/cppunittest/TestFastBernoulliTrial; /builds/worker/wor[taskcluster 2018-03-21 23:52:12.466Z] === Task Finished ===
[taskcluster 2018-03-21 23:52:12.466Z] Unsuccessful task run with exit code: -1 completed in 7202.129 seconds


More succinctly...

[task 2018-03-21T22:11:20.363Z] 22:11:20 ...
[taskcluster 2018-03-21 23:52:12.466Z] === Task Finished ===
[taskcluster 2018-03-21 23:52:12.466Z] Unsuccessful task run with exit code: -1 completed in 7202.129 seconds
Another example:

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169418192&lineNumber=31454-31459

[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182436 res/drawable-mdpi-v4/ic_pause_dark.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182604 res/drawable-mdpi-v4/ic_pause_light.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182784 res/drawable-mdpi-v4/ic_play_dark.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183076 res/drawable-mdpi-v4/ic_play_light.png (OK)
[task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183364 res/drawable-mdpi-v4/ic_speaker_dark.png (OK)[taskcluster 2018-03-21 13:56:00.630Z] === Task Finished ===
[taskcluster 2018-03-21 13:56:00.631Z] Unsuccessful task run with exit code: -1 completed in 7202.27 seconds
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169362207&lineNumber=18600-18606

[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/components/reflect'
[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  libtoolkit_components_reflect.a.desc
[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  rm -f libtoolkit_components_reflect.a
[task 2018-03-21T04:04:33.344Z] 04:04:33     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/components/reflect'
[task 2018-03-21T04:04:33.384Z] 04:04:33     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/components/backgroundhangmonitor'
[task 2018-03-21T04:04:33.391Z] 04:04:33     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_ackgroundhangmonitor0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DMOZ_ENABLE_BACKGROUND_HANG_MONITOR=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor -I/builds/worker/workspace/build/src/obj-firefox/toolkit/components/backgroundhangmonitor -I/builds/worker/workspace/build/src/caps -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worke[taskcluster 2018-03-21 05:45:34.476Z] === Task Finished ===
[taskcluster 2018-03-21 05:45:34.476Z] Unsuccessful task run with exit code: -1 completed in 7201.71 seconds
(In reply to Geoff Brown [:gbrown] from comment #1)
> Another example:
> 
> https://treeherder.mozilla.org/logviewer.
> html#?repo=autoland&job_id=169418192&lineNumber=31454-31459
> 
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182436
> res/drawable-mdpi-v4/ic_pause_dark.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182604
> res/drawable-mdpi-v4/ic_pause_light.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3182784
> res/drawable-mdpi-v4/ic_play_dark.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183076
> res/drawable-mdpi-v4/ic_play_light.png (OK)
> [task 2018-03-21T12:19:14.691Z] 12:19:14     INFO -   3183364
> res/drawable-mdpi-v4/ic_speaker_dark.png (OK)[taskcluster 2018-03-21
> 13:56:00.630Z] === Task Finished ===
> [taskcluster 2018-03-21 13:56:00.631Z] Unsuccessful task run with exit code:
> -1 completed in 7202.27 seconds

This one in particular seems like the worker just died.  That output is the middle of a packaging task; it's essentially infallible and neither CPU nor IO bound.

In the same way we see "cannot write to file", it looks we we see "cannot continue computation".  I see nothing to suggest this is Android specific, or even really specific to the build system.  It seems more likely to be an issue with the compute provider or the container/image/kernel combination.
Test failures continue in bug 1411358 but I rarely see such build failures now. And yeah, maybe the cause is not specific to Android builds.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX
Product: Firefox for Android → Firefox Build System
You need to log in before you can comment on or make changes to this bug.