1380844 - Mac build: command timed out: 10800 seconds without output running ['make', '-f', 'client.mk', 'build', u'MOZ_BUILD_DATE=20170715030203'], attempting to kill

Status: RESOLVED FIXED

(Thunderbird :: Build Config, enhancement)

Description

[Jorg K (CEST = GMT+2)]

Started a few days ago:

command timed out: 10800 seconds without output running ['make', '-f', 'client.mk', 'build', u'MOZ_BUILD_DATE=20170713101319'], attempting to kill
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=a6a1279292d055c75b9554b01dab78b1cd9f27de

Right now I'm seeing:
make[5]: *** [force-cargo-library-build] Error 101
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=3a4c51fcb4aa98885607679be7c4c7b5082641d4

Test also failed with:
Unable to set installer_url from the buildbot config!

The last two are strange since there we no M-C landings since the last push:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=b61e73be467cef8a7231c42c0aba1a82ddfffada

But then, that failed on the timeout (first problem), so we don't know what the test would have done.

Practically we lost one platform.

Comment 1

[Tom Prince [:tomprince]]

> make[5]: *** [force-cargo-library-build] Error 101

This is due to a broken slave, I'll file a bug with releng for it later.

> Unable to set installer_url from the buildbot config!

This is in the neighbourhood of Bug 1380236.

Comment 2

[Jorg K (CEST = GMT+2)]

(In reply to Tom Prince [:tomprince] from comment #1)
> > make[5]: *** [force-cargo-library-build] Error 101
> This is due to a broken slave, I'll file a bug with releng for it later.
Yes, just saw bug 1380166.

> > Unable to set installer_url from the buildbot config!
> This is in the neighbourhood of Bug 1380236.
Hmm, so another bug/patch required.

Comment 3

[Jorg K (CEST = GMT+2)]

So with "make[5]: *** [force-cargo-library-build] Error 101" being due to a broken machine (bug 922783) and "Unable to set installer_url from the buildbot config!" being addressed by another patch in bug 1380236, we're left with the timeout issue, right?

command timed out: 10800 seconds without output running ['make', '-f', 'client.mk', 'build', u'MOZ_BUILD_DATE=20170715030203'], attempting to kill

Comment 4

[Jorg K (CEST = GMT+2)]

We currently don't have Mac builds, although every once in a while, one succeeds.

I've looked at a log:
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-macosx64-debug/1500362422/comm-central-macosx64-debug-bm84-build1-build31.txt.gz
and the compile is working away and in the end gets killed since it took over 4 hours. So if we gave it more time, it would succeed.

Finished processing /builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/js/src/jsapi-tests/jsapi-tests in 24.14s
Running: /builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/_virtualenv/bin/python /builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/toolkit/crashreporter/tools/symbolstore.py -c -a x86_64 --vcs-info --install-manifest=/builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/_build_manifests/install/dist_include,/builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/dist/include -s /builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla /builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/dist/host/bin/dump_syms /builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/dist/crashreporter-symbols /builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/js/src/jsapi-tests/jsapi-tests

command timed out: 10800 seconds without output running ['make', '-f', 'client.mk', 'build', u'MOZ_BUILD_DATE=20170718002022'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=14231.618353
========= master_lag: 0.82 =========
========= Finished compile failed (results: 2, elapsed: 3 hrs, 57 mins, 12 secs) (at 2017-07-18 04:19:24.438044) =========

Any idea why this is to terribly slow now? Mac compiles used to be the fastest with under 30 minutes.

Comment 5

[Tom Prince [:tomprince]]

I just realized that the build is parallel, and so the build is probably hanging in part of the build that isn't shown near the end of the logs. I've got 3 succesful pushes to try disabling stylo here: 

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&author=[e-mail]&filter-searchStr=os%20x&fromchange=23768de55f83a0689a7e58a04af27ca6133e433d&tochange=b12845e07ec032db2ed689c9762972e0390e0c33

This suggests to me that it is actually stylo causing the issues, so I've disabled the build again.

I'm wondering if the rust build is causing swapping death or something.

Comment 6

[Tom Prince [:tomprince]]

This appears to be fixed by https://hg.mozilla.org/comm-central/pushloghtml?changeset=db4917476acb

Comment 7

[Intermittent Failures Robot]

11 failures in 822 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* comm-central: 8
* try-comm-central: 3

Platform breakdown:
* osx-10-7: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1380844&startday=2017-07-17&endday=2017-07-23&tree=all