1298258 - Firefox desktop l10n jobs fail in second configure call

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=01748a2b1a463f24efd9cd8abad9ccfd76b037b8&filter-searchStr=nightly&exclusion_profile=false

This is mac, but affecting all platforms in a similar way:
 06:39:18     INFO -   0:14.98 checking for posix_fadvise... (cached) no
 06:39:18     INFO -   0:14.98 checking for posix_fallocate... (cached) no
 06:39:18     INFO -   0:15.08 creating ./config.data
 06:39:19     INFO -   0:15.63 Exception in thread Thread-3:
 06:39:19     INFO -   0:15.63 Traceback (most recent call last):
 06:39:19     INFO -   0:15.63   File "/tools/python/lib/python2.7/threading.py", line 551, in __bootstrap_inner
 06:39:19     INFO -   0:15.63     self.run()
 06:39:19     INFO -   0:15.63   File "/tools/python/lib/python2.7/threading.py", line 504, in run
 06:39:19     INFO -   0:15.63     self.__target(*self.__args, **self.__kwargs)
 06:39:19     INFO -   0:15.63   File "/tools/python/lib/python2.7/multiprocessing/pool.py", line 353, in _handle_results
 06:39:19     INFO -   0:15.63     task = get()
 06:39:19     INFO -   0:15.63 TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
 06:39:19     INFO -   0:15.63
 
 command timed out: 1800 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_l10n.py', '--environment-config', 'single_locale/production.py', '--branch-config', 'single_locale/mozilla-central.py', '--platform-config', 'single_locale/macosx64.py', '--total-chunks', '8', '--this-chunk', '1', '--balrog-config', 'balrog/production.py'], attempting to kill
process killed by signal 9

This is a clobber build, which has run configure, grabbed the en-US build, updated to the revision of that, then attempted to configure again.

The previous nightly was OK, leaving a regression range of http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=bd7645928990&tochange=01748a2b1a46

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The next set of nightlies worked fine, rev was a551f534773c. ¯\_(ツ)_/¯

Comment 2

[Justin Wood (:Callek)]

To follow up here on the why...

* We use mozharness from the rev of nightly: `01748a2b1a46`
* The Job clones m-c from `default`, at time of job was: `7963ebdd52b9`
* We then use the `default` rev of m-c to pull all locales, and tooltool artifacts
* Then configure using this default rev <in order to get Makefile.in's generated to Makefile's>
* Then we use the build system to grab the english binary, and extract that and query for the rev that it was built with
* Finally we update the tree to that rev, and try to configure again (incase a configure was needed again)

Using https://archive.mozilla.org/pub/firefox/nightly/2016/08/2016-08-25-03-02-26-mozilla-central-l10n/mozilla-central-linux-l10n-nightly-1-unknown-bm77-build1-build2.txt.gz  (A linux chunk 1 of l10n repacks)

 < We pull mozharness from the rev of the nightly: `01748a2b1a46` and extract it to use>
...

06:29:22     INFO - repositories:  {'repo': 'https://hg.mozilla.org/mozilla-central', 'dest': 'mozilla-central', 'vcs': 'hgtool', 'revision': 'default'}]
...
06:29:25     INFO - Copy/paste: /builds/slave/m-cen-lx-l10n-ntly-1-000000000/build/tools/buildfarm/utils/hgtool.py -r default https://hg.mozilla.org/mozilla-central /builds/slave/m-cen-lx-l10n-ntly-1-000000000/build/mozilla-central
...
06:31:55     INFO -  command: hg parent --template {node|short}
06:31:55     INFO -  command: cwd: /builds/slave/m-cen-lx-l10n-ntly-1-000000000/build/mozilla-central
06:31:55     INFO -  command: env: {'HGPLAIN': '1'}
06:31:56     INFO -  command: END (0.50s elapsed)
06:31:56     INFO -  command: output:
06:31:56     INFO -  7963ebdd52b9
...
06:36:21     INFO -  Downloaded https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux/1472119346/firefox-51.0a1.en-US.linux-i686.tar.bz2 to /builds/slave/m-cen-lx-l10n-ntly-1-000000000/build/mozilla-central/obj-l10n/dist//firefox-51.0a1.en-US.linux-i686.tar.bz2
...
<< Extract that binary >>
...
06:36:30     INFO - Copy/paste: mock_mozilla ... make ident"
...
06:36:31     INFO - Copy/paste: mock_mozilla ... hg update -r 01748a2b1a463f24efd9cd8abad9ccfd76b037b8"
...
<Configure that failed>


So the regression range is a bit bigger than what was said here, (in that it includes `7963ebdd52b9`) specifically it is really the range of ``` 01748a2b1a46::7963ebdd52b9 ``` due to the first-configure-pass succeeding.

Of note, this range explicitly wanted us to CLOBBER:

callek@Centaurus:~/mozilla/hg/mozilla-central$ hg diff --stat -r 01748a2b1a46::7963ebdd52b9 | head
 CLOBBER   |     2 +-

The reason we didn't clobber with that file between configure runs is that we *explicitly* disabled that over in Bug 1283438.

The disable was needed otherwise we'd fail because the clobber would also purge the en-US build we have. I'm not sure if that bug should be backed out due to this even-more-cryptic type of failure.

Comment 3

[Axel Hecht [:Pike]]

There are many reasons for clobbers that don't affect configure itself, so I think we should keep the hack in for now.

I'd say that the real problem is

> * We use mozharness from the rev of nightly: `01748a2b1a46`
> * The Job clones m-c from `default`, at time of job was: `7963ebdd52b9`

^^^ this one.

Which reminds me about that one-line I had in that email about l10n builds to you, https://groups.google.com/d/msg/mozilla.dev.l10n/dOX2ckL5kUU/JwWr8nBiAwAJ.

mozharness probably shouldn't change the version of the repo. Even more so, for engineers that are tempted to use mozharness to test their changes to their tree, the fact that mozharness creates a new source tree is unhelpful to begin with?

In that sense I was asking about a clear scope of what mozharness should or shouldn't do.