1143013 - Ensure b2g_build passes number of cores to commands to workaround any previous -j settings...

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[James Lal [:lightsofapollo]]

I found some issues while debugging performance in taskcluster that revealed -j was set in .userconfig after running some commands and -j could happen to be 1 (due to some of the conditionals) once this is set and cached the b2g build system will not attempt to detect cores and your stuck building with 1 core. The workaround is to pass -j every time.

Comment 1

[James Lal [:lightsofapollo]]

Attachment: Pass number of cores via -j

Comment 2

[James Lal [:lightsofapollo]]

https://hg.mozilla.org/build/mozharness/rev/bc2b3ab7c9c6

Comment 3

[[e-mail] (:bhearsum)]

This busted things:
11:56:20     INFO - Running pre-action listener: influxdb_recording_pre_action
11:56:20     INFO - Running main action method: build
11:56:20     INFO - Running post-action listener: influxdb_recording_post_action
11:56:20    FATAL - Uncaught exception: Traceback (most recent call last):
11:56:20    FATAL -   File "/tools/checkouts/mozharness/mozharness/base/script.py", line 1278, in run
11:56:20    FATAL -     self.run_action(action)
11:56:20    FATAL -   File "/tools/checkouts/mozharness/mozharness/base/script.py", line 1220, in run_action
11:56:20    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
11:56:20    FATAL -   File "/tools/checkouts/mozharness/mozharness/base/script.py", line 1161, in _possibly_run_method
11:56:20    FATAL -     return getattr(self, method_name)()
11:56:20    FATAL -   File "/tools/checkouts/mozharness/scripts/b2g_build.py", line 564, in build
11:56:20    FATAL -     cmds = [self.generate_build_command(t) for t in build_targets]
11:56:20    FATAL -   File "/tools/checkouts/mozharness/scripts/b2g_build.py", line 553, in generate_build_command
11:56:20    FATAL -     cmd.append('-j{}'.format(multiprocessing.cpu_count()))
11:56:20    FATAL - ValueError: zero length field name in format

I backed it out because it wasn't obvious to be me what the fix was, and the tree was burning.

Comment 4

[[e-mail] (:bhearsum)]

In production: https://hg.mozilla.org/build/mozharness/rev/bc2b3ab7c9c6

Comment 5

[James Lal [:lightsofapollo]]

Attachment: Attempt #2 which should work on python2.6

r? to bhearsum who tested this on the builders for me (thanks !!!!)

Comment 6

[James Lal [:lightsofapollo]]

Comment on attachment 8577422 [details]
Attempt #2 which should work on python2.6

 import sys
 import os
 import glob
 import re
 import tempfile
 from datetime import datetime
 import urlparse
+import multiprocessing
 import xml.dom.minidom
 
 try:
     import simplejson as json
     assert json
 except ImportError:
     import json
 
@@ -540,16 +541,21 @@
         self.run_command(["diff", "-u", sourcesfile_orig, sourcesfile], success_codes=[1])
 
     def generate_build_command(self, target=None):
         cmd = ['./build.sh']
         if target is not None:
             # Workaround bug 984061
             if target == 'package-tests':
                 cmd.append('-j1')
+            else:
+                # Ensure we always utilize the correct number of cores
+                # regardless of the configuration which may be set by repo
+                # config changes...
+                cmd.append('-j{0}'.format(multiprocessing.cpu_count()))
             cmd.append(target)
         return cmd
 
     def build(self):
         dirs = self.query_abs_dirs()
         gecko_config = self.load_gecko_config()
         build_targets = gecko_config.get('build_targets', [])
         if not build_targets:

Comment 7

[James Lal [:lightsofapollo]]

Attachment: Python 2.6 friendly version

Comment 8

[James Lal [:lightsofapollo]]

Relanded with 2.7 / 2.7 support 

https://hg.mozilla.org/build/mozharness/rev/15d338b42f69