970918 - b2g build improvements

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

I'd like to clean up some of the things we do for b2g builds:
- Stop hacking .userconfig and use config.sh (depends on bug 965803)
- Stop using 'script' to run repo since git clones work in quiet/headless mode
- Retry repo sync on failure, and recover from various git bustages
- Fix update_source_manifests to work for all manifests (see bug 970601)

Comment 1

[Chris AtLee [:catlee]]

- checking out l10n repos should be parallelized

Comment 2

[Chris AtLee [:catlee]]

Attachment: build improvements

The primary improvement here is that we're using config.sh directly instead of writing .config ourselves.

Because of that and how we reference configs and names in b2g_config.py and gecko configs, I've updated the target munging in _pre_config_lock.

Since we're relying on config.sh to write .config, we now read various settings from it, like DEVICE, which indicates the internal target name. Most of the build output is in out/target/product/$DEVICE/...

I also added some extra calls to repo to cleanup and fix busted git clones. We can run in quiet mode now that bug 857158 is fixed. Failing in config.sh will cause a build failure (unlike right now!)

'make package-tests' isn't safe with -j >1 (see bug 984061), so force it to run with -j1. Otherwise we'll run with whatever config.sh specifies, which is usually -j10.

I also fixed the script so --build works alone if you haven't checked out l10n repos (use self.query_locales() instead of self.locales)

Comment 3

[Chris AtLee [:catlee]]

Attachment: build improvements

tweaks to before:
- update target name munging for emulator-kk
- do git reset/clean inside config.sh loop

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8393041 [details] [diff] [review]
build improvements

Pyflakes sez:
scripts/b2g_build.py:468: local variable 'dirs' is assigned to but never used

>-            # self.run_command([repo, "sync", "--quiet"], cwd=dirs['work_dir'], halt_on_failure=True)
>-            # XXX Workaround git failing to clone some repositories when
>-            # running in quiet mode. See bug 857158
>-            self.run_command(['script', '-q', '-c', '%s sync' % repo], cwd=dirs['work_dir'], halt_on_failure=True)
>+            max_tries = 5
>+            for _ in range(max_tries):

Guessing this is easier than self.retry() ?

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8393041 [details] [diff] [review]
build improvements

removed extra dirs variable

and yeah, I find a simple for loop easier than self.retry for simple things.

https://hg.mozilla.org/build/mozharness/rev/36f74e4dab7b

Comment 6

[Armen [:armenzg]]

Live in production.

Comment 7

[Chris AtLee [:catlee]]

Looks like it's sticking!

We've seen some make core dumps today from this change possibly. Maybe we can force build.sh to use -j <10. I'd like to get some memory profiles from machines hitting this first though.