800364 - Set up b2g unagi builds in production

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Some progress notes:
* got a unagi compile that finishes using a hp build slave and a mock environment, the call looked like this:
python scripts/scripts/b2g_build.py --target unagi   --repo http://hg.mozilla.org/mozilla-central   --gonk-snapshot file:///builds/slave/nthomas-unagi/B2G/B2G_default_unagi.tar.gz   --gecko-config $PWD/unagi.json

* I used these mock settings in an unagi.json:
    "mock_target": "mozilla-centos6-i386",
    "mock_packages": ["ccache", "make", "bison", "flex", "gcc", "g++", "mpfr", "zlib-devel", "ncurses-devel", "zip", "autoconf213", "glibc-static", "perl-Digest-SHA", "wget", "alsa-lib", "atk", "cairo", "dbus-glib", "fontconfig", "freetype", "glib2", "gtk2", "libXRender", "libXt", "pango"]
* everything from wget on is there to satisfy the runtime deps of xulrunner, in order to provide a gaia profile (can we switch to jsshell ?)

* catlee says we just need to call build.sh without args in the build action of mozharness/scripts/bg2_builds.py (different from the panda usage of this script). This means we don't get tarballs of the three partitions. We'll need to put a list of targets into the config and use that in the build action

* the make-updates action fails because build/objdir-gecko/dist/host/bin/mar doesn't get compiled, despite --enable-update-packaging' being present when gecko configures. Also, why do we initialize mock again in this action?

* tried to add a 'make buildsymbols' call (really ./build.sh buildsymbols) and it failed after processing libxul.so. Hacking in a print in:
02:22:05     INFO -  Processing file: ./dist/bin/libxul.so
02:22:49     INFO -  HGFileInfo: Going to read srcdir=/builds/slave/nthomas-unagi/build/gecko, file=/builds/slave/nthomas-unagi/build/gecko/modules/freetype2/src/autofit/afangles.c
02:22:49     INFO -  Unexpected error:  <type 'exceptions.OSError'>
02:22:49    ERROR -  Traceback (most recent call last):
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 792, in <module>
02:22:49     INFO -      main()
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 788, in main
02:22:49     INFO -      dumper.Process(arg)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 452, in Process
02:22:49     INFO -      return self.ProcessDir(file_or_dir)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 469, in ProcessDir
02:22:49     INFO -      if not self.ProcessFile(fullpath):
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 519, in ProcessFile
02:22:49     INFO -      (filename, rootname) = GetVCSFilename(filename, self.srcdirs)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 343, in GetVCSFilename
02:22:49     INFO -      fileInfo = HGFileInfo(file, srcdir)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 251, in __init__
02:22:49     INFO -      'parent', '--template={node|short}')
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 232, in read_output
02:22:49     INFO -      (stdout, _) = subprocess.Popen(args=args, stdout=subprocess.PIPE).communicate()
02:22:49     INFO -    File "/usr/lib/python2.6/subprocess.py", line 639, in __init__
02:22:49     INFO -      errread, errwrite)
02:22:49     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1228, in _execute_child
02:22:49     INFO -      raise child_exception
02:22:49     INFO -  OSError: [Errno 2] No such file or directory
02:22:49     INFO -  error writing symbol file: Broken pipe

That's a strange file to read.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The buildsymbols error is from having no hg in the mock environment. Easy fix.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozilla-central] unagi dep

Adds configs for unagi, based on your Oct 22 B2G snapshot, and panda changes for the build target and upload differences (used in the mozharness patch that follows). This also has the Panda README change.

I don't know why we need to set B2GUPDATER in the env for unagi but not for panda, but that's what creates dist/host/bin/mar in the former case.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] unagi dep

Drops make-updates as a default action, will add that back for unagi nightlies later. Otherwise this will look pretty familiar. In prep_upload I've ended up dropping the likes of 
 f = os.path.join(dirs['abs_upload_dir'], os.path.basename(f))
completely, it seems to be dangling in the middle of nowhere and have no effect.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] unagi dep

Provided these land in the order attached there should be no burning panda builds.

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Review of attachment 675454 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good overall with a few nits.

I'm not sure who 'owns' b2g/config - ted can you review this?

::: b2g/config/unagi/sources.xml
@@ +1,3 @@
> +<?xml version="1.0" ?><manifest>
> +  <!-- This is only a record of which revisions were pulled to generate the
> +       gonk.tar.xz snapshot referred to by releng-pandaboard.tt -->

s/pandaboard/unagi/

@@ +14,5 @@
> +    <copyfile dest="Makefile" src="core/root.mk"/>
> +  </project>
> +  <project name="fake-dalvik" path="dalvik" remote="b2g" revision="ca1f327d5acc198bb4be62fa51db2c039032c9ce"/>
> +  <project name="gaia" path="gaia" remote="b2g" revision="ed73042e6feceb003d0114d95f1d6cfb1b3a9794"/>
> +  <project name="releases-mozilla-central" path="gecko" remote="mozilla" revision="030e7a5dcfc0e8d4ec0c0227d121249366a9c0ec"/>

I think we should strip out this line, since we don't include gecko in the snapshot.

Comment 7

[(not currently active) Ted Mielczarek]

Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

I'm happy to rubber-stamp this, but I wouldn't claim ownership over anything under b2g/. Would you feel better with cjones or mwu stamping it?

Comment 8

[Chris AtLee [:catlee]]

Attachment: Add 'enable_nightly' support to mozharness builds

I have one concern about this. Elsewhere in misc.py we have logic that looks like:

if config['enable_nightly']:
    if 'enable_nightly' in pf:
       do_nightly = pf['enable_nightly']
    else:
       do_nightly = True
else:
    do_nightly = False

i.e., whether we do a nightly is a function of if the branch has nightlies enabled, and if the platform has them specifically enabled or disabled.

should we copy that behaviour for these b2g builds? The way I've implemented it now, we only obey the platform config.

Comment 9

[Chris AtLee [:catlee]]

Enabling nightlies for unagi would then be:
+BRANCHES['mozilla-central']['platforms']['unagi']['enable_nightly'] = True

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 675580 [details] [diff] [review]
Add 'enable_nightly' support to mozharness builds

Review of attachment 675580 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Chris AtLee [:catlee] from comment #8)
> Created attachment 675580 [details] [diff] [review]
> Add 'enable_nightly' support to mozharness builds
> 
> I have one concern about this. Elsewhere in misc.py we have logic that looks
> like:
> 
> if config['enable_nightly']:
>     if 'enable_nightly' in pf:
>        do_nightly = pf['enable_nightly']
>     else:
>        do_nightly = True
> else:
>     do_nightly = False
> 
> i.e., whether we do a nightly is a function of if the branch has nightlies
> enabled, and if the platform has them specifically enabled or disabled.
> 
> should we copy that behaviour for these b2g builds? The way I've implemented
> it now, we only obey the platform config.

It seems redundant to me to have a master switch at the branch level. I'd rather see that ripped out than copied here, frankly.

Comment 11

[Michael Wu [:mwu]]

Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Review of attachment 675454 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine to me as long as we're only using this to verify that gecko builds on unagi.

::: b2g/config/unagi/config.json
@@ +3,5 @@
> +    "mock_target": "mozilla-centos6-i386",
> +    "mock_packages": ["ccache", "make", "bison", "flex", "gcc", "g++", "mpfr", "zlib-devel", "ncurses-devel", "zip", "autoconf213", "glibc-static", "perl-Digest-SHA", "wget", "alsa-lib", "atk", "cairo", "dbus-glib", "fontconfig", "freetype", "glib2", "gtk2", "libXRender", "libXt", "pango", "mozilla-python27-mercurial", "openssh-clients"],
> +    "build_targets": [],
> +    "upload_files": [
> +        "{workdir}/out/target/product/unagi/*.tar.bz2",

These aren't generated without the tarball targets, right?

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Michael Wu [:mwu] from comment #11)
> Looks fine to me as long as we're only using this to verify that gecko
> builds on unagi.

That's the first step.
 
> > +    "upload_files": [
> > +        "{workdir}/out/target/product/unagi/*.tar.bz2",
> 
> These aren't generated without the tarball targets, right?

That's right. catlee says on IRC that we should be uploading the images, so I'll change this to *.img.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Landed with requested changes:
 https://hg.mozilla.org/mozilla-central/rev/e19e170d2f6d

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 675455 [details] [diff] [review]
[mozharness] unagi dep

Slightly modified on landing to use .get() to retrieve build_targets and upload_files from the gecko_config, to avoid bustage on branches that haven't synced the change from m-c. In that case panda builds won't build image tar.bz2's or upload much, but should stay green rather than hitting a python error.

http://hg.mozilla.org/build/mozharness/rev/7ec8685c7abe

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 675456 [details] [diff] [review]
[buildbot-configs] unagi dep

http://hg.mozilla.org/build/buildbot-configs/rev/6735bd4e1c26

Comment 16

[[e-mail] (:bhearsum)]

The buildbot-configs patch has been merged to production.

Comment 17

[Ed Morley [:emorley]]

(These are showing up on TBPL under "Other", pending bug 806669 - TBPL pushes rq

Comment 18

[Ed Morley [:emorley]]

These are showing up on TBPL under "Other", pending bug 806669 - TBPL pushes require IT intervention, so may be a few days more. For any future platform additions, I'm happy to add the TBPL support pre-emptively to avoid delays like this - just CC me on the relevant bugs early on :-)

Comment 19

[Ed Morley [:emorley]]

(In reply to Nick Thomas [:nthomas] from comment #14)
> Comment on attachment 675455 [details] [diff] [review]
> [mozharness] unagi dep
> 
> Slightly modified on landing to use .get() to retrieve build_targets and
> upload_files from the gecko_config, to avoid bustage on branches that
> haven't synced the change from m-c. In that case panda builds won't build
> image tar.bz2's or upload much, but should stay green rather than hitting a
> python error.
> 
> http://hg.mozilla.org/build/mozharness/rev/7ec8685c7abe

Unfortunately those builds are still failing on aurora:
https://tbpl.mozilla.org/php/getParsedLog.php?id=16587239&tree=Mozilla-Aurora

Comment 20

[Ed Morley [:emorley]]

{
08:12:55     INFO - #####
08:12:55     INFO - ##### Running download-gonk step.
08:12:55     INFO - #####
Traceback (most recent call last):
  File "scripts/scripts/b2g_build.py", line 337, in <module>
    myScript.run()
  File "/builds/slave/b2g-m-aurora-unagi-dep/scripts/mozharness/base/script.py", line 711, in run
    self._possibly_run_method(method_name, error_if_missing=True)
  File "/builds/slave/b2g-m-aurora-unagi-dep/scripts/mozharness/base/script.py", line 668, in _possibly_run_method
    return getattr(self, method_name)()
  File "scripts/scripts/b2g_build.py", line 183, in download_gonk
    gecko_config = self.load_gecko_config()
  File "scripts/scripts/b2g_build.py", line 129, in load_gecko_config
    self.gecko_config = json.load(open(conf_file))
IOError: [Errno 2] No such file or directory: 'build/gecko/b2g/config/unagi/config.json'
}

Comment 21

[Chris AtLee [:catlee]]

we'll need to uplift those changes to aurora to get the builds working

Comment 22

[Chris AtLee [:catlee]]

Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Bug caused by (feature/regressing bug #): this bug
User impact if declined: no unagi builds on aurora
Testing completed (on m-c, etc.): green on m-c
Risk to taking this patch (and alternatives if risky): none
String or UUID changes made by this patch: none

Comment 23

[Ryan VanderMeulen [:RyanVM]]

I've hidden unagi builds on m-c and inbound for the time being due to the frequent red.

Comment 24

[Chris AtLee [:catlee]]

(In reply to Ryan VanderMeulen from comment #23)
> I've hidden unagi builds on m-c and inbound for the time being due to the
> frequent red.

Are there bugs / logs for these?

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Thanks RyanVM. The error is

13:56:30     INFO -  run-js-command  webapp-zip
13:56:31     INFO -  build/utils.js:67: NS_ERROR_FILE_ALREADY_EXISTS: Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIFile.create]
13:56:31    ERROR -  make[1]: *** [webapp-zip] Error 3

fabrice says that he's seen this occasionally in his own builds, and suggests there is a race condition in the build system. I saw it occasionally in staging but not reproducibly. I'm figuring out how to file that against gaia, probably on github.

Comment 26

[Ed Morley [:emorley]]

> I'm figuring out how to file
> that against gaia, probably on github.

/me bangs head on desk... why can't all projects stick to at least filing in b.m.o, even if the repos are elsewhere :-(

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] clobber gaia/profile

This runs clean in staging, and is a no-op for panda when I had -v there too. Just speculative to fix the intermittent fail in production, as I can't reproduce that in staging.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris AtLee [:catlee] from comment #22)
> Comment on attachment 675454 [details] [diff] [review]
> [mozilla-central] unagi dep

FYI until this lands on aurora we'll have to manually trigger b2g nightlies. The burning unagi builds convince buildbot there's no good revision in the last 24 hours.

Comment 29

[Chris AtLee [:catlee]]

Still failing even after cleaning gaia profile:

https://tbpl.mozilla.org/php/getParsedLog.php?id=16658496&tree=Mozilla-Inbound&full=1

07:49:45     INFO -  build/utils.js:67: NS_ERROR_FILE_ALREADY_EXISTS: Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIFile.create]
07:49:45     INFO -  user_pref("browser.manifestURL", "app://system.gaiamobile.org/manifest.webapp");
07:49:45     INFO -  user_pref("browser.homescreenURL", "app://system.gaiamobile.org/index.html");
07:49:45     INFO -  user_pref("dom.send_after_paint_to_content", true);
07:49:45     INFO -  user_pref("network.http.max-connections-per-server", 15);
07:49:45     INFO -  user_pref("network.dns.localDomains", "gaiamobile.org,feedback.gaiamobile.org,clock.gaiamobile.org,sms.gaiamobile.org,camera.gaiamobile.org,homescreen.gaiamobile.org,keyboard.gaiamobile.org,costcontrol.gaiamobile.org,calendar.gaiamobile.org,pdfjs.gaiamobile.org,calculator.gaiamobile.org,gallery.gaiamobile.org,browser.gaiamobile.org,communications.gaiamobile.org,system.gaiamobile.org,video.gaiamobile.org,wallpaper.gaiamobile.org,email.gaiamobile.org,fm.gaiamobile.org,settings.gaiamobile.org,music.gaiamobile.org");
07:49:45    ERROR -  make[1]: *** [webapp-zip] Error 3

Comment 30

[Chris AtLee [:catlee]]

the profile is also deleted as part of the build process anyway:

07:49:44     INFO -  rm -rf /builds/slave/b2g-m-in-unagi-dep/build/gaia/profile /builds/slave/b2g-m-in-unagi-dep/build/gaia/profile.tar.gz

Comment 31

[Chris AtLee [:catlee]]

also fails in make webapp-manifests:

https://tbpl.mozilla.org/php/getParsedLog.php?id=16660028&tree=Mozilla-Inbound&full=1

08:19:28     INFO -  build/webapp-manifests.js:32: NS_ERROR_FILE_ALREADY_EXISTS: Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsILocalFile.copyTo]
08:19:28     INFO -  user_pref("browser.manifestURL", "app://system.gaiamobile.org/manifest.webapp");
08:19:28     INFO -  user_pref("browser.homescreenURL", "app://system.gaiamobile.org/index.html");
08:19:28     INFO -  user_pref("dom.send_after_paint_to_content", true);
08:19:28     INFO -  user_pref("network.http.max-connections-per-server", 15);
08:19:28     INFO -  user_pref("network.dns.localDomains", "gaiamobile.org,feedback.gaiamobile.org,clock.gaiamobile.org,sms.gaiamobile.org,camera.gaiamobile.org,homescreen.gaiamobile.org,keyboard.gaiamobile.org,costcontrol.gaiamobile.org,calendar.gaiamobile.org,pdfjs.gaiamobile.org,calculator.gaiamobile.org,gallery.gaiamobile.org,browser.gaiamobile.org,communications.gaiamobile.org,system.gaiamobile.org,video.gaiamobile.org,wallpaper.gaiamobile.org,email.gaiamobile.org,fm.gaiamobile.org,settings.gaiamobile.org,music.gaiamobile.org");
08:19:28    ERROR -  make[1]: *** [webapp-manifests] Error 3

Comment 32

[Chris AtLee [:catlee]]

I'm still unable to reproduce in staging, and clobbering the builders seems to have fixed the problem. Can we unhide these builders?

Comment 33

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #32)
> I'm still unable to reproduce in staging, and clobbering the builders seems
> to have fixed the problem. Can we unhide these builders?

Unagi builds unhidden on:
{m-c,inbound,try,fx-team,services-central,aurora}

(Aurora runs prior to 7814cae6099a landing have been starred, subsequent green).

-> resolved fixed?

Comment 34

[Chris AtLee [:catlee]]

Attachment: nightly build support, upload symbols

Comment 35

[Ryan VanderMeulen [:RyanVM]]

Still seeing occasional failures on inbound:
https://tbpl.mozilla.org/php/getParsedLog.php?id=16691567&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=16696620&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=16698615&tree=Mozilla-Inbound

Comment 36

[Daniel Holbert [:dholbert]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=16715374&tree=Mozilla-Inbound

Comment 37

[Chris AtLee [:catlee]]

Attachment: clone fresh gaia

get a fresh gaia each time

we'll use hg.m.o/projects/gaia for now

Comment 38

[Chris AtLee [:catlee]]

Attachment: update mock requirements for unagi/panda

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 678461 [details] [diff] [review]
update mock requirements for unagi/panda

*stamp*

Comment 40

[Chris AtLee [:catlee]]

Attachment: clone fresh gaia

Comment 41

[Kim Moir [:kmoir] ET]

In production

Comment 42

[Chris AtLee [:catlee]]

Attachment: update mock requirements for unagi/panda

Comment 43

[Chris AtLee [:catlee]]

Attachment: re-enable symbols

Comment 44

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 678749 [details] [diff] [review]
update mock requirements for unagi/panda

http://hg.mozilla.org/mozilla-central/rev/475a89bd44c3

Comment 45

[Chris AtLee [:catlee]]

Attachment: enable unagi nightlies on central/aurora

Comment 46

[Chris AtLee [:catlee]]

Attachment: enable unagi nightlies on central/aurora

Comment 47

[Rail Aliiev [:rail]]

in production

Comment 48

[Chris AtLee [:catlee]]

Comment on attachment 678749 [details] [diff] [review]
update mock requirements for unagi/panda

need this for releng unagi builds on aurora

Comment 49

[Chris AtLee [:catlee]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/39be69bb9956

Comment 50

[Chris AtLee [:catlee]]

green on central/aurora. let's open new bugs for any outstanding issues.