Closed Bug 800364 Opened 7 years ago Closed 7 years ago

Set up b2g unagi builds in production

Categories

(Release Engineering :: General, defect, P2, major)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

Details

(Whiteboard: [b2g])

Attachments

(10 files, 3 obsolete files)

35.32 KB, patch
catlee
: review+
mwu
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
5.52 KB, patch
catlee
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
3.10 KB, patch
catlee
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
2.57 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.67 KB, patch
catlee
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
14.67 KB, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
9.44 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
32.25 KB, patch
rail
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
2.21 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.47 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
No description provided.
Some progress notes:
* got a unagi compile that finishes using a hp build slave and a mock environment, the call looked like this:
python scripts/scripts/b2g_build.py --target unagi   --repo http://hg.mozilla.org/mozilla-central   --gonk-snapshot file:///builds/slave/nthomas-unagi/B2G/B2G_default_unagi.tar.gz   --gecko-config $PWD/unagi.json

* I used these mock settings in an unagi.json:
    "mock_target": "mozilla-centos6-i386",
    "mock_packages": ["ccache", "make", "bison", "flex", "gcc", "g++", "mpfr", "zlib-devel", "ncurses-devel", "zip", "autoconf213", "glibc-static", "perl-Digest-SHA", "wget", "alsa-lib", "atk", "cairo", "dbus-glib", "fontconfig", "freetype", "glib2", "gtk2", "libXRender", "libXt", "pango"]
* everything from wget on is there to satisfy the runtime deps of xulrunner, in order to provide a gaia profile (can we switch to jsshell ?)

* catlee says we just need to call build.sh without args in the build action of mozharness/scripts/bg2_builds.py (different from the panda usage of this script). This means we don't get tarballs of the three partitions. We'll need to put a list of targets into the config and use that in the build action

* the make-updates action fails because build/objdir-gecko/dist/host/bin/mar doesn't get compiled, despite --enable-update-packaging' being present when gecko configures. Also, why do we initialize mock again in this action?

* tried to add a 'make buildsymbols' call (really ./build.sh buildsymbols) and it failed after processing libxul.so. Hacking in a print in:
02:22:05     INFO -  Processing file: ./dist/bin/libxul.so
02:22:49     INFO -  HGFileInfo: Going to read srcdir=/builds/slave/nthomas-unagi/build/gecko, file=/builds/slave/nthomas-unagi/build/gecko/modules/freetype2/src/autofit/afangles.c
02:22:49     INFO -  Unexpected error:  <type 'exceptions.OSError'>
02:22:49    ERROR -  Traceback (most recent call last):
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 792, in <module>
02:22:49     INFO -      main()
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 788, in main
02:22:49     INFO -      dumper.Process(arg)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 452, in Process
02:22:49     INFO -      return self.ProcessDir(file_or_dir)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 469, in ProcessDir
02:22:49     INFO -      if not self.ProcessFile(fullpath):
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 519, in ProcessFile
02:22:49     INFO -      (filename, rootname) = GetVCSFilename(filename, self.srcdirs)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 343, in GetVCSFilename
02:22:49     INFO -      fileInfo = HGFileInfo(file, srcdir)
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 251, in __init__
02:22:49     INFO -      'parent', '--template={node|short}')
02:22:49     INFO -    File "/builds/slave/nthomas-unagi/build/gecko/toolkit/crashreporter/tools/symbolstore.py", line 232, in read_output
02:22:49     INFO -      (stdout, _) = subprocess.Popen(args=args, stdout=subprocess.PIPE).communicate()
02:22:49     INFO -    File "/usr/lib/python2.6/subprocess.py", line 639, in __init__
02:22:49     INFO -      errread, errwrite)
02:22:49     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1228, in _execute_child
02:22:49     INFO -      raise child_exception
02:22:49     INFO -  OSError: [Errno 2] No such file or directory
02:22:49     INFO -  error writing symbol file: Broken pipe

That's a strange file to read.
The buildsymbols error is from having no hg in the mock environment. Easy fix.
Adds configs for unagi, based on your Oct 22 B2G snapshot, and panda changes for the build target and upload differences (used in the mozharness patch that follows). This also has the Panda README change.

I don't know why we need to set B2GUPDATER in the env for unagi but not for panda, but that's what creates dist/host/bin/mar in the former case.
Attachment #675454 - Flags: review?(catlee)
Drops make-updates as a default action, will add that back for unagi nightlies later. Otherwise this will look pretty familiar. In prep_upload I've ended up dropping the likes of 
 f = os.path.join(dirs['abs_upload_dir'], os.path.basename(f))
completely, it seems to be dangling in the middle of nowhere and have no effect.
Attachment #675455 - Flags: review?(catlee)
Provided these land in the order attached there should be no burning panda builds.
Attachment #675456 - Flags: review?(catlee)
Attachment #675455 - Flags: review?(catlee) → review+
Attachment #675456 - Flags: review?(catlee) → review+
Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Review of attachment 675454 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good overall with a few nits.

I'm not sure who 'owns' b2g/config - ted can you review this?

::: b2g/config/unagi/sources.xml
@@ +1,3 @@
> +<?xml version="1.0" ?><manifest>
> +  <!-- This is only a record of which revisions were pulled to generate the
> +       gonk.tar.xz snapshot referred to by releng-pandaboard.tt -->

s/pandaboard/unagi/

@@ +14,5 @@
> +    <copyfile dest="Makefile" src="core/root.mk"/>
> +  </project>
> +  <project name="fake-dalvik" path="dalvik" remote="b2g" revision="ca1f327d5acc198bb4be62fa51db2c039032c9ce"/>
> +  <project name="gaia" path="gaia" remote="b2g" revision="ed73042e6feceb003d0114d95f1d6cfb1b3a9794"/>
> +  <project name="releases-mozilla-central" path="gecko" remote="mozilla" revision="030e7a5dcfc0e8d4ec0c0227d121249366a9c0ec"/>

I think we should strip out this line, since we don't include gecko in the snapshot.
Attachment #675454 - Flags: review?(ted)
Attachment #675454 - Flags: review?(catlee) → review+
Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

I'm happy to rubber-stamp this, but I wouldn't claim ownership over anything under b2g/. Would you feel better with cjones or mwu stamping it?
I have one concern about this. Elsewhere in misc.py we have logic that looks like:

if config['enable_nightly']:
    if 'enable_nightly' in pf:
       do_nightly = pf['enable_nightly']
    else:
       do_nightly = True
else:
    do_nightly = False

i.e., whether we do a nightly is a function of if the branch has nightlies enabled, and if the platform has them specifically enabled or disabled.

should we copy that behaviour for these b2g builds? The way I've implemented it now, we only obey the platform config.
Attachment #675580 - Flags: review?(bhearsum)
Enabling nightlies for unagi would then be:
+BRANCHES['mozilla-central']['platforms']['unagi']['enable_nightly'] = True
Attachment #675454 - Flags: review?(ted) → review?(mwu)
Comment on attachment 675580 [details] [diff] [review]
Add 'enable_nightly' support to mozharness builds

Review of attachment 675580 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Chris AtLee [:catlee] from comment #8)
> Created attachment 675580 [details] [diff] [review]
> Add 'enable_nightly' support to mozharness builds
> 
> I have one concern about this. Elsewhere in misc.py we have logic that looks
> like:
> 
> if config['enable_nightly']:
>     if 'enable_nightly' in pf:
>        do_nightly = pf['enable_nightly']
>     else:
>        do_nightly = True
> else:
>     do_nightly = False
> 
> i.e., whether we do a nightly is a function of if the branch has nightlies
> enabled, and if the platform has them specifically enabled or disabled.
> 
> should we copy that behaviour for these b2g builds? The way I've implemented
> it now, we only obey the platform config.

It seems redundant to me to have a master switch at the branch level. I'd rather see that ripped out than copied here, frankly.
Attachment #675580 - Flags: review?(bhearsum) → review+
Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Review of attachment 675454 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine to me as long as we're only using this to verify that gecko builds on unagi.

::: b2g/config/unagi/config.json
@@ +3,5 @@
> +    "mock_target": "mozilla-centos6-i386",
> +    "mock_packages": ["ccache", "make", "bison", "flex", "gcc", "g++", "mpfr", "zlib-devel", "ncurses-devel", "zip", "autoconf213", "glibc-static", "perl-Digest-SHA", "wget", "alsa-lib", "atk", "cairo", "dbus-glib", "fontconfig", "freetype", "glib2", "gtk2", "libXRender", "libXt", "pango", "mozilla-python27-mercurial", "openssh-clients"],
> +    "build_targets": [],
> +    "upload_files": [
> +        "{workdir}/out/target/product/unagi/*.tar.bz2",

These aren't generated without the tarball targets, right?
Attachment #675454 - Flags: review?(mwu) → review+
(In reply to Michael Wu [:mwu] from comment #11)
> Looks fine to me as long as we're only using this to verify that gecko
> builds on unagi.

That's the first step.
 
> > +    "upload_files": [
> > +        "{workdir}/out/target/product/unagi/*.tar.bz2",
> 
> These aren't generated without the tarball targets, right?

That's right. catlee says on IRC that we should be uploading the images, so I'll change this to *.img.
Depends on: 806669
Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Landed with requested changes:
 https://hg.mozilla.org/mozilla-central/rev/e19e170d2f6d
Attachment #675454 - Flags: checked-in+
Comment on attachment 675455 [details] [diff] [review]
[mozharness] unagi dep

Slightly modified on landing to use .get() to retrieve build_targets and upload_files from the gecko_config, to avoid bustage on branches that haven't synced the change from m-c. In that case panda builds won't build image tar.bz2's or upload much, but should stay green rather than hitting a python error.

http://hg.mozilla.org/build/mozharness/rev/7ec8685c7abe
Attachment #675455 - Flags: checked-in+
The buildbot-configs patch has been merged to production.
(These are showing up on TBPL under "Other", pending bug 806669 - TBPL pushes rq
These are showing up on TBPL under "Other", pending bug 806669 - TBPL pushes require IT intervention, so may be a few days more. For any future platform additions, I'm happy to add the TBPL support pre-emptively to avoid delays like this - just CC me on the relevant bugs early on :-)
(In reply to Nick Thomas [:nthomas] from comment #14)
> Comment on attachment 675455 [details] [diff] [review]
> [mozharness] unagi dep
> 
> Slightly modified on landing to use .get() to retrieve build_targets and
> upload_files from the gecko_config, to avoid bustage on branches that
> haven't synced the change from m-c. In that case panda builds won't build
> image tar.bz2's or upload much, but should stay green rather than hitting a
> python error.
> 
> http://hg.mozilla.org/build/mozharness/rev/7ec8685c7abe

Unfortunately those builds are still failing on aurora:
https://tbpl.mozilla.org/php/getParsedLog.php?id=16587239&tree=Mozilla-Aurora
{
08:12:55     INFO - #####
08:12:55     INFO - ##### Running download-gonk step.
08:12:55     INFO - #####
Traceback (most recent call last):
  File "scripts/scripts/b2g_build.py", line 337, in <module>
    myScript.run()
  File "/builds/slave/b2g-m-aurora-unagi-dep/scripts/mozharness/base/script.py", line 711, in run
    self._possibly_run_method(method_name, error_if_missing=True)
  File "/builds/slave/b2g-m-aurora-unagi-dep/scripts/mozharness/base/script.py", line 668, in _possibly_run_method
    return getattr(self, method_name)()
  File "scripts/scripts/b2g_build.py", line 183, in download_gonk
    gecko_config = self.load_gecko_config()
  File "scripts/scripts/b2g_build.py", line 129, in load_gecko_config
    self.gecko_config = json.load(open(conf_file))
IOError: [Errno 2] No such file or directory: 'build/gecko/b2g/config/unagi/config.json'
}
we'll need to uplift those changes to aurora to get the builds working
Comment on attachment 675454 [details] [diff] [review]
[mozilla-central] unagi dep

Bug caused by (feature/regressing bug #): this bug
User impact if declined: no unagi builds on aurora
Testing completed (on m-c, etc.): green on m-c
Risk to taking this patch (and alternatives if risky): none
String or UUID changes made by this patch: none
Attachment #675454 - Flags: approval-mozilla-aurora?
I've hidden unagi builds on m-c and inbound for the time being due to the frequent red.
(In reply to Ryan VanderMeulen from comment #23)
> I've hidden unagi builds on m-c and inbound for the time being due to the
> frequent red.

Are there bugs / logs for these?
Thanks RyanVM. The error is

13:56:30     INFO -  run-js-command  webapp-zip
13:56:31     INFO -  build/utils.js:67: NS_ERROR_FILE_ALREADY_EXISTS: Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIFile.create]
13:56:31    ERROR -  make[1]: *** [webapp-zip] Error 3

fabrice says that he's seen this occasionally in his own builds, and suggests there is a race condition in the build system. I saw it occasionally in staging but not reproducibly. I'm figuring out how to file that against gaia, probably on github.
> I'm figuring out how to file
> that against gaia, probably on github.

/me bangs head on desk... why can't all projects stick to at least filing in b.m.o, even if the repos are elsewhere :-(
This runs clean in staging, and is a no-op for panda when I had -v there too. Just speculative to fix the intermittent fail in production, as I can't reproduce that in staging.
Assignee: catlee → nthomas
Status: NEW → ASSIGNED
Attachment #677266 - Flags: review?(catlee)
(In reply to Chris AtLee [:catlee] from comment #22)
> Comment on attachment 675454 [details] [diff] [review]
> [mozilla-central] unagi dep

FYI until this lands on aurora we'll have to manually trigger b2g nightlies. The burning unagi builds convince buildbot there's no good revision in the last 24 hours.
Attachment #677266 - Flags: review?(catlee) → review+
Attachment #677266 - Flags: checked-in+
Attachment #675454 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Still failing even after cleaning gaia profile:

https://tbpl.mozilla.org/php/getParsedLog.php?id=16658496&tree=Mozilla-Inbound&full=1

07:49:45     INFO -  build/utils.js:67: NS_ERROR_FILE_ALREADY_EXISTS: Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsIFile.create]
07:49:45     INFO -  user_pref("browser.manifestURL", "app://system.gaiamobile.org/manifest.webapp");
07:49:45     INFO -  user_pref("browser.homescreenURL", "app://system.gaiamobile.org/index.html");
07:49:45     INFO -  user_pref("dom.send_after_paint_to_content", true);
07:49:45     INFO -  user_pref("network.http.max-connections-per-server", 15);
07:49:45     INFO -  user_pref("network.dns.localDomains", "gaiamobile.org,feedback.gaiamobile.org,clock.gaiamobile.org,sms.gaiamobile.org,camera.gaiamobile.org,homescreen.gaiamobile.org,keyboard.gaiamobile.org,costcontrol.gaiamobile.org,calendar.gaiamobile.org,pdfjs.gaiamobile.org,calculator.gaiamobile.org,gallery.gaiamobile.org,browser.gaiamobile.org,communications.gaiamobile.org,system.gaiamobile.org,video.gaiamobile.org,wallpaper.gaiamobile.org,email.gaiamobile.org,fm.gaiamobile.org,settings.gaiamobile.org,music.gaiamobile.org");
07:49:45    ERROR -  make[1]: *** [webapp-zip] Error 3
the profile is also deleted as part of the build process anyway:

07:49:44     INFO -  rm -rf /builds/slave/b2g-m-in-unagi-dep/build/gaia/profile /builds/slave/b2g-m-in-unagi-dep/build/gaia/profile.tar.gz
also fails in make webapp-manifests:

https://tbpl.mozilla.org/php/getParsedLog.php?id=16660028&tree=Mozilla-Inbound&full=1

08:19:28     INFO -  build/webapp-manifests.js:32: NS_ERROR_FILE_ALREADY_EXISTS: Component returned failure code: 0x80520008 (NS_ERROR_FILE_ALREADY_EXISTS) [nsILocalFile.copyTo]
08:19:28     INFO -  user_pref("browser.manifestURL", "app://system.gaiamobile.org/manifest.webapp");
08:19:28     INFO -  user_pref("browser.homescreenURL", "app://system.gaiamobile.org/index.html");
08:19:28     INFO -  user_pref("dom.send_after_paint_to_content", true);
08:19:28     INFO -  user_pref("network.http.max-connections-per-server", 15);
08:19:28     INFO -  user_pref("network.dns.localDomains", "gaiamobile.org,feedback.gaiamobile.org,clock.gaiamobile.org,sms.gaiamobile.org,camera.gaiamobile.org,homescreen.gaiamobile.org,keyboard.gaiamobile.org,costcontrol.gaiamobile.org,calendar.gaiamobile.org,pdfjs.gaiamobile.org,calculator.gaiamobile.org,gallery.gaiamobile.org,browser.gaiamobile.org,communications.gaiamobile.org,system.gaiamobile.org,video.gaiamobile.org,wallpaper.gaiamobile.org,email.gaiamobile.org,fm.gaiamobile.org,settings.gaiamobile.org,music.gaiamobile.org");
08:19:28    ERROR -  make[1]: *** [webapp-manifests] Error 3
I'm still unable to reproduce in staging, and clobbering the builders seems to have fixed the problem. Can we unhide these builders?
(In reply to Chris AtLee [:catlee] from comment #32)
> I'm still unable to reproduce in staging, and clobbering the builders seems
> to have fixed the problem. Can we unhide these builders?

Unagi builds unhidden on:
{m-c,inbound,try,fx-team,services-central,aurora}

(Aurora runs prior to 7814cae6099a landing have been starred, subsequent green).

-> resolved fixed?
Attachment #677869 - Flags: review?(aki)
Attachment #675580 - Flags: checked-in+
Attachment #677869 - Flags: review?(aki) → review+
Attachment #677869 - Flags: checked-in+
Attached patch clone fresh gaia (obsolete) — Splinter Review
get a fresh gaia each time

we'll use hg.m.o/projects/gaia for now
Attachment #678014 - Flags: review?(aki)
Attachment #678014 - Flags: review?(aki) → review+
Depends on: 808513
Attachment #678461 - Flags: review?(nthomas)
Comment on attachment 678461 [details] [diff] [review]
update mock requirements for unagi/panda

*stamp*
Attachment #678461 - Flags: review?(nthomas) → review+
Attached patch clone fresh gaiaSplinter Review
Attachment #678014 - Attachment is obsolete: true
Attachment #678536 - Flags: review?(nthomas)
In production
Attachment #678536 - Flags: review?(nthomas) → review+
Attachment #678536 - Flags: checked-in+
Attachment #678461 - Attachment is obsolete: true
Attachment #678749 - Flags: review?(rail)
Attachment #678749 - Flags: review?(rail) → review+
Attachment #678923 - Flags: review?(nthomas)
Attachment #678923 - Flags: review?(nthomas) → review+
Assignee: nthomas → catlee
Attachment #678923 - Flags: checked-in+
Attachment #678996 - Flags: review?(nthomas)
Attachment #678996 - Attachment is obsolete: true
Attachment #678996 - Flags: review?(nthomas)
Attachment #679006 - Flags: review?(nthomas)
Attachment #679006 - Flags: review?(nthomas) → review+
Attachment #679006 - Flags: checked-in+
Depends on: 809436
Comment on attachment 678749 [details] [diff] [review]
update mock requirements for unagi/panda

need this for releng unagi builds on aurora
Attachment #678749 - Flags: approval-mozilla-aurora?
Attachment #678749 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
green on central/aurora. let's open new bugs for any outstanding issues.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Duplicate of this bug: 805674
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.