819058 - Upload manifests to ftp.m.o for unagi & otoro

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We need to upload source code manifests for our builds to
  https://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/

They should be based on the sources.xml we have in snapshot, and add a comment for the the gecko and gaia we pull from hg.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This is just for the nightly builds on the mozilla-beta repo, so the configs are
http://hg.mozilla.org/build/mozharness/file/default/configs/b2g/releng-beta.py
and releng-beta-stable.py.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] wip

This is mostly finished. Notes:
* probably makes sense to do update-sources-xml for all builds. The sources.xml.original might look a bit hacky but it saves a minute compared to blindly extracting sources.xml from gonk every job.
* The
  if not self.query_is_nightly() or self.buildbot_config['properties'] is not 'beta':
at the top of upload_manifest() isn't great. Probably better to have a new commmand line arg for b2g_build.py and set that only on mozilla-beta using b2g_config, then just test for being a nightly.
* the optimize.py script that jgriffin runs pretty quickly, so the chance of races is with his builds are hopefully small

There are some test builds running/pending at 
http://dev-master01.build.scl1.mozilla.com:8710/builders/b2g_mozilla-beta_unagi_nightly/builds/20
http://dev-master01.build.scl1.mozilla.com:8710/builders/b2g_mozilla-beta_otoro_nightly/builds/1
http://dev-master01.build.scl1.mozilla.com:8710/builders/b2g_mozilla-beta_unagi_stable_nightly/builds/0
http://dev-master01.build.scl1.mozilla.com:8710/builders/b2g_mozilla-central_unagi_nightly/builds/0
First three should upload the xml, last shouldn't.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Obviously upload_remote_host needs to change to stage.m.o for production.

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 689631 [details] [diff] [review]
[mozharness] wip

Review of attachment 689631 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/b2g/releng-beta-stable.py
@@ +43,5 @@
> +        "upload_remote_host": "dev-stage01.srv.releng.scl3.mozilla.com",
> +        "upload_remote_basepath": "/pub/mozilla.org/b2g/manifests",
> +        "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"),
> +        "ssh_user": "b2gbld",
> +        "target_modifier": "-stable",

use target_suffix like we do elsewhere?

::: scripts/b2g_build.py
@@ +276,2 @@
>          self.run_command(["cat", "sources.xml"], cwd=dirs['work_dir'])
> +        self.run_command(["cp", "-p", "sources.xml", "sources.xml.original"], cwd=dirs['work_dir'])

we could run this block higher up? if sources.xml.original doesn't exist, or is newer than sources.xml? that would save the transition code below in update_sources_xml.

@@ +700,5 @@
>  
> +    def upload_manifest(self):
> +        if not self.query_is_nightly() or self.buildbot_config['properties'] is not 'beta':
> +            self.info("Not a nightly build on beta. Skipping...")
> +            return

instead of using the 'beta' branch as a trigger, maybe check the presence of self.config['manifest']?

@@ +753,5 @@
> +               '-l', self.config['manifest']['ssh_user'],
> +               '-i', self.config['manifest']['ssh_key'],
> +               self.config['upload_remote_host'],
> +               'python ~/organize.py --directory %s' % self.config['manifest']['upload_remote_basepath'],
> +               ]

the new post_upload.py?

do we need to run this? can we upload to the right place directly?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris AtLee [:catlee] from comment #4)
> use target_suffix like we do elsewhere?

Ok, good plan.
 
> ::: scripts/b2g_build.py
> @@ +276,2 @@
> >          self.run_command(["cat", "sources.xml"], cwd=dirs['work_dir'])
> > +        self.run_command(["cp", "-p", "sources.xml", "sources.xml.original"], cwd=dirs['work_dir'])
> 
> we could run this block higher up? if sources.xml.original doesn't exist, or
> is newer than sources.xml? that would save the transition code below in
> update_sources_xml.

I'm not sure where you mean, perhaps inside the test on mtime_file ? The transition code only needs to live as long as we keep build directories, which should be 7 days if we did the same as desktop.

> instead of using the 'beta' branch as a trigger, maybe check the presence of
> self.config['manifest']?

I did that originally, but to get otoro uploading manifests we need the addition of the upload-manifest action to releng.py, but then we'd get manifests from many branches. We could modify lines like this in b2g_config.py instead
 BRANCHES['mozilla-beta']['platforms']['unagi']['mozharness_config']['extra_args'] = ['--target', 'unagi', '--config', 'b2g/releng-beta.py', '--gaia-languages-file', 'shared/resources/languages-dev.json']
if there was a new parameter --upload-manifest.
 
> the new post_upload.py?
> 
> do we need to run this? can we upload to the right place directly?

We could, but that requires a bunch of extra logic to created dated directories and manage symlinks, and I was concerned that our systems would race with jgriffins when the emulator manifest is uploaded. By utilizing the existing script we should avoid all of that, at least for now.

Comment 6

[Chris AtLee [:catlee]]

(In reply to Nick Thomas [:nthomas] from comment #5)
> (In reply to Chris AtLee [:catlee] from comment #4)
> > use target_suffix like we do elsewhere?
> 
> Ok, good plan.
>  
> > ::: scripts/b2g_build.py
> > @@ +276,2 @@
> > >          self.run_command(["cat", "sources.xml"], cwd=dirs['work_dir'])
> > > +        self.run_command(["cp", "-p", "sources.xml", "sources.xml.original"], cwd=dirs['work_dir'])
> > 
> > we could run this block higher up? if sources.xml.original doesn't exist, or
> > is newer than sources.xml? that would save the transition code below in
> > update_sources_xml.
> 
> I'm not sure where you mean, perhaps inside the test on mtime_file ? The
> transition code only needs to live as long as we keep build directories,
> which should be 7 days if we did the same as desktop.

The transition code below is coping with the fact that we aren't going to hit this block because we bail out early if we're not unpacking gonk.tar.xz. To me it's simpler to make sure that sources.xml.original always exists, regardless of if we unpack gonk or not. Are you worried that we don't know the state of sources.xml in this case?

> > instead of using the 'beta' branch as a trigger, maybe check the presence of
> > self.config['manifest']?
> 
> I did that originally, but to get otoro uploading manifests we need the
> addition of the upload-manifest action to releng.py, but then we'd get
> manifests from many branches. We could modify lines like this in
> b2g_config.py instead
>  BRANCHES['mozilla-
> beta']['platforms']['unagi']['mozharness_config']['extra_args'] =
> ['--target', 'unagi', '--config', 'b2g/releng-beta.py',
> '--gaia-languages-file', 'shared/resources/languages-dev.json']
> if there was a new parameter --upload-manifest.

The 'manifest' entry in self.config is only set in releng-beta.py and releng-beta-stable.py though, right?

> > the new post_upload.py?
> > 
> > do we need to run this? can we upload to the right place directly?
> 
> We could, but that requires a bunch of extra logic to created dated
> directories and manage symlinks, and I was concerned that our systems would
> race with jgriffins when the emulator manifest is uploaded. By utilizing the
> existing script we should avoid all of that, at least for now.

Ok, sounds good.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris AtLee [:catlee] from comment #6)
> The transition code below is coping with the fact that we aren't going to
> hit this block because we bail out early if we're not unpacking gonk.tar.xz.
> To me it's simpler to make sure that sources.xml.original always exists,
> regardless of if we unpack gonk or not. Are you worried that we don't know
> the state of sources.xml in this case?

The 'logic' was:
* it's more reliable to tweak a copy of sources.xml that hasn't already been tweaked
* avoid the 60 seconds it takes to extract from gonk.tar.xz on every job, by saving a copy to sources.xml.original when we do unpack gonk.tar.xz. Trust that it's good between updates of gonk snapshots
* ease the transition for existing build dirs by extracting sources.xml if no .original file is present, ie not trust any sources.xml already present, since I got burned a couple of times in staging where I'd munged the file

Did you want to take the 60 second hit on every job, or just move the sources.xml.original code into unpack-gonk ?

> The 'manifest' entry in self.config is only set in releng-beta.py and
> releng-beta-stable.py though, right?

I've set it in releng.py too, because AIUI that's how I reach otoro builds.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] Support for updating and uploading source manifests

As before except for:
* move creation of sources.xml.original into unpack_gonk(), to handle existing build directories that won't unpack gonk again
* add a branches variable to the script config to control which jobs upload a manifest
* swap to target_suffix variable to match existing usage
* change action names

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 690263 [details] [diff] [review]
[mozharness] Support for updating and uploading source manifests

http://hg.mozilla.org/build/mozharness/rev/46139ac290fd

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 690263 [details] [diff] [review]
[mozharness] Support for updating and uploading source manifests

Review of attachment 690263 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +332,5 @@
> +                new_sources.append('  <!-- Mercurial-Information: <project name="%s" path="gaia" remote="hgmozillaorg" revision="%s"/> -->' % \
> +                                     (gaia_config['repo'].replace('http://hg.mozilla.org/',''), self.buildbot_properties['gaia_revision']))
> +
> +        self.write_to_file(sourcesfile, "\n".join(new_sources), verbose=False)
> +        self.run_command(["diff", "-u", sourcesfile_orig, sourcesfile])

this is generating a spurious error in the logs:

08:17:23     INFO - Copy/paste: diff -u /builds/slave/b2g-try-panda-dep/build/sources.xml.original /builds/slave/b2g-try-panda-dep/build/sources.xml
08:17:23     INFO -  --- /builds/slave/b2g-try-panda-dep/build/sources.xml.original	2012-12-04 17:10:07.000000000 -0800
08:17:23     INFO -  +++ /builds/slave/b2g-try-panda-dep/build/sources.xml	2012-12-11 08:17:23.082434815 -0800
08:17:23     INFO -  @@ -8,6 +8,9 @@
08:17:23     INFO -     <default remote="linaro" revision="refs/tags/android-4.0.4_r2.1" sync-j="4"/>
08:17:23     INFO -     <!-- Gonk specific things and forks -->
08:17:23     INFO -  +  <!-- Mercurial-Information: <remote fetch="http://hg.mozilla.org/" name="hgmozillaorg"> -->
08:17:23     INFO -  +  <!-- Mercurial-Information: <project name="try" path="gecko" remote="hgmozillaorg" revision="c14d81b395bf"/> -->
08:17:23     INFO -  +  <!-- Mercurial-Information: <project name="https://git.mozilla.org/releases/gaia.git" path="gaia" remote="hgmozillaorg" revision="460ba0c43e1a0c942aedf272ec0f3cbe20905ed5"/> -->
08:17:23     INFO -     <project name="platform_build" path="build" remote="b2g" revision="273ba23d5c6c9f6a34995a3cc429804d1449ca9f">
08:17:23     INFO -       <copyfile dest="Makefile" src="core/root.mk"/>
08:17:23     INFO -     </project>
08:17:23    ERROR - Return code: 1

Comment 11

[Jonathan Griffin (:jgriffin)]

I don't see the manifests being uploaded here:  https://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/.  Are they being uploaded elsewhere?  Our partners need these for testing.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per catlee: 

...
You should see manifests being published in e.g.
http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/2012-12-10/source_unagi_2012-12-10.xml
...

Comment 13

[Jonathan Griffin (:jgriffin)]

(In reply to John O'Duinn [:joduinn] from comment #12)
> per catlee: 
> 
> ...
> You should see manifests being published in e.g.
> http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/2012-12-10/
> source_unagi_2012-12-10.xml
> ...

Unfortunately, this doesn't do our partners a lot of good, as they can't use this to reproduce one of our builds, which is what they want to use it for.  They need the git gecko and gaia commits that were the basis of this build.

Comment 14

[Tony Chung [:tchung]]

(In reply to Jonathan Griffin (:jgriffin) from comment #13)
> (In reply to John O'Duinn [:joduinn] from comment #12)
> > per catlee: 
> > 
> > ...
> > You should see manifests being published in e.g.
> > http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/2012-12-10/
> > source_unagi_2012-12-10.xml
> > ...
> 
> Unfortunately, this doesn't do our partners a lot of good, as they can't use
> this to reproduce one of our builds, which is what they want to use it for. 
> They need the git gecko and gaia commits that were the basis of this build.

Is this work tracked in bug 820245?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Yes, lets do that there. I'll fix the exit status from diff here (comment #10).

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] exit status of 1 is the new 0

This makes the log say INFO instead of ERROR, eg
18:31:37     INFO - Running command: ['diff', '-u', '/builds/slave/b2g-m-cen-unagi-dep/build/sources.xml.original', '/builds/slave/b2g-m-cen-unagi-dep/build/sources.xml']
18:31:37     INFO - Copy/paste: diff -u /builds/slave/b2g-m-cen-unagi-dep/build/sources.xml.original /builds/slave/b2g-m-cen-unagi-dep/build/sources.xml
18:31:37     INFO -  --- /builds/slave/b2g-m-cen-unagi-dep/build/sources.xml.original	2012-12-10 09:23:49.000000000 -0800
18:31:37     INFO -  +++ /builds/slave/b2g-m-cen-unagi-dep/build/sources.xml	2012-12-11 18:31:37.968884290 -0800
18:31:37     INFO -  @@ -11,6 +11,9 @@
18:31:37     INFO -     <default remote="caf" revision="ics_chocolate_rb4.2" sync-j="4"/>
18:31:37     INFO -     <!-- Gonk specific things and forks -->
18:31:37     INFO -  +  <!-- Mercurial-Information: <remote fetch="http://hg.mozilla.org/" name="hgmozillaorg"> -->
18:31:37     INFO -  +  <!-- Mercurial-Information: <project name="mozilla-central" path="gecko" remote="hgmozillaorg" revision="f6580fbf3b0b"/> -->
18:31:37     INFO -  +  <!-- Mercurial-Information: <project name="integration/gaia-nightly" path="gaia" remote="hgmozillaorg" revision="bf21e0c83930"/> -->
18:31:37     INFO -     <project name="platform_build" path="build" remote="b2g" revision="273ba23d5c6c9f6a34995a3cc429804d1449ca9f">
18:31:37     INFO -       <copyfile dest="Makefile" src="core/root.mk"/>
18:31:37     INFO -     </project>
18:31:37     INFO - Return code: 1

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 691177 [details] [diff] [review]
[mozharness] exit status of 1 is the new 0

http://hg.mozilla.org/build/mozharness/rev/ce6a335ddea2

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Bug 820245 will make the manifests partner-compatible.