Get Gaia sources onto build machines for building

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: jhford, Assigned: joduinn)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 2 obsolete attachments)

We need a copy of Gaia in order to package a Gaia profile into the B2G builds.  Currently, I have a copy of Gaia that lives on my People account.  This is not the best long term solution, but does unblock Firefox OS Desktop builds from being useful.

Let's figure out something better.
It turns out that we can't use my people account.  I've also been told that we can't use github's zip downloads.

I am not sure how this is going to be solved, but I'll need to know where the Gaia source is located.  Assuming $topsrcdir is the root of a mozilla-central clone, I need to know where gaia/Makefile resides.  The easiest is probably to put it in $topsrcdir/gaia.

Another option is to download the zip file that is generated by Github.com.
Created attachment 659800 [details] [diff] [review]
idea 1 -- store a copy of Gaia in Mercurial

This is an idea to clone Gaia using HG, similar to how build tools is cloned (and mobile-browser used to be).  I am updating the user repo by hand using hg-git.  This is not a perfect solution, but it gets us unblocked on creating useful b2g desktop nightly builds.
Attachment #659800 - Flags: review?(bhearsum)
Created attachment 659802 [details] [diff] [review]
idea 1 -- store a copy of Gaia in Mercurial (configs)
Attachment #659802 - Flags: review?(bhearsum)
I've tested idea one by:
-running and passing ./setup-masters.py -t -R build
-set up a local master and slave on my Mac machine.  Gaia was cloned and put in the correct location.  The property was set:

gaia_rev 	bbc6a371a125 	SetProperty Step
Comment on attachment 659802 [details] [diff] [review]
idea 1 -- store a copy of Gaia in Mercurial (configs)

Review of attachment 659802 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/b2g_config.py
@@ +30,4 @@
>      'product_prefix': 'b2g',
>      'unittest_suites': [],
>      'unittest_masters': [],
> +    'gaia_repo': 'http://hg.mozilla.org/users/jford_mozilla.com/gaia',

Is this really where we're pulling Gaia from?

@@ +190,4 @@
>                                'gnome-vfs2-devel', 'mpfr', 'xorg-x11-font',
>                                'imake', 'ccache'],
>              'tooltool_manifest_src': 'b2g/config/tooltool-manifests/linux32/releng.manifest',
> +            'clone_gaia': True,

Please put gaia_repo here instead of clone_gaia. You can factor it out to a top-level variable if you want, but there's no reason for this flag to exist.
Attachment #659802 - Flags: review?(bhearsum) → review-
Attachment #659800 - Flags: review?(bhearsum) → review-
Comment on attachment 659800 [details] [diff] [review]
idea 1 -- store a copy of Gaia in Mercurial

Review of attachment 659800 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +788,4 @@
>          self.tooltool_script = tooltool_script
>          self.tooltool_bootstrap = tooltool_bootstrap
>          self.runAliveTests = runAliveTests
> +        self.cloneGaia = cloneGaia

cloneGaia doesn't need to exist. This decision can be made based on gaiaRepo being None or not.

@@ +1144,5 @@
> +                self.addStep(ShellCommand(
> +                    name="gaia_rev",
> +                    command=["hg", "up", "-C", "-r", self.gaiaRevision],
> +                    workdir="build/gaia",
> +                ))

As Catlee pointed out on IRC, hgtool should be used here. You can call self.makeHgToolStep to do that, like this: https://github.com/mozilla/buildbotcustom/blob/master/process/factory.py#L3192

You should also let hgtool handle updating to the rev you want, so you'll want to pass something like "rev=self.gaiaRevision or 'default'" to it.
That location is where Gaia is stored for the purposes of the desktop builds.  I'm happy to have a better location for the repository or a different way of getting the sources onto the machine, but this unblocks us having useful desktop builds.
(In reply to John Ford [:jhford] from comment #7)
> That location is where Gaia is stored for the purposes of the desktop
> builds.  I'm happy to have a better location for the repository or a
> different way of getting the sources onto the machine, but this unblocks us
> having useful desktop builds.

We need to store them somewhere more official. I know Hal has some thoughts on this, so I'll let him jump in here.
Grabbing this, as I'm already working through mechanics of exact repo names and locations for each B2G branch with the B2G, l10n, RelCoord and RelEng teams.
Assignee: nobody → joduinn
OS: Mac OS X → All
(In reply to Ben Hearsum [:bhearsum] from comment #8)
> (In reply to John Ford [:jhford] from comment #7)
> > That location is where Gaia is stored for the purposes of the desktop
> > builds.  I'm happy to have a better location for the repository or a
> > different way of getting the sources onto the machine, but this unblocks us
> > having useful desktop builds.
> 
> We need to store them somewhere more official. I know Hal has some thoughts
> on this, so I'll let him jump in here.

That'd be great, I'm happy for this repository to be replaced.  The intention of using this repository is to unblock useful Desktop B2G builds from being generated.
Created attachment 659891 [details] [diff] [review]
idea 1v2 -- store a copy of Gaia in Mercurial (configs)
Attachment #659802 - Attachment is obsolete: true
Attachment #659891 - Flags: review?(bhearsum)
Created attachment 659892 [details] [diff] [review]
idea 1v2 -- store a copy of Gaia in Mercurial
Attachment #659800 - Attachment is obsolete: true
Attachment #659892 - Flags: review?(bhearsum)
Blocks: 776783
Any progress here?

Updated

6 years ago
Depends on: 791772
Comment on attachment 659891 [details] [diff] [review]
idea 1v2 -- store a copy of Gaia in Mercurial (configs)

Review of attachment 659891 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/b2g_config.py
@@ +30,4 @@
>      'product_prefix': 'b2g',
>      'unittest_suites': [],
>      'unittest_masters': [],
> +    'gaia_repo': 'users/jford_mozilla.com/gaia',

Sorry, I wasn't clear in my original comment - this should be in the global scope of the file, not in GLOBAL_VARS. Putting it here ends up with it being in multiple levels of the BRANCHES data structure, which is confusing.

r=me with that + changing the path to match the one in bug 791772.
Attachment #659891 - Flags: review?(bhearsum) → review-
Comment on attachment 659891 [details] [diff] [review]
idea 1v2 -- store a copy of Gaia in Mercurial (configs)

Review of attachment 659891 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/b2g_config.py
@@ +273,4 @@
>              'talos_masters': GLOBAL_VARS['talos_masters'],
>              'test_pretty_names': False,
>              'l10n_check_test': False,
> +            'gaia_repo': GLOBAL_VARS['gaia_repo'],

Oh, and just to confirm: we want this set for all branches, right?
Attachment #659892 - Flags: review?(bhearsum) → review+
Now that bug 791772 is resolved, should I file a bug for the releng hg mirror of gaia to be pushed to that location?
(In reply to John Ford [:jhford] from comment #16)
> Now that bug 791772 is resolved, should I file a bug for the releng hg
> mirror of gaia to be pushed to that location?

Can you push what you've got there? We have nothing set up for automatically mirroring this repo.
Created attachment 664234 [details] [diff] [review]
install wget in B2G Desktop build roots

We need wget to grab a copy of xulrunner off ftp in order build the Gaia profiles
Attachment #664234 - Flags: review?(bhearsum)
Attachment #664234 - Flags: review?(bhearsum) → review+
Comment on attachment 664234 [details] [diff] [review]
install wget in B2G Desktop build roots

Unbreaks B2G linux32_gecko mozilla-central nightly build in staging.
Attachment #664234 - Flags: feedback+
Comment on attachment 664234 [details] [diff] [review]
install wget in B2G Desktop build roots

Landed in https://hg.mozilla.org/build/buildbot-configs/rev/fbcb8ee43f65
Attachment #664234 - Flags: checked-in+
We have gaia sources on the build machines
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.