1023007 - Allow in-tree json for Gaia-Try to specify which Gecko build to use.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

A whole class of problems could be averted if I was able to set the URL to the specific bundle of Gecko to use for Gaia-Try tests in the gaia.json file.

Specific cases I care about right now are:

1. Race condition when updating Latest -- build from one commit, tests from a different one
2. bugs like bug 1022909
3. when /latest/ updating breaks
4. being able to test non-mozilla-central Gecko based branches on Gaia-Try

Number 4 is currently my main motivation.

My proposal is for my triggering system to insert an appropriate URL into the gaia.json file and have the mozharness script read that.  Ideally, leaving a fallback right now for downloading from /latest/ would be helpful while we get this set up.

This would change:

{
  "git": {
    "git_revision": "replacedByPrNumber",
    "remote": "https://github.com/jhford/test-repo.git",
    "github_pr_number": 2
  }
}

into:

{
  "git": {
    "git_revision": "replacedByPrNumber",
    "remote": "https://github.com/jhford/test-repo.git",
    "github_pr_number": 2
  },
  "gecko": {
    "gecko_base_dir": "https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/1402329723/en-US"
  }
}

If you can set up a branch that's identical to gaia-try for me to test with, I can try writing the mozharness patches.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

Alternately, please set up the following branches:

gaia-try-v1.4
gaia-try-v2.0

These branches should point to the appropriate Gecko version for that version of Gaia.

Comment 2

[Aki Sasaki (not active)]

Dup of bug 1007370 ?

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Aki Sasaki [:aki] from comment #2)
> Dup of bug 1007370 ?

maybe?  that was specifically about setting the value of the hardcoded /latest/ link to a value stored in the gaia tree.  This bug is about being able to override the hardcoded link using the Gaia-Try mechanics at runtime.

In an ideal world, both options would exist.  I think the option described in this bug is more important in the short term.

Comment 4

[Aki Sasaki (not active)]

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #0)

> 4. being able to test non-mozilla-central Gecko based branches on Gaia-Try
> 
> Number 4 is currently my main motivation.
> 
> My proposal is for my triggering system to insert an appropriate URL into
> the gaia.json file and have the mozharness script read that.  Ideally,
> leaving a fallback right now for downloading from /latest/ would be helpful
> while we get this set up.

>     "gecko_base_dir":
> "https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-
> central-linux64_gecko/1402329723/en-US"
>   }


Only specifying the base url is problematic if you're trying to test 2.0 and 1.4; the version is baked into the filename, and we'll look for b2g-33.0a1-prepended files when we should be looking for 32.0a2 (2.0) or 30.0 (1.4).

We could change this to installer_url and test_url (both need to be specified).  Once installer_url and test_url are both defined at the top level of the gaia.json (peers of "git", not inside a "gecko" dict), we could pass the gaia.json url as a second config file argument to the mozharness script and remove the --installer-url and --test-url args from the commandline.

We will still hit bug 1022909 type bustage on merge day.  To solve, we'd have to either have some sort of ftp scraping (ugh), update a branch-to-version map in the hook while potentially pausing gaia-try pushes until everything's lined up, or removing version numbers from the url.

As a side note, if your hook can look at the pull request for the releng.json (or whatever) file and finds it, it could then alter the installer_url and test_url in gaia.json, solving bug 1007370.  I don't know if that would be too much hook-side logic or not.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Aki Sasaki [:aki] from comment #4)
> Only specifying the base url is problematic if you're trying to test 2.0 and
> 1.4; the version is baked into the filename, and we'll look for
> b2g-33.0a1-prepended files when we should be looking for 32.0a2 (2.0) or
> 30.0 (1.4).

Good call.  I guess we could be also specify the version number in the gaia.json file, but that feels silly.  I like your approach of direct URLs better.

> We could change this to installer_url and test_url (both need to be
> specified).  Once installer_url and test_url are both defined at the top
> level of the gaia.json (peers of "git", not inside a "gecko" dict), we could
> pass the gaia.json url as a second config file argument to the mozharness
> script and remove the --installer-url and --test-url args from the
> commandline.

I think I understand, but I'm not entirely sure what you mean with this.  Can you show me how I would need to change:

{
  "git": {
    "git_revision": "replacedByPrNumber",
    "remote": "https://github.com/jhford/test-repo.git",
    "github_pr_number": 2
  }
}

into a format that'll work with your idea?

> We will still hit bug 1022909 type bustage on merge day.  To solve, we'd
> have to either have some sort of ftp scraping (ugh), update a
> branch-to-version map in the hook while potentially pausing gaia-try pushes
> until everything's lined up, or removing version numbers from the url.

This wouldn't be an issue if we go with direct installer and test urls, right?  My plan is to periodically scrape FTP to determine the URLs, so what I'm putting in the file should already have the correct version number.  Is there something I'm missing?

Comment 6

[Aki Sasaki (not active)]

Oh, crap, I just figured out why my idea won't work as-is.
However, to clarify:

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #5)
> I think I understand, but I'm not entirely sure what you mean with this.
> Can you show me how I would need to change:
>
> {
>   "git": {
>     "git_revision": "replacedByPrNumber",
>     "remote": "https://github.com/jhford/test-repo.git",
>     "github_pr_number": 2
>   }
> }
>
> into a format that'll work with your idea?

 {
   "git": {
     "git_revision": "replacedByPrNumber",
     "remote": "https://github.com/jhford/test-repo.git",
     "github_pr_number": 2
   },
   "installer_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-33.0a1.en-US.linux-x86_64.tar.bz2",
   "test_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-33.0a1.en-US.linux-x86_64.tests.zip"
 }

Written out, it's obvious that the installer_url and test_url are hardcoded to linux64 urls, and osx and linux32 tests will burn or test the wrong thing.

Brainstorming:
a) We could have multiple files here, e.g. linux64.json, linux32.json, macosx64.json, each with only the installer_url and test_url:

linux64.json
{
  "installer_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-33.0a1.en-US.linux-x86_64.tar.bz2",
  "test_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-33.0a1.en-US.linux-x86_64.tests.zip"
}

b) Or we could do some fugly string manipulation.
c) Or we could turn off macosx64 and linux32 tests and hardcode linux64 links in gaia.json as above.

Of these three, I like (c) the best and (a) second best, and (b) is pretty fugly.  I don't know how kosher it would be to turn off mac+linux32 tests on gaia-try.

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #5)
> (In reply to Aki Sasaki [:aki] from comment #4)
> > Only specifying the base url is problematic if you're trying to test 2.0 and
> > 1.4; the version is baked into the filename, and we'll look for
> > b2g-33.0a1-prepended files when we should be looking for 32.0a2 (2.0) or
> > 30.0 (1.4).
>
> Good call.  I guess we could be also specify the version number in the
> gaia.json file, but that feels silly.  I like your approach of direct URLs
> better.

Yeah, we can templatize the url to try to get around the platform thing too (option b above) but to me it feels more complex, more fragile, more difficult to understand.

> > We will still hit bug 1022909 type bustage on merge day.  To solve, we'd
> > have to either have some sort of ftp scraping (ugh), update a
> > branch-to-version map in the hook while potentially pausing gaia-try pushes
> > until everything's lined up, or removing version numbers from the url.
>
> This wouldn't be an issue if we go with direct installer and test urls,
> right?  My plan is to periodically scrape FTP to determine the URLs, so what
> I'm putting in the file should already have the correct version number.  Is
> there something I'm missing?

That would be one of the solutions to bug 1022909 that I listed, yes:
> > To solve, we'd
> > have to either have some sort of ftp scraping (ugh),
The periodic scraping seems preferable than per-gaia-pr scraping given that we just want a relatively recent desktop build.

You'll need to scrape m-c for linux{32,64} and macosx64, as well as other gecko repos you care about (mozilla-b2g30_v1_4 and mozilla-aurora for 2.0).  In 6 weeks, mozilla-aurora's b2g builds will move to mozilla-b2g32_v2_0; 6 weeks after that, 2.1 builds will move from central to aurora, so you'll have scraper config changes every 6 weeks.

Comment 7

[Aki Sasaki (not active)]

In addendum to comment 6, linux64-debug will be another set of urls, so we'll either need to also turn those off or have a linux64-debug.json as appropriate.

Comment 8

[Aki Sasaki (not active)]

As a[nother] side note, I'm sure you are aware, but in case:

Mulet will be EOLing b2g desktop.  Hopefully our work here can carry over, but it's not entirely clear yet.
https://groups.google.com/forum/#!msg/mozilla.dev.b2g/2WUg10993OM/VSceN9i2w2EJ
bug 997096

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

I don't think we can turn off coverage because of this.  I'd much rather not have to manage four extra json files, but if that's the only option I'd vastly prefer that to what we have now.

(In reply to Aki Sasaki [:aki] from comment #8)
> Mulet will be EOLing b2g desktop.  Hopefully our work here can carry over,
> but it's not entirely clear yet.

We will still need to run Gaia unit tests, whether that's with Mullet or B2G Desktop. You're talking specifically about getting paths to B2G Desktop bundle, right?

(In reply to Aki Sasaki [:aki] from comment #6)
> The periodic scraping seems preferable than per-gaia-pr scraping given that
> we just want a relatively recent desktop build.

Yep, the plan was to scrape once every X minutes, store that value in a DB.

Comment 10

[Aki Sasaki (not active)]

Added default platform configs to gaia-try: http://hg.mozilla.org/integration/gaia-try/rev/0f09e16be90e

Comment 11

[Aki Sasaki (not active)]

Attachment: withprops

Use WithProperties() and mozharness' ability to specify a url to a config file.

Comment 12

[Aki Sasaki (not active)]

Attachment: withprops

With fake WithProperties goodness

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8443006 [details] [diff] [review]
withprops

https://hg.mozilla.org/build/buildbot-configs/rev/b3f5e433dd70

Comment 14

[Aki Sasaki (not active)]

This is live, which means that jhford should be able to solve this bug, the future version of bug 1022909, and bug 1007370 without further releng intervention.

Comment 15

[Aki Sasaki (not active)]

Re-reading this bug, it looks like the hook-side isn't included in comment 0, only the releng portion, which is done.  Resolving.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

Actually, this is busted.

20:08:10    FATAL - Unexpected number of files in buildbot config buildprops.json.
20:08:10    FATAL - Expected these number(s) of files: [1, 2, 3], but got: 5
20:08:10    FATAL - Running post_fatal callback...
20:08:10    FATAL - Exiting -1

I've attached a patch on bug 1025322 which I think should fix this.