695921 - test Jetpack tree against latest mozilla-{aurora,beta,central,release} opt & debug builds (instead of latest nightly m-c opt only)

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Myk Melez [:myk] [@mykmelez]]

Jetpack tree test runs currently test against the latest nightly central build of Firefox.  Ideally, they would test against the most recent build produced by the Firefox tree buildbots, so integration testing of SDK changes is more continuous.

As example of where this would help is the following SDK commit:

https://tbpl.mozilla.org/?tree=Jetpack&usetinderbox=1&rev=1e7de63d8710

That commit changed the behavior of the SDK yesterday in response to a change in Firefox on the same day.  Because the automation tested the SDK against the morning's nightly build, a test failed.  If it had tested against the most recent build produced by the Firefox tree buildbots instead (as I did manually before landing the change), the test would have succeeded.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

This means changing the ftp url we use for the jetpack poller triggered tests from

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central

to the most-recent build dir in 

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-$platform/

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: run_jetpack.py now looks in the timestamp dirs in tinderbox-builds for executable

ran this in local staging with a snowleopard builder - this now adds in to the script the ability to look into tinderbox-build dirs which are timestamped dirs. It finds the most recent dir and grabs the executable from there. This goes with a new ftp_url that will be uploaded shortly in a separate patch.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: update to ftp_url for jetpack poller runs

for the newly updated run_jetpack.py which wants the most recent branch build and no longer the latest-nightly for a branch

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 568578 [details] [diff] [review]
update to ftp_url for jetpack poller runs

>diff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
>-        'ftp_url': 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central',
>+        'ftp_url': 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-%(platform)s',

Might be worth checking windows doesn't barf on the % and brackets when this gets passed in the call to run_jetpack.py. Or does it get expanded by the master before it gets into the call ?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 568577 [details] [diff] [review]
run_jetpack.py now looks in the timestamp dirs in tinderbox-builds for executable

>diff --git a/buildfarm/utils/run_jetpack.py b/buildfarm/utils/run_jetpack.py
>+PLATFORMS = {'snowleopard': 'macosx64',
>+             'xp': 'win32',
>+             'win7': 'win32',
>+             'w764': 'win64',
>+             'fedora': 'linux',
>+             'fedora64': 'linux64',
>+             }

You'll need a mapping from leopard to macosx64 here too I think.

>+        for line in lines:
>+            try:
>+                directory = int(line.split(" ")[-1])
>+                if directory > most_recent:
>+                    most_recent = directory
>+            except:
>+                continue

The 'except:' hides the 
 ValueError: invalid literal for int() with base 10: '20111004125034-70e4de45a0d0'
we hit on the new-style directory naming that bug 671450 will switch us to (BuildID-revision). That bug may also going to make problems for you by symlinking old-style to new-style during the transition, and the aborted deployment gives us a few 2011... dirs for old builds (but these can be fixed up by moving the files where they should be). eg ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64/

Otherwise looks fine.

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: run_jetpack.py now looks in tinderbox-builds dirs for executable

I did a test in staging with talos-r3-w7-002 to check if the ftp url caused choking, it did not.

Also I changed the way of finding the most recent dir in tinderbox-builds so that it doesn't care about the name of the dir - looking at those bugs you mentioned made me think looking at dir names would be too fragile (and let's face it, this whole script is somewhat fragile enough). Now it just grabs the dirname that is the most recent based on the modified times. Worked in local staging.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: run_jetpack.py now looks in tinderbox-builds dirs for executable

expanded this to include working with debug builds as well, which mainly means that I had to add special-casing for 32bit macosx debug builds and I also hard-coded '.dmg' to the installdmg.sh call since I doubt we'll be using a non-dmg installer package any time soon and I needed the options.ext to have a better pattern for matching with both mac.dmg and mac64.dmg filenames.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: update to ftp_url for jetpack poller runs and also enable debug & branch config vars

adds in turning on runs against debug builds and adding multiple branch support

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 569445 [details] [diff] [review]
run_jetpack.py now looks in tinderbox-builds dirs for executable

>diff --git a/buildfarm/utils/run_jetpack.py b/buildfarm/utils/run_jetpack.py
>+                if ":" in parts[-2]:
>+                    time = " ".join([str(datetime.now().year), parts[-4], parts[-3], parts[-2]]) 
>+                    dir_time = datetime.strptime(time, "%Y %b %d %H:%M")
>+                    if dir_time > most_recent:
>+                        most_recent = dir_time
>+                        directory = parts[-1]

As we discussed on IRC, when 2011 comes along all the modification times will have a year instead of a time in parts[-2]. All the old directories will be ignored, so we'll need another set of builds from each branch before jetpack jobs will succeed again. If that's OK with Myk we can take that as a known issue.

>+        # Now get the executable for this platform
>+        urls = urllib.urlopen("%s/%s" % (ftp_url, directory))

Should add a check for the directory var before this hunk, and exit if it is still None. Otherwise the urlopen failing on .../None will be a little cryptic.

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 569446 [details] [diff] [review]
update to ftp_url for jetpack poller runs and also enable debug & branch config vars

http://hg.mozilla.org/build/buildbot-configs/rev/bedb50ffa494 on default

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 569445 [details] [diff] [review]
run_jetpack.py now looks in tinderbox-builds dirs for executable

http://hg.mozilla.org/build/tools/rev/331630a8506a landed and the configs are in production - so now we need a push to verify this

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: jetpack buildbotcustom patch to enable more builders for branch/build type

can't believe I left this one out - will need to do another reconfig

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Test runs came up fine on the no-op push, looks like this is good to go.

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

I am curious if there is a reason for the ftp_url key being defined twice, starting with

http://hg.mozilla.org/build/buildbot-configs/rev/cbc037eaf61d

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to John Ford [:jhford] from comment #15)
> I am curious if there is a reason for the ftp_url key being defined twice,
> starting with
> 
> http://hg.mozilla.org/build/buildbot-configs/rev/cbc037eaf61d

Thanks for catching, landed fix on default http://hg.mozilla.org/build/buildbot-configs/rev/108a2df322e5

Comment 17

[Armen [:armenzg]]

This went live at 8:45 AM PDT.

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Armen Zambrano G. [:armenzg] - (off Friday 4th) from comment #17)
> This went live at 8:45 AM PDT.

No, it didn't :) It went live in comment #14

Comment 19

[Armen [:armenzg]]

Oh! I thought you only did reconfigs on Friday for the release.
I thought 108a2df322e5 did not make it until today's reconfig.