654003 - test mozilla-aurora, mozilla-beta changes against specific Add-on SDK revision

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Phil Ringnalda (:philor)]

+++ This bug was initially created as a clone of Bug #653824 +++

The way that the Jetpack tests on mozilla-aurora went from passing on the depend build of c90cd210cad3 (which ran against http://hg.mozilla.org/projects/addon-sdk/rev/8469e1d8229f) to failing across the board on the nightly build on the same cset (which ran against http://hg.mozilla.org/projects/addon-sdk/rev/55fcda231ddc, which continues to fail on aurora) indicates a need to have our semi-stable Firefox trees also not be testing against the tip of the SDK repo.

Jetpack's currently hidden on mozilla-aurora.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

It seems to me that updating testing/jetpack/jetpack-location.txt for a new jetpack rev is going to cause a whole set of builds and tests, just to update the tests package and trigger a bunch of jetpack tests. If we can figure out a way to get the same result without all the other jobs running then that's a win for load on the infrastructure.

Comment 2

[Phil Ringnalda (:philor)]

That'd wfm: I was completely satisfied the moment I hid them, so if someone wants to pick a good rev (the current rev m-c is using is one push past the last rev that was successfully tested on m-a, so I have no way of knowing whether or not it works there), push it to m-a DONTBUILD, then be on the hook to wait for the next push that builds, look at &noignore=1, and unhide the things that green up, I certainly wouldn't object.

Comment 3

[Myk Melez [:myk] [@mykmelez]]

Attachment: update test automation to latest stable SDK revision - beta, aurora

Here's a patch that updates mozilla-aurora and mozilla-beta to the latest stable SDK revision.

I'll submit a patch to update mozilla-central to the same revision next.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Attachment: update test automation to latest stable SDK revision - central

Here's a patch that updates mozilla-central to the same latest stable SDK revision to which the previous patch updates mozilla-aurora and mozilla-beta.

Comment 5

[cmtalbert]

Do you need me to land these?

Comment 6

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #5)
> Do you need me to land these?

Yes, please!

Comment 7

[cmtalbert]

We need to land the attachment: https://bugzilla.mozilla.org/attachment.cgi?id=530193 on the aurora and beta repos in order to ensure that proper jetpack testing is being performed.  I think I'm doing this correctly, but please tell me if I'm wrong:
* I've set tracking-firefox-5 for the landing on beta (as firefox 5 is already in beta, correct?)
* I've set tracking-firefox-6 for the landing to aurora (as firefox 6 is already in aurora?  Or is the May 17 the move of mozilla-central into aurora?)  

If we move m-c into aurora on may 17, then the m-c change could be picked up at that time and we *don't* need an aurora landing.  We'd only need a landing on the beta repo so that the beta automation performs the jetpack tests using the proper version of jetpack.

Landed the Mozilla-central change here: http://hg.mozilla.org/mozilla-central/rev/2a7aeaf22502

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

May 17 is the switch for 5 to move to beta, and 6 to move to Aurora, and m-c to become 7.

Comment 9

[cmtalbert]

(In reply to comment #8)
> May 17 is the switch for 5 to move to beta, and 6 to move to Aurora, and m-c
> to become 7.
Thanks very much for that clarification!
So in that regard, I think what I would like to land this on aurora repo, not beta.  Then on May 17:
* the m-c change will go into aurora for firefox 6
* the proposed aurora landing will go into beta for firefox 5.

Myk, release drivers - does that sound do-able?

Comment 10

[cmtalbert]

Myk, it looks like the windows XP test run with this changeset has change the way the red failure occurs there.  It now looks like the browser is hanging because we once again see the "unable to delete profile" issue, where we didn't see that before hand.

Compare:
* This change: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1305054510.1305055502.13114.gz&fulltext=1
* The previous addon-sdk changeset: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1305050127.1305051135.24069.gz&fulltext=1

Is that expected to have occurred?  I'm leaving it in since it doesn't materially change the behavior of the jetpack tests, they were red on windows previously, they are still red on windows now.  

Let me know if you want me to back out.

Comment 11

[Johnathan Nightingale [:johnath]]

(In reply to comment #7)
> We need to land the attachment:
> https://bugzilla.mozilla.org/attachment.cgi?id=530193 on the aurora and beta
> repos in order to ensure that proper jetpack testing is being performed.  I
> think I'm doing this correctly, but please tell me if I'm wrong:
> * I've set tracking-firefox-5 for the landing on beta (as firefox 5 is
> already in beta, correct?)
> * I've set tracking-firefox-6 for the landing to aurora (as firefox 6 is
> already in aurora?  Or is the May 17 the move of mozilla-central into
> aurora?)  
> 
> If we move m-c into aurora on may 17, then the m-c change could be picked up
> at that time and we *don't* need an aurora landing.  We'd only need a
> landing on the beta repo so that the beta automation performs the jetpack
> tests using the proper version of jetpack.
> 
> Landed the Mozilla-central change here:
> http://hg.mozilla.org/mozilla-central/rev/2a7aeaf22502

Clint - is this just test changes? If so, we don't need tracking-firefox flags, just approval (and barely that - test-only changes are almost always virtuous). If this is test only, and doesn't change production code, a=me.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #10)
> Myk, it looks like the windows XP test run with this changeset has change
> the way the red failure occurs there.  It now looks like the browser is
> hanging because we once again see the "unable to delete profile" issue,
> where we didn't see that before hand.

Subsequent builds have shown both kinds of hangs:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1305080308.1305081341.19507.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1305082205.1305082912.27259.gz

It isn't clear whether these are symptoms of the same problem or two different problems.  It sure would help for bug 468736 to be fixed, so we can eliminate that source of failure, in any case.


> Let me know if you want me to back out.

No, let's leave this in.


Note that we'll want to land another such change on mozilla-central before the mozilla-central version changes to 7 next week; otherwise, these tests will start failing on mozilla-central.  I filed bug 656195 on that.

Comment 13

[cmtalbert]

(In reply to comment #11)
> (In reply to comment #7)

> Clint - is this just test changes? If so, we don't need tracking-firefox
> flags, just approval (and barely that - test-only changes are almost always
> virtuous). If this is test only, and doesn't change production code, a=me.
Yep, this is test-only.  Thanks Johnath, I didn't know what our process was for test-only changes on the new branches.

One last question, once I land this on aurora, should I set the status-firefox5 flag to 'fixed'?  It seems to me that if you're using that field to note "everything that has changed on aurora" you might want that for test-only changes too. (in case in some future a test-only change makes something behave differently on aurora versus m-c, or in case a feature's backout forgets to include the feature's tests)

Comment 14

[Johnathan Nightingale [:johnath]]

Ya - please mark the status-firefox5 fixed when it is, so that this doesn't show up in our list of "ff5 stuff that isn't done yet" queries

Comment 15

[cmtalbert]

Landed on Aurora as well: http://hg.mozilla.org/releases/mozilla-aurora/rev/c088d30784c6

Already landed on m-c, moving to fixed.