629263 - [Tracking bug] Make Jetpack-sdk meet the requirements for unhiding on TBPL

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Discovered that jetpack has moved to using another repo, when we disabled the jetpack tests that were hanging our test slaves and taking down the entire win7 pool.

Now that bug#627604 is fixed, we are now testing the code in the new jetpack-sdk repo. Since turning this on yesterday, the tests are failing. It looks like the failures are not setup failures. Instead, it looks like these failures are caused by regressions introduced sometime between when the repos moved (~1st-15th Dec 2010) and today (26 Jan 2011)

Using http://tree.mozilla.org/?noignore=1, you can see the failing Jp tests on mozilla-central and on Tryserver. Once all these are passing green (depbugs filed to track specific test failures), then this bug is to track enabling the Jp test suite on tinderbox/tbpl.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From bug#627604, comment#8:

(In reply to comment #8)
> (In reply to comment #4)
> > Leaving this open to track tests reporting green, and then unhiding them on
> > tinderbox/tbpl.
> 
> They are not hidden because they were red, they were red because they were
> hidden, 
The dep bugs here are to track turning the red suites to green. 



> and they were hidden because the decision reached in the public
> discussion in m.d.planning was that it wasn't acceptable to have the tip of a
> separate repository close mozilla-central, so we were going to wait until
> jetpack had something stable that we could import into mozilla-central, and
> then have visible tests that ran against that. Are we now talking about
> unhiding them in the same state that was previously unacceptable because we
> privately reverted that decision, or just because we forgot why they were
> hidden?
Ah - news to me. I thought the tests were hidden because they were failing. If there are other reasons for not displaying those tests, it would be great if you can point me to them?

Comment 2

[Phil Ringnalda (:philor)]

http://groups.google.com/group/mozilla.dev.planning/browse_thread/thread/d8dcc91d4f004bae/e7f968f6ed012d53

Comment 3

[Phil Ringnalda (:philor)]

And more generally (if you include the quote that GooGroups hides, and up and down the thread, and probably another broken fragment of thread nearby), http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/f5aeec1990ac36b0/7dc2e6f25d1e0af5#7dc2e6f25d1e0af5

Comment 4

[Armen [:armenzg]]

Attachment: [configs] disable jetpack from mozilla-central and try server

There is no point of running something that has been red for months.

This should be fixed by requesting on adding this type of job on a specific tree and once proven to releng that it runs green on that tree we can then enable it for the other branches.

Please make a comment if you don't agree or reply to the email thread with involved stakeholders.

Comment 5

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #4)
> This should be fixed by requesting on adding this type of job on a specific
> tree and once proven to releng that it runs green on that tree we can then
> enable it for the other branches.

I don't quite understand this suggested fix, but the problems are being worked on, and I would very much like for these tests to continue running so we can observe the effects of the fixes we land for them.

Comment 6

[Armen [:armenzg]]

I sent an email claryfying.

You would still be able to do your checkins on projects/addon-sdk and get your results in http://tinderbox.mozilla.org/showbuilds.cgi?tree=Jetpack

* jetpack as a project would still run
* jetpack as test suite for Firefox/TryServe would stop until re-enabled

sorry for any confusion.

Comment 7

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #6)
> I sent an email claryfying.
> 
> You would still be able to do your checkins on projects/addon-sdk and get your
> results in http://tinderbox.mozilla.org/showbuilds.cgi?tree=Jetpack

That's good, but we need to fix both kinds of automated tests, and the problems appear to be different.


> * jetpack as test suite for Firefox/TryServe would stop until re-enabled

How involved is reenabling this?  Is it just flipping a switch, or will you have to do some hacking?  This took a long time to set up initially, and I want to avoid a situation where it takes a long time to get it back up and running.

I'm ok with fixing the projects/addon-sdk-triggered tests first, but I want to make sure we can quickly pivot to fixing the moz-central-triggered tests afterward.

Comment 8

[Armen [:armenzg]]

(In reply to comment #7)
> (In reply to comment #6)
> > * jetpack as test suite for Firefox/TryServe would stop until re-enabled
> 
> How involved is reenabling this?  Is it just flipping a switch, or will you
> have to do some hacking?  This took a long time to set up initially, and I want
> to avoid a situation where it takes a long time to get it back up and running.
> 
> I'm ok with fixing the projects/addon-sdk-triggered tests first, but I want to
> make sure we can quickly pivot to fixing the moz-central-triggered tests
> afterward.

Replied through email.
Enabling/disabling is as easy as flipping a switch (like attachment 523319 [details] [diff] [review]).

Comment 9

[Armen [:armenzg]]

Comment on attachment 523319 [details] [diff] [review]
[configs] disable jetpack from mozilla-central and try server

This was checked-in on the "default" branch:
http://hg.mozilla.org/build/buildbot-configs/rev/0ef805b8e001

This will be picked up in our next reconfig.
Probably tomorrow morning.

Comment 10

[Armen [:armenzg]]

Myk thanks a lot for filing those bugs.
We will investigate the issues for projects/addon-sdk (http://tinderbox.mozilla.org/showbuilds.cgi?tree=Jetpack) and once solved those issues we will start re-enabling for mozilla-central and other branches.

Comment 11

[Armen [:armenzg]]

Assigning to lsblakk as per email convo.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

Status Update:

1. We've resolved all the consistent failures on the Jetpack branch.

2. We still have at least one intermittent failure there:

  bug 647123 - jetpack-fedora64 build reports failure even though all tests pass

3. Lukas has reenabled SDK tests on the moz-central branch (bug 651152).  We have three consistent failures there, one for each platform:

  bug 647136 - mozilla-central Windows XP jetpack test runs fail with "cannot find zipfile directory"
  bug 647135 - moz-central Mac jetpack test runs fail with "installdmg.sh: No such file or directory"
  bug 651255 - Linux Add-on SDK moz-central test runs fail with "bzip2: (stdin) is not a bzip2 file"

4. We're tracking at least one other bug on making SDK testing better:

  bug 649846 - make run_jetpack.sh more resilient against multiple matched builds in latest-mozilla-central

5. And we have bugs that we think are fixed but are open pending confirmation:

  bug 647126 - jetpack-win7 build fails tests due to test timeout
  bug 629259 - Jetpack test suite fails on OSX in cuddlefish.run()

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

I've just gone and hidden linux-debug jetpack and macosx debug jetpack since they are doing this timeout: 

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1303266375.1303267749.17111.gz&fulltext=1

In the morning hopefully someone will be able to investigate/file bugs if necessary.

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

also hid linux,linux64,macosx debug on Mozilla-Aurora and MozillaTry

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

also macosx64 debug for all three of the affected branches, it's intermittently timing out there too

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: reverse order of preparing the package/checking for APP_PATH in run_jetpack

all the jetpack-poller tests went orange on their most recent build because of not finding APP_PATH (this didn't show up in staging, will need to look more closely at clobbering old dirs) - this reverses the order so that the APP_PATH should exist after the preparation (unzip,untar,installdmg) is done.

Comment 17

[Lukas Blakk [:lsblakk] use ?needinfo]

(this didn't show up in staging, will need to look more
> closely at clobbering old dirs) 

-- not actually a clobbering dirs problem, more of a "I need to look closely at which repo I am pushing to and which is being cloned" problem.

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 527307 [details] [diff] [review]
reverse order of preparing the package/checking for APP_PATH in run_jetpack

http://hg.mozilla.org/build/tools/rev/cd63395104ea
landed and will push a new set of jetpack-poller tests now to confirm

Comment 19

[Phil Ringnalda (:philor)]

We seem to have trouble remembering things like comment 2 and comment 3, and unhiding these without actually addressing them. All the JP tests are hidden again on the Firefox tree, and will remain hidden until they are actually addressed.

Comment 20

[Mark A. Hershberger (hexmode)]

(In reply to Phil Ringnalda (:philor) from comment #19)
> We seem to have trouble remembering things like comment 2 and comment 3, and
> unhiding these without actually addressing them. All the JP tests are hidden
> again on the Firefox tree, and will remain hidden until they are actually
> addressed.

I'll go through and check the open blockers in just a sec, but are these issues included on the open blockers?  Can the tests be un-hidden now?

Comment 21

[Alexandre Poirot [:ochameau]]

I went through all recent jetpack test failures and it looks like we are facing three major intermittents: 689291, 758279, 758283.
Here I'm only talking about bugs related to failing tests of SDK.
There is many violets due to the test runner that can't download firefox binary.

Comment 22

[Phil Ringnalda (:philor)]

(In reply to Mark A. Hershberger (hexmode) from comment #20)
> I'll go through and check the open blockers in just a sec, but are these
> issues included on the open blockers?  Can the tests be un-hidden now?

No and no. The only one that's filed is the no-progress bug 695916, because there's no useful way to file "Gecko isn't willing to gate its checkins on code in a separate repository using separate source control on a separate site using a separate bug tracker, and Jetpack isn't willing to budge from any of those much less all of them."

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Moving the assignee over to Hal since he's been working with Jetpack lately on this and other bugs.

Comment 24

[Ed Morley [:emorley]]

Think this is pretty much ready to be called fixed.

The only thing I did spot, was this intermittent on inbound:
https://tbpl.mozilla.org/php/getParsedLog.php?id=21339204&tree=Mozilla-Inbound

...which doesn't appear in the TBPL annotated summary.

Any chance we could output a TBPL parsable error string? :-)

See links to regexp etc at:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#6.29_Outputs_failures_in_a_TBPL-starrable_format

(Happy to add additional regexp if needed & generic enough to be used in several places)

Comment 25

[Dave Townsend [:mossop]]

(In reply to Ed Morley [:edmorley UTC+1] from comment #24)
> Think this is pretty much ready to be called fixed.
> 
> The only thing I did spot, was this intermittent on inbound:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=21339204&tree=Mozilla-
> Inbound
> 
> ...which doesn't appear in the TBPL annotated summary.

Yeah we're not actually sure what is failing there so it makes it a bit difficult. I have identified a few cases where we won't get a failure message and am fixing those in bug 855086 so hopefully that will reveal it to us.

Comment 26

[Ed Morley [:emorley]]

Great - thank you :-)

Comment 27

[Ed Morley [:emorley]]

As far as I can tell, all the requirements are now met - we just need to green jetpack up on inbound/m-c/... (recently busted) and we can mark this bug as fixed. As soon as I see the bugmail from this being closed, I'll assign bug  	854451 to myself. Almost there! :-)

Comment 28

[Dave Townsend [:mossop]]

We're now green on m-c and m-i and as an added bonus have probably killed an intermittent that was showing up. I think we're done here.