Last Comment Bug 504344 - Get unit tests turned on for SeaMonkey comm-central-trunk
: Get unit tests turned on for SeaMonkey comm-central-trunk
Status: RESOLVED FIXED
:
Product: SeaMonkey
Classification: Client Software
Component: Release Engineering (show other bugs)
: Trunk
: All All
: -- normal (vote)
: seamonkey2.1a1
Assigned To: Robert Kaiser (not working on stability any more)
:
Mentors:
Depends on: 485821 503807 526206 526208 526213 548085
Blocks: 492421 543528 CcMcBuildIssues 536940
  Show dependency treegraph
 
Reported: 2009-07-15 09:41 PDT by Robert Kaiser (not working on stability any more)
Modified: 2010-04-01 14:09 PDT (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Robert Kaiser (not working on stability any more) 2009-07-15 09:41:14 PDT
Once we have all the slaves we need available in the new buildbot pools, we can turn on unit tests for comm-central-trunk SeaMonkey builds.
Comment 1 Serge Gautherie (:sgautherie) 2009-07-15 13:43:22 PDT
Until we get the additional slaves, would it be possible to build and upload (but not run) something like a "nightly packaged build+test"?
Comment 2 Robert Kaiser (not working on stability any more) 2009-07-15 13:49:17 PDT
Not right now, as AFAIK, the factory doesn't support that. Also, I don't have time to even think about this before 2.0b1 builds run, and we are still missing some slave VMs and have instabilities in others, let's not push things to fast there.
Comment 3 Robert Kaiser (not working on stability any more) 2010-01-23 17:27:00 PST
I am turning on packaged xpcshell tests permanently on the main SeaMonkey tree, even if bug 541235 mailnews failures are still happening there.

For the other suite, I have done a test run for all of them on Linux debug builds, with the following results:

crashtest: green (1466/0/15) - http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey-Ports/1264293428.1264293820.28256.gz&fulltext=1

reftest: green (4109/0/294) - http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey-Ports/1264292500.1264295466.13505.gz&fulltext=1

mochitest: orange (T-FAIL L-FAIL) - http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey-Ports/1264292467.1264293906.29221.gz&fulltext=1
This one ended up crashing in libimglib2.so

mochitest-other: orange - http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey-Ports/1264292467.1264293435.24275.gz&fulltext=1
mochitest-chrome: T-FAIL L-FAIL
Another crash in in libimglib2.so
mochitest-browser-chrome: 591/22/12
A number of different errors, probably worth investigating
mochitest-a11y: 31463/554411/2
most of those are "JavaScript error: chrome://mochikit/content/a11y/accessible/events.js, line 782: nsIAccessibleEvent is not defined"

Please follow up to the errors on different bugs.
Comment 4 Serge Gautherie (:sgautherie) 2010-01-24 19:07:52 PST
(In reply to comment #3)

> crashtest: green (1466/0/15) -
> reftest: green (4109/0/294) -

Could we enable them? (They don't take too long, etc)

> mochitest: orange (T-FAIL L-FAIL) -
> mochitest-chrome: T-FAIL L-FAIL
> mochitest-browser-chrome: 591/22/12

Some failures may be intermittent and/or already filed:
running them once was nice, but is not enough (for me) to sort them out yet :-|
Would it be possible to run these suites with a low priority (= when truly idled)?

> mochitest-a11y: 31463/554411/2

This one is bug 535891.
Comment 5 Robert Kaiser (not working on stability any more) 2010-01-26 12:28:24 PST
(In reply to comment #4)
> (In reply to comment #3)
> 
> > crashtest: green (1466/0/15) -
> > reftest: green (4109/0/294) -
> 
> Could we enable them? (They don't take too long, etc)

Actually, reftest took a lot longer than I expected (about 45min), and I'd like to not enable things right now if they don't bring much of a value, as the machines are strained enough with what they are doing.

> > mochitest: orange (T-FAIL L-FAIL) -
> > mochitest-chrome: T-FAIL L-FAIL
> > mochitest-browser-chrome: 591/22/12
> 
> Some failures may be intermittent and/or already filed:
> running them once was nice, but is not enough (for me) to sort them out yet :-|
> Would it be possible to run these suites with a low priority (= when truly
> idled)?

Unfortunately, we only can make them be triggered on every debug build or not at all, and we can't assign different priorities.
But that said, if we have actual work being done on them and they don't run overly long, I might be willing to do something there, as this might create actual benefit for trunk.

> > mochitest-a11y: 31463/554411/2
> 
> This one is bug 535891.

Thanks, good to know.
Comment 6 Serge Gautherie (:sgautherie) 2010-01-26 14:18:39 PST
(In reply to comment #5)

> we can't assign different priorities.

I was wondering if prioritizeBuilders() at
http://mxr.mozilla.org/build/source/buildbot-configs/seamonkey/master.cfg#51
could be used to achieve a very low prioritization for tests builds?

> if we have actual work being done on them and they don't run
> overly long, I might be willing to do something there, as this might create
> actual benefit for trunk.

I'm feeling we're in an egg-chicken situation: it's much harder to get work started/done without a build to check :-|
Comment 7 Robert Kaiser (not working on stability any more) 2010-01-27 06:11:11 PST
(In reply to comment #6)
> (In reply to comment #5)
> 
> > we can't assign different priorities.
> 
> I was wondering if prioritizeBuilders() at
> http://mxr.mozilla.org/build/source/buildbot-configs/seamonkey/master.cfg#51
> could be used to achieve a very low prioritization for tests builds?

I have no idea how I would use that to selectively un-prioritize only those trunk test builds. I don't even claim to understand the code I copied there. Let's not take this too far, I don't think it's necessary.

> > if we have actual work being done on them and they don't run
> > overly long, I might be willing to do something there, as this might create
> > actual benefit for trunk.
> 
> I'm feeling we're in an egg-chicken situation: it's much harder to get work
> started/done without a build to check :-|

I think you misunderstood me, I meant I can turn something on that doesn't run too long, if I know that someone will pay attention and try to fix the problems. I wanted to indicate that I just don't want to put our machines under more strain than necessary without knowing that it will actively help us.
Comment 8 Serge Gautherie (:sgautherie) 2010-01-27 10:41:39 PST
(In reply to comment #7)

> I think you misunderstood me

Ah, got it:
*I would monitor them to file bugs.
*I probably wouldn't investigate/fix any of them (or a few at most),
 at least until I'm done with the "configure" backlog...
*And then they're the problem that c-1.9.2 hasn't branched yet, which is blocking some fixes wrt m-c, which may cause some failures. (As we already know for a fact ;-<)

In the very short term, I would be interested on enabling all Linux Debug tests for multiple (instead of just 1) builds, once on 2.1/m-c, once on 2.1/m-1.9.2 (if possible).
I expect many (bad) failures (as we saw), be that would be enough to file (mochitests-*) bugs this time.
Comment 9 Robert Kaiser (not working on stability any more) 2010-01-27 18:02:32 PST
(In reply to comment #8)
> *I would monitor them to file bugs.
> *I probably wouldn't investigate/fix any of them (or a few at most),
>  at least until I'm done with the "configure" backlog...

OK, those are reasons to have at least some of them running.

> In the very short term, I would be interested on enabling all Linux Debug tests
> for multiple (instead of just 1) builds, once on 2.1/m-c, once on 2.1/m-1.9.2
> (if possible).

Running all suites is out of the question, we just don't have the machine power right now. What I can and will do is putting up mochitest-other again there. mochitest-plain has the risk of running too long and block other builds, esp. as we have fewer Linux slaves than ones for any other platform right now,  and reftest/crashtest are too little reward right now as they have been green in the tests.
Comment 10 Robert Kaiser (not working on stability any more) 2010-02-26 17:25:51 PST
Now that the boxes we need are all online, I'll let them run a few cycles over night and then I'll turn all test suites on for debug builds on all platforms to fix this bug. \o/
Comment 11 Robert Kaiser (not working on stability any more) 2010-02-27 04:49:15 PST
http://hg.mozilla.org/build/buildbot-configs/rev/40fb1c3de8b5 checked in the config changes, tests should come up now!

\o/

Note You need to log in before you can comment on or make changes to this bug.