497353 - Thunderbird try server should also run unit tests

Status: RESOLVED FIXED

(Mozilla Messaging Graveyard :: Try Server, enhancement, P2)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Thunderbird try server should also run unit tests. Cross reference bug 445611 and/or bug 464256.

These unit tests should also have stacks. xref bug 483111

Also note running "packaged unittests on tryserver" at bug 493228

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

That said, I'm not sure how applicable this is, as TB tryserver stuff is on Amazon S2...

Although, it could offer the download on compile completion, then start (optional?) unit testing after that.

Comment 2

[Philippe M. Chiasson (:gozer)]

100% feasible, no reason not to be doing it. I haven't gotten around to porting that part of the try-server logic over to the comm-central way of doing things, that's all.

Comment 3

[Mark Banner (:standard8)]

I'm going to bump this up the priority scale. If we're now requiring tests and we want to avoid landings and backouts due to broken tests (especially MozMill which can be a bit more sensitive to different platforms), we're going to need a try server capable of doing unit tests.

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

Completely agreed that this is important.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Where in the pantheon of priorities does this dwell?  I'm starting to get patch review requests where the patches are not important enough for me to justify the effort of my running mozmill tests locally on all 3 platforms, but where the risk is high enough that I am also unwilling to just have the patch land and potentially require a backout.

I would personally rank this as pretty important because I am going to appear increasingly unresponsive to review requests without this.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Er, and I want it to run mozmill tests too.  Not sure if this bug exactly covers that.

Comment 7

[Philippe M. Chiasson (:gozer)]

(In reply to comment #5)
> Where in the pantheon of priorities does this dwell? 

The top build engineering priority right now, aside from regular scheduled releases is to get the new tryserver out the door.

We are pretty close to being ready (days, not weeks), and that includes unit tests.

> I'm starting to get patch
> review requests where the patches are not important enough for me to justify
> the effort of my running mozmill tests locally on all 3 platforms, but where
> the risk is high enough that I am also unwilling to just have the patch land
> and potentially require a backout.

Running mozmill tests might be a bit more complicated, but I don't see any significant problems with adding them to the unittest run, maybe some time after the new tryserver is 'out'

> I would personally rank this as pretty important because I am going to appear
> increasingly unresponsive to review requests without this.

Understood.

Comment 8

[Mark Banner (:standard8)]

(In reply to comment #7)
> (In reply to comment #5)
> > I'm starting to get patch
> > review requests where the patches are not important enough for me to justify
> > the effort of my running mozmill tests locally on all 3 platforms, but where
> > the risk is high enough that I am also unwilling to just have the patch land
> > and potentially require a backout.
> 
> Running mozmill tests might be a bit more complicated, but I don't see any
> significant problems with adding them to the unittest run, maybe some time
> after the new tryserver is 'out'

gozer, what's the issue with MozMill tests? If it helps, then with a small bit of work we should be able to get those working as packaged tests.

Comment 9

[rsx11m]

Is there any update or ETA on running MozMill tests on the tryservers?

Keep in mind that many volunteer contributors have just a single platform available for development and testing, so this would provide some relief from
the pain the new test requirements impose, especially as a test may run fine
on one platform but fail on another one.

Comment 10

[Mark Banner (:standard8)]

(In reply to comment #9)
> Is there any update or ETA on running MozMill tests on the tryservers?

As per the weekly status meetings and individual updates, we're working on it.

Current ETA for the new try server is to get it out early this week, hopefully with MozMill tests coming later this week.

> Keep in mind that many volunteer contributors have just a single platform
> available for development and testing, so this would provide some relief from
> the pain the new test requirements impose, especially as a test may run fine
> on one platform but fail on another one.

We know, that is why the new try server has been #1 priority for the build team for the last month or so. It is also why I have been prioritising fixing our MozMill tests for the new try server over switching to libxul for the last couple of weeks, despite the fact the general health of the tree really needs the latter.

Comment 11

[Mark Banner (:standard8)]

Attachment: Enable xpcshell-tests in package-test builders

I've got this running on try server at the moment with a patch to buildbot-custom.

This patch renames the builders "unit" (from "mozmill"), and enables running of both xpcshell-tests and mozmill on those builders.

I'm currently testing the two patches for this bug, with the one for the code from bug 599305 and it is working fine so far...

Comment 12

[Mark Banner (:standard8)]

Attachment: Valid builders update for buildbotcustom

This previously said "mozmill" in this addition on the local changes you had in buildbotcustom. I'm not proposing we check this in at this stage (unless you want to), just putting it here ftr.

Comment 13

[Philippe M. Chiasson (:gozer)]

Wouldn't it make more sense to have a set of builders doing mozmill only, and another one doing xpcshell tests ?

I just mean that in the current structure of the try stuff, that's the way they are done in Firefox land.

Comment 14

[Mark Banner (:standard8)]

Turning your bit around:

(In reply to comment #13)
> I just mean that in the current structure of the try stuff, that's the way they
> are done in Firefox land.

Generally, although they also have "Other" which has three different sets of tests in it.

> Wouldn't it make more sense to have a set of builders doing mozmill only, and
> another one doing xpcshell tests ?

I think it is a trade off over the amount of time spent downloading and extracting the executable and tests, versus the responsiveness of the results.

As I see you've adjusted try server, I'll push again with the patch so we can get some timing results.

(If you can review bug 599305 as well, then I can push that to c-c as well).

Comment 15

[Mark Banner (:standard8)]

(In reply to comment #14)
> > Wouldn't it make more sense to have a set of builders doing mozmill only, and
> > another one doing xpcshell tests ?

Having looked at the timings, yes I agree it would be better to have one set of builders for mozmill and one for xpcshell.

Comment 16

[Mark Banner (:standard8)]

Comment on attachment 478235 [details] [diff] [review]
Enable xpcshell-tests in package-test builders

This is already live in a slightly modified form and gozer is pushing to the hg repo.

Comment 17

[Philippe M. Chiasson (:gozer)]

changeset:   3024:886cec85dca5
tag:         tip
user:        Philippe M. Chiasson <[e-mail]>
date:        Mon Sep 27 16:35:02 2010 -0400
summary:     Thunderbird Try fixes