485032 - Set up boxes/builders to run the core reftest & crashtest tests at/near release times

Status: RESOLVED WONTFIX

(Mozilla Messaging Graveyard :: Release Engineering, defect, P3)

Description

[Mark Banner (:standard8)]

As discussed in the meeting today, we want to have some extra builders so that we can:

a) Run reftest and crashtest in the code freeze period before a release.
b) Be able to run reftest & crashtest on the candidate build for a release.

We may want to do just a) here and address b) later.

There's two ways we can do this:

1) Completely Separate builders, that do a dep build in the same way as the unit check boxes and then run tests.
2) Wait a while and use the new test-split-from-build mechanism that Firefox is using. This would mean that the existing unit test boxes could do the build, upload, then run make check, and then in parallel the new builds could download & run reftest/crashtest.

Bug 482768 would be needed (which we're fixing anyway) if we did option 2, and we would possibly want to wait for FF to do Bug 383136 before we did option 2 as well.

I think option 2 would probably be nicer in the long run and reduce cpu requirements (as well as potentially build times when enabled).

In any case, this is low priority, and if necessary we'll just run a few tests by hand for b3.

Comment 1

[Philippe M. Chiasson (:gozer)]

I think the simplest way to go about this will be to setup special builders on the master buildbot, with no scheduler attached, so they never normally run.

Then, when we want them running, we can either manually trigger them or enable a periodic scheduler to drive them for a few days then turn it off again.

Comment 2

[Mark Banner (:standard8)]

(In reply to comment #1)
> I think the simplest way to go about this will be to setup special builders on
> the master buildbot, with no scheduler attached, so they never normally run.

Agreed.

> Then, when we want them running, we can either manually trigger them or enable
> a periodic scheduler to drive them for a few days then turn it off again.

Yep, a manual push would be good so that we can trigger them now and again if we just want to check things are ok.

Comment 3

[Philippe M. Chiasson (:gozer)]

(In reply to comment #2)
> (In reply to comment #1)
> > Then, when we want them running, we can either manually trigger them or enable
> > a periodic scheduler to drive them for a few days then turn it off again.
> 
> Yep, a manual push would be good so that we can trigger them now and again if
> we just want to check things are ok.

Or possibly even simpler, keep a Scheduler enabled for them, but run it very slowly, like once every week, and simply increase the period when we are getting close to release time.

Comment 4

[Mark Banner (:standard8)]

I've just been playing around with the new "make package-tests" target. It looks like it will work for us if that's what we want to do - we just need to fix bug 489450 (patch already attached) so that Thunderbird works correctly with reftests and crashtests.

So in theory what I think we can do is add a step on our existing unit test masters to run "make package-tests" and upload the result. Then the new ref platforms can download the dep builds (or maybe a packaged unit test build) and run reftests and crashtests.

Comment 5

[Mark Banner (:standard8)]

Once bug 495299 is fixed, this could be relatively simple. Firefox have already got some of this working.

This is the heart of their config:

http://hg.mozilla.org/build/buildbot-configs/file/778a7e29d182/mozilla2/master.cfg#l478
http://hg.mozilla.org/build/buildbot-configs/file/778a7e29d182/mozilla2/config.py#l66

That is using UnittestPackagedBuildFactory:

http://hg.mozilla.org/build/buildbotcustom/file/86e4b0c992b9/process/factory.py#l3442

The only issues I've seen (without testing) is that hard-codes firefox, and assumes the firefox system for version numbers. I believe if we fix those issues, then getting the rest running shouldn't be too hard because afaik the commands should be exactly the same structure and directories as firefox.

Comment 6

[John Hopkins (:jhopkins)]

standard8: Is there anything left to be done here, now that we have packaged unit tests?

Comment 7

[Mark Banner (:standard8)]

Yes, set up reftest & crashtest builders ;-)

I'm not actually sure if this will create any significant benefit for us, but lets give it a try and see what happens.

I think in the effort of not overloading our build network at the moment, I would propose that we only enable reftest & crashtests on Thunderbird-Beta tree. That is sufficiently near release and low rate enough that we should be able to cope, in addition, it should be plenty of time for us to assess regressions before release.

We probably also want these on ThunderbirdTry if it isn't going to increase the end-to-end times too much (if we get the hg shared stuff, then that would help).

Comment 8

[Philippe M. Chiasson (:gozer)]

reftest & crashtest builders enabled on try for:

- macosx64
- win32
- linux64

Comment 9

[Mark Banner (:standard8)]

Not going to fix this at the current time, as I'm not sure it produces much value for us.