Closed Bug 422817 Opened 16 years ago Closed 16 years ago

Request unittest be run on trunk thunderbird

Categories

(Mozilla Messaging Graveyard :: Release Engineering, defect)

Product:
Mozilla Messaging Graveyard
Component:
Release Engineering
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dmosedale, Assigned: gozer)

References

Details

One per platform for each existing tinderbox.
Do you have any specific tests to run with this? We could probably set it up to run Ts but Tp, Txul, Tdhtml? We've never run these with Thunderbird and might not even make sense there.

We'll need some more details here.
Mostly, we want infrastructure to start hanging new tests off of.  Any existing tests that are useful would be great though.  As you say, Ts seems likely to be in this set, and I bet Tunit too.  I know Standard8 wanted to see codesize numbers, though I'm not sure if that's a talos thing or not.  We'll soon have some other tests to run as well, people on the CC list will have more ideas, I'll bet...
(In reply to comment #2)
> Mostly, we want infrastructure to start hanging new tests off of.  Any existing
> tests that are useful would be great though.  As you say, Ts seems likely to be
> in this set, and I bet Tunit too.  I know Standard8 wanted to see codesize
> numbers, though I'm not sure if that's a talos thing or not.  We'll soon have
> some other tests to run as well, people on the CC list will have more ideas,
> I'll bet...

These could be turned on in the existing Thunderbird tinderboxes if you want. There are separate setups for the unit test machines ("make check" plus mochitest etc), and Talos (which is for perf testing only).

Talos involves bringing up and maintaining a bunch of standalone machines, it'd probably be best to get it up and running in a dev environment first, as the tests are all for Firefox and someone would need to add Thunderbird support, IIRC.

Anyway, you can turn on TUnit ("make check" unit tests), codesize (Z and or mZ), etc. in the Thunderbird tinderboxes right now and they should work. For perf testing, you really need physical machines not VMs, which is what Talos is good for; but the existing tests need to be ported to Thunderbird first.
Code size is covered by bug 422214 for the current tinderboxes (awaiting Dan's review ;-) ).

I think TUnit and mochitest are the ones we really want to be able to run at the moment - I think we're still a few months off perf testing, unless someone steps up to do that work.

TUnit currently won't pass (bug 406227), and mochitest won't run (I've gotta file a bug on this, but there are already ideas on m.d.quality).

If we got some more VMs specifically for running the unit tests (which I think we'll need as we want mochitest as well) then we could run them to a different tree (or even the same one) for the time being and get a visible representative view of TUnit without causing the current Tinderboxes to fail (and hence not generate nightlies). We'll then be in a situation where we can start up mochitest as soon as we are ready to go.
Can these tests be run on a "normal" Firefox buildbot slave? Or do you require a Thunderbird specific slave? We're moving from purpose-specific-tinderbox-slaves to using a pool of generic-buildbot-slaves.
Component: Build & Release → Release Engineering: Projects
QA Contact: build → release
Updating summary to match comment#4
Summary: request talos instances be setup for trunk thunderbird → request unittest and talos be run on trunk thunderbird
waiting on test harnesses before we can begin work on this.
Priority: -- → P3
robcee: can you add details about what exactly we're waiting on, what sort of time frame we're looking at, and whether there's anything MoMo can do to help unblock...

(In reply to comment #8)
> robcee: can you add details about what exactly we're waiting on, what sort of
> time frame we're looking at, and whether there's anything MoMo can do to help
> unblock...

With regards to unittests, in comment #8, Mark said TUnit currently won't pass and mochitest won't run because it needs to be rearchitected or an alternative rewritten to work in Thunderbird. IOW, there's nothing to automate at this time.
(In reply to comment #9)
> With regards to unittests, in comment #8, Mark said TUnit currently won't pass
> and mochitest won't run because it needs to be rearchitected or an alternative
> rewritten to work in Thunderbird. IOW, there's nothing to automate at this
> time.

Dan and I discussed this on irc. Would it be possible to have one unit test machine reporting to MozillaTest whilst we work on getting the tests passing?

Based on the discussion in bug 406227, we could probably be looking at getting patches up and the testing progressed in the next 1-3 weeks (subject to reviews etc).
(In reply to comment #10)
> (In reply to comment #9)
> > With regards to unittests, in comment #8, Mark said TUnit currently won't pass
> > and mochitest won't run because it needs to be rearchitected or an alternative
> > rewritten to work in Thunderbird. IOW, there's nothing to automate at this
> > time.
> 
> Dan and I discussed this on irc. Would it be possible to have one unit test
> machine reporting to MozillaTest whilst we work on getting the tests passing?

Do you anticipate running these new Thunderbird unit tests on the same machines that are currently running Firefox unit tests? Or do you have other machines available for us to start setup work on?
Depends on: 406227
Given the current comments on suitability of Talos, whether that's what we want and the fact we are still a few weeks (I expect) before having got any performance tests set up, removing the option for Talos for the time being - we can always file a separate bug on that.

Reassigning to the new MoMo IT guy Gozer - we'd like the unit tests running on all three platforms. These will fail currently so we'll need them reporting to a different tree to being with, but we can talk specifics when we are closer to having them set up.
Assignee: nobody → gozer
Summary: request unittest and talos be run on trunk thunderbird → Request unittest be run on trunk thunderbird
I imagine it's mostly a matter of getting 3 VMs up and running, running this code and reporting back to some existing server ?

If that's the case, where should I be looking at the VM images ? I imagine there are existing ones (or reference ones) I can base this from ?
(In reply to comment #13)
> I imagine it's mostly a matter of getting 3 VMs up and running, running this
> code and reporting back to some existing server ?

Except for a Mac which you can't run in a VM.
 
> If that's the case, where should I be looking at the VM images ? I imagine
> there are existing ones (or reference ones) I can base this from ?

You could probably get some VM images/set up information from the Mozilla Corporation IT guys (I'm told #build is a good channel to ask on).

The three platforms are run with buildbot, there is a buildbot master on one of the "machines" (on the SeaMonkey unit test boxes, this is the Linux one).

On a side note, which you'll probably find out anyway, most (all?) of the tinderbox configurations are already in cvs (mozilla/tools/tinderbox-configs and other places?) and it'll be easier for us if we maintain that with the MoMo tinderboxes.
Component: Release Engineering: Future → Release Engineering
Priority: P3 → --
Product: mozilla.org → Mozilla Messaging
QA Contact: release → release
And just to make sure, once things are building, what extra step is necessary to run the unittest ?

gmake -C objdir-*/mail check

Or something similar ?
(In reply to comment #15)
> And just to make sure, once things are building, what extra step is necessary
> to run the unittest ?
> 
> gmake -C objdir-*/mail check
> 
> Or something similar ?

If you are able to initially just run make check in objdir-*/mail then that would be useful, as doing a full make check currently will fail. I'd want to change it later to be the full make check once we have fixed those tests.

Though I was more expecting you to ask about the tinderbox/buildbot configuration changes...

I'm not sure if you're setting up tinderboxes or buildbots. The mozilla2 buildbot configs for the unit test machines are here: http://hg.mozilla.org/build/buildbot-configs/index.cgi/file/ac594f8c7668/mozilla2-unittest/

We'd only want to run the "MozillaCheck" step at the moment (which I believe is the make check option).

However its done, as long as we can process the log file and get the correct TUnit numbers up in a similar style to the current Firefox unit tests (http://tinderbox.mozilla.org/Mozilla2/) then I don't mind.
make check in objdir-*/mail should be fairly easy to accomplish.

I am setting up buildbots, for the record. These sample configs will be very usefull.

I believe it's fairly straightforward to send buildbot output into Tinderbox, so that should hopefully do the trick.
(In reply to comment #17)
> make check in objdir-*/mail should be fairly easy to accomplish.

Excellent. If you're struggling to get mac/win up and running, then just getting a linux box going and reporting to the Thunderbird tinderbox tree (http://tinderbox.mozilla.org/Thunderbird/) would be miles better than the zero coverage we have now.
Yup, I am working on that right now. I've discovered the MozillaTest tinderbox tree, so I don't need to be afraid to mess things up in the beginning.
At this point, I've got 3 buildbots running on MoMo hardware, happilly building trunk and running unit tests successfully.

See http://tinderbox.mozilla.org/MozillaTest/:
 - TB-Win32-Dep-Check
 - TB-Linux-Dep-Check
 - TB-Darwin-Dep-Check

I think they are pretty much ready, and I'd like to move them to report into the official Thunderbird tinderbox tree. Anybody think it's not a good idea ?
Gozer: awesome!  We should figure out what's up with the orangeness on the Linux box, but after we get that sorted out, moving them over sounds great.
The orangeness is caused by a gconf problem. Something happens while running the unit tests. Normally, gconf clients will automatically cause gconfd to spawn if needed, but it's not hapenning in this case. This results in a nasty core dump and the following whining:

(process:2567): GLib-GObject-CRITICAL **: gtype.c:2240: initialization assertion failed, use IA__g_type_init() prior to this function

(process:2567): GLib-GObject-CRITICAL **: g_object_new: assertion `G_TYPE_IS_OBJECT (object_type)' failed

(process:2567): GLib-GObject-CRITICAL **: g_object_ref: assertion `G_IS_OBJECT (object)' failed

The workaround I've used for now is to try and ensure gconfd is running for that user with cron, but it's not reliable enough, as gconfd shuts down quite aggressively when not in use.
(In reply to comment #22)
> The orangeness is caused by a gconf problem. Something happens while running
> the unit tests. Normally, gconf clients will automatically cause gconfd to
> spawn if needed, but it's not hapenning in this case. This results in a nasty
> core dump and the following whining:

Well it seems to have been green for most of today at least, though currently it seems to be red with:

Unexpected error:  exceptions.ValueError
make[1]: *** [buildsymbols] Error 1


Are the build configs in hg yet? (i.e. accessible to anyone with appropriate permissions to change)

Do they have clobber support? (I'm not actually sure if the main MoCo ones do, which does raise the question, how do we clobber buildbots if we really need to?)

I'd like to check over the build/mozconfigs (as appropriate), just to make sure we aren't doing anything silly, but apart from that Windows and Mac both look stable.

I would say that we can move Windows and Mac over now, but we need to be confident that the Linux box is also stable. I don't want to get into the situation where we have random failures.
There is something crashing buildsymols (bug 445090 for the details) and it's only intermittent. Of course, it's not really a necessary step in unit-testing, just something I added at some point to help in trying to reproduce a specific failure on OS X. I kept it there because it was one more step, and would be testing another build step that should be working all the time.

Configs are not in Hg yet, but will be shortly.

They certainly don't have clobber support right now (the main reason being the buildbot config I copied them from didn't have it either). I imagine it should be possible to add some form of clobber support. I'd first look at how MoCo does it for inspiration.

The mozconfigs will be in Hg, so I'll post info about them as soon as they are up.

The simplest thing to get 'stable' unittest buildbots would be to just disable the buildsymbols step for now and address what's broken about that separately.
I'm happy with disabling the buildsymbols step for now. We don't really need it on the unit test tinderboxes anyway I would expect.

We could also push the clobber support out till later - again I think we can probably survive without it for the time being, especially as they don't generate nightlies.
Having just thought about this a bit more, I think we can probably just move the reporting from MozillaTest to Thunderbird asap (once you've disabled the buildsymbols), and post-review the mozconfig/buildconfigs later - I think given the tests are passing, the current configuration should be good enough.
The Windows mozconfig is here, not as mentionned in commment #27

http://hg.mozilla.org/build/buildbot-configs/index.cgi/file/tip/thunderbird/mozconfig-win
(In reply to comment #27)
> It's in Hg now, see them here:

My Comments:

- Assuming we are keeping unit tests separate from nightlies (IMHO I think we do, as unit tests need different configurations), we don't need to pull mozilla/tools/update-packaging or have ac_add_options --enable-update-channel=nightly (and --enable-update-packaging where appropriate)

- I think we should be doing --enable-optimize and --disable-debug now we've got the problems sorted out. I think this should make the builds a bit faster, and ensure that we have builds that are as similar as possible to the nightlies.

- Given the nightly builds are running codesighs, I don't think we need to do it on the unit test boxes (the test isn't running anyway), so we can just drop the mozilla/tools/codesighs and --enable-codesighs.

- We don't need these in the unittest boxes as we won't be running breakpad/publishing builds:

# symbols for breakpad
export CFLAGS="-gstabs+"
export CXXFLAGS="-gstabs+"

- Object directory: mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir - Can we change this to /objdir-tb or /obj-tb ? The reason being is that when we move to hg, the .hgignore file will have ^obj- and ^objdir- defined, and I know that not having the objdir in .hgignore will slow hg down.

- Please file a follow-up bug for fixing ac_add_options --disable-vista-sdk-requirements. I don't think its a problem at the moment, but we're going to want it if we start doing our own nightlies.

- I'm not sure about using ccache, although I guess it may speed up the builds, as it is working I think we can leave it on for unit test boxes. It'd be useful to know what some of the Firefox build team think about it.
Removed debugging and added optimizations globally

http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/151
Renamed the objdir to objdir-tb should have been:

http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/154
--disable-vista-sdk-requirements is not even necessary anymore, as the correct SDKs are installed, so it was only needed temporarily. 

http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/155 
I'm now happy with the mozconfigs.

The windows and linux tinderboxes have just gone down with a strange error, could be due to clock skew? 
(In reply to comment #37)
> I'm now happy with the mozconfigs.
> 
> The windows and linux tinderboxes have just gone down with a strange error,
> could be due to clock skew? 
> 
I checked in some changes to interface files again this morning, and all three have broken again with the same error, so I clobbered the Linux build via thunderbot and set it off again, and it passed all the tests correctly.

I tried simulating this on my mac box (but with a debug build rather than an optimised one) but I couldn't reproduce. So I'm not quite sure what is happening here.
I've just pushed clobber support for all unittest buildbots, so you should now be able to clobber them all, if you ever need to.

Sounds like some of these changes are not getting rebuilt proprely in the dep build. Could be a build problem or a clock skew issue.
Moved to the main Tinderbox/Thunderbird tree in

http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/158

Will close once the tree is green for a few revolutions.
And it is indeed green.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.