406227 - Enable Unit Tests on a Thunderbird trunk tinderbox

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect, P2)

Description

[Mark Banner (:standard8)]

Now that bug 405147 has provided some initial unit tests for mailnews, we should enable unit testing on at least one of the Thunderbird trunk tinderboxes.

(SeaMonkey has already got xpcshell tests running http://tinderbox.mozilla.org/SeaMonkey/)

I would suggest getting a unit test box aka Firefox, but I don't think we need to that just yet as we haven't got anything complicated on the way, and I think the initial address book tests that we need can all be done at the xpcshell test level which is what SeaMonkey has running on their main tinderboxes.

As File I/O tends to be slow on Windows I'd suggest not using that one.

If we enable it on the Linux box, then we'll need to supply a packages file (like we do for windows) so that it doesn't package the test files that get generated. I'm guessing the same may apply for the Mac box, though I'm not fully sure about that.

So any thoughts/comments on this? If nothing major is raised, I'll put together a packages-static file for Thunderbird Linux and sort out what we'll need to enable where.

Comment 1

[Mark Banner (:standard8)]

Attachment: Allow tests to be built on dev builds of Thunderbird (checked in)

So I was starting to look at this and realised that although I'd removed --disable-tests from my mozconfig, I still wasn't getting the tests built (the address book tests I've done so far I've tested with SeaMonkey - but that's not too bad as I know the address book code is common.

Having done some investigation, it turns out that ENABLE_TESTS is forced to "nothing" in mail/confvars.sh, the default in configure.in is to build with tests, so having "ENABLE_TESTS=" in confvars.sh means you can't build Thunderbird with tests.

I then did some more digging, the Thunderbird trunk tinderboxes all have --disable-tests in their mozconfig files:

http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/thunderbird/win32/mozconfig
http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/thunderbird/linux/mozconfig
http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/thunderbird/macosx/mozconfig

So in summary, this'll affect personal builds only (where --disable-tests hasn't already been put into the mozconfig file). I'll do a post to mozilla.dev.apps.thunderbird when I check the patch in so that hopefully some folks will spot it.

It also prepares us for enabling unit tests on one or more of the tinderboxes once I've got the other issues sorted out (see comment 0) and checked that the tests pass.

Comment 2

[Mark Banner (:standard8)]

Just to let everyone know the current status of this bug.

Its currently possible to build Thunderbird with tests and run the xpcshell tests. The ones in /mailnews pass (as per SeaMonkey).

The ones for cookie testing fail because Thunderbird blocks any cookies apart from RSS. I haven't yet thought of a way of getting around this problem.

Comment 3

[Mark Banner (:standard8)]

(In reply to comment #2)
> The ones for cookie testing fail because Thunderbird blocks any cookies apart
> from RSS. I haven't yet thought of a way of getting around this problem.

Additionally, I found out yesterday the password manager tests will fail because Thunderbird doesn't build with FTP, and the PM tests use ftp to test some of the migration.

I'm starting to think of a few options:

1) Find a way just to run the /mailnews specific tests on tinderbox (yet another tinderbox hack :-( )
2) Complete migration to xulrunner and then xulrunner can do its tests and we can do ours (sort of connected to item 1).
3) Find a way to "miss out" core tests.
4) Add some extra code to the tests (in a TB specific sense) to provide the ftp protocol/enable cookies/whatever is required so that those tests pass.

Comment 4

[Magnus Melin [:mkmelin]]

Would it be possible (well, easy) to make tb build with FTP also? Would be needed e.g. for bug 273476. Possibly bug 390057 is also a result of lacking of ftp support.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Nominating to keep it on the radar.

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

4) seems like the right strategy here.  Core tests should not be written in such a way that they are executed but fail in the non-Firefox case.

Comment 7

[(not currently active) Ted Mielczarek]

Ideally, wouldn't we just ifdef MOZ_FTP or whatever around the tests, so they only get run if they can succeed?

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

Ted: sounds right w.r.t. the FTP tests.  The cookie tests presumably require a slightly different tack.

Comment 9

[Mark Banner (:standard8)]

bug 431146 is a debug only problem, not blocking this bug. I'm detailing debug-only problems here: http://wiki.mozilla.org/MailNews:Automated_Testing#xpcshell_tests

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

This bug appears fixed to me.  Resolving, and removing the blocking flag.