405007 - Create unittest buildbots for Calendar

Status: RESOLVED FIXED

(Calendar :: Build Config, defect)

Description

[Martin Schröder [:mschroeder]]

We don't use the Trunk tinderboxen for production at the moment, and the Mozilla1.8 tinderboxen can't run unit tests automatically. So we decided to enable automated unit testing on the Trunk tinderboxen in the QA chat.

What has to be done:
* Change Makefiles on Trunk
* Change config of Trunk tinderboxen

Comment 1

[Martin Schröder [:mschroeder]]

Attachment: [checked in] Patch for Makefile v1 (TRUNK ONLY)

We have to change the Makefile in the test directory on Trunk to run XPCShell unit tests with "make check" from the OBJDIR.

Comment 2

[Martin Schröder [:mschroeder]]

Comment on attachment 291122 [details] [diff] [review]
[checked in] Patch for Makefile v1 (TRUNK ONLY)

Shifting to Daniel for a quick review. I tested this patch thoroughly.

Comment 3

[Daniel Boelzle [:dbo]]

Comment on attachment 291122 [details] [diff] [review]
[checked in] Patch for Makefile v1 (TRUNK ONLY)

r=dbo, try it out

> # Note: Invoke any additional (non-xpcshell) test programs here.
> check::
>-	$(RUN_TEST_PROGRAM) $(DIST)/bin/test_all.sh $(DIST)/bin/$(MODULE)
Why have you left the check target in?

Comment 4

[Michiel van Leeuwen (email: mvl+moz@)]

(In reply to comment #3)
> > # Note: Invoke any additional (non-xpcshell) test programs here.
> > check::
> >-	$(RUN_TEST_PROGRAM) $(DIST)/bin/test_all.sh $(DIST)/bin/$(MODULE)
> Why have you left the check target in?

I personally like it, because of the comment. We might want to add some extra tests here (I have a few ideas for that, wanting some way to measure performance)

Comment 5

[Martin Schröder [:mschroeder]]

Comment on attachment 291122 [details] [diff] [review]
[checked in] Patch for Makefile v1 (TRUNK ONLY)

Checked in on HEAD.

Comment 6

[Mark Banner (:standard8)]

I don't see how this affects TB-integration. This is essentially a Sunbird bug. What you really want to get working/passing for Lightning integration is any tests which are run in calendar/ when you do "make check". I think this bug covers more than that.

Comment 7

[Stefan Sitter [:ssitter]]

(In reply to comment #6)
As far as I know Thunderbird now requires unittests. If Lightning gets integrated the same rules should be applied to the calendar code. If this isn't required than this bug is probably not blocking the integration.

Comment 8

[David Ascher (:davida)]

I don't think we'd block based on this bug.

Comment 9

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: Make xpcshell tests work when building with Thunderbird/Lightning

As a first step, we should make sure that unit tests work. This patch allows running make xpcshell-tests when building Lightning with Thunderbird and also fixes our unit tests.

I have a strange error on one test that seems to be fixed by adding a dump() statement. This is probably some sort of timing issue, but I haven't found out why.

Comment 10

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: [backed out] Make unit tests work with comm-central

comm/mozilla-central now requires the use of xpcshell.ini, see bug 658666. This patch takes care and uses buildlist.py to include the lightning tests in the app xpcshell.ini. Not sure if the latter fully needed, but I had the feeling it is.

A slightly modified version could be checked in on comm-miramar.

Comment 11

[Martin Schröder [:mschroeder]]

Comment on attachment 534892 [details] [diff] [review]
[backed out] Make unit tests work with comm-central

This seems to work fine after adding the two additional test files (test_webcal.js and test_bug668222.js). r=mschroeder

Comment 12

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 534892 [details] [diff] [review]
[backed out] Make unit tests work with comm-central

changeset:   8242:ec28666abc48
user:        Philipp Kewisch <[e-mail]>
date:        Fri Aug 05 21:00:48 2011 +0200
summary:     Bug 405007 - Create unittest buildbots for Calendar. r=mschroeder


Keeping open for buildbot work.

Comment 13

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Undid accidental mail changes in comm-central changeset 8951a95a0755

Comment 14

[Philip Chee]

Fallen FYI:

Building SeaMonkey trunk with
ac_add_options --enable-calendar
fails since
/suite/test/xpcshell.ini is missing the folllowing line:
[include:calendar/test/unit/xpcshell.ini]

Comment 15

[Justin Wood (:Callek)]

Someone, please back this out (r=bustage) per me.

(In reply to Philipp Kewisch [:Fallen] from comment #13)
> Undid accidental mail changes in comm-central changeset 8951a95a0755

This was caused due to buildlist.py pointing at the SRCDIR and modifying the thunderbird xpcshell.ini when it shouldn't, keeping this in will cause someone else to checkin that change by accident, I guarantee it.

(In reply to Philip Chee from comment #14)
> Fallen FYI:
> 
> Building SeaMonkey trunk with
> ac_add_options --enable-calendar
> fails since
> /suite/test/xpcshell.ini is missing the folllowing line:
> [include:calendar/test/unit/xpcshell.ini]

This is sadly also a reason I want you to backout, please do $(MOZ_BUILD_APP) if you need to reference a file in mail/ (that is also in suite/) otherwise TEST on MOZ_BUILD_APP here.

Thanks

Comment 16

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Backed out, sorry for the hassle.

changeset:   8250:8e61adaa68ab
tag:         tip
user:        Philipp Kewisch <[e-mail]>
date:        Sat Aug 06 19:27:00 2011 +0200
summary:     Backout bug 405007 - Create unittest buildbots for Calendar r=bustage


(In reply to Justin Wood (:Callek) from comment #15)
> Someone, please back this out (r=bustage) per me.
> 
> (In reply to Philipp Kewisch [:Fallen] from comment #13)
> > Undid accidental mail changes in comm-central changeset 8951a95a0755
> 
> This was caused due to buildlist.py pointing at the SRCDIR and modifying the
> thunderbird xpcshell.ini when it shouldn't, keeping this in will cause
> someone else to checkin that change by accident, I guarantee it.
Seems this was changed in the meanwhile, but iirc then mail was doing something similar to copy over the mozilla/testing xpcshell.ini. I may be mistaking though.

> 
> (In reply to Philip Chee from comment #14)
> > Fallen FYI:
> > 
> > Building SeaMonkey trunk with
> > ac_add_options --enable-calendar
> > fails since
> > /suite/test/xpcshell.ini is missing the folllowing line:
> > [include:calendar/test/unit/xpcshell.ini]
> 
> This is sadly also a reason I want you to backout, please do
> $(MOZ_BUILD_APP) if you need to reference a file in mail/ (that is also in
> suite/) otherwise TEST on MOZ_BUILD_APP here.
Ok, will do so in the next iteration.

Comment 17

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Forgot to correctly remove the file, changeset d57afbc83f6b

Comment 18

[Mark Banner (:standard8)]

Attachment: [checked in] Alternate temporary get tests runnable

This is a slightly different approach.

As discovered in bug 659898 we need to add our own stuff to mozinfo.js which may be a bit more complicated, but doable, just not got time.

The basic idea here is to skip the calendar tests if we're running TB's or SM's full suite of tests. That's because we haven't got anything into mozinfo.js to say if calendar/lightning is built yet or not. However, by doing this, we've got an entry in the relevant xpcshell.ini which keeps the build system happy.

Although we're skipping the tests in the main suite, we can still run them with:

make -C objdir/calendar/test xpcshell-tests

which means that a) the tests can actually be run, b) we can hack buildbot to make them run, c) they become useful.

We can then do a follow-up later to make a switch so that the tests will be run if Lightning is being built.

I've tested both configurations and the build system doesn't complain, and the tests run fine.

Comment 19

[Mark Banner (:standard8)]

Oh note that test_webcal.js fails for me on my Mac, which is why I disabled it (we can deal in a follow-up if you wnat).

Comment 21

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 552359 [details] [diff] [review]
[checked in] Alternate temporary get tests runnable

Looks fine, sorry for the delay. r=philipp

Comment 22

[Mark Banner (:standard8)]

Comment on attachment 552359 [details] [diff] [review]
[checked in] Alternate temporary get tests runnable

Checked in:

http://hg.mozilla.org/comm-central/rev/5826442dafae

with a follow-up that completely disables test_webcal.js due to an error, which bug 680201 will fix:

http://hg.mozilla.org/comm-central/rev/1f48f873106e

Comment 23

[Mark Banner (:standard8)]

I've now enabled building of tests for all the Lightning comm-central builders:

http://hg.mozilla.org/build/buildbot-configs/rev/2909046e0a50
http://hg.mozilla.org/build/buildbot-configs/rev/de8369b8cd85

I've also done a local addition to buildbot's master.cfg just after the upload step:

# XXX Hack to allow calendar to run xpcshell-tests
if mozilla_branch == "mozilla-central":
     mozilla2_dep_factory.addStep(unittest_steps.MozillaCheck,
                                  test_name="xpcshell-tests",
                                  warnOnWarnings=True,
                                  workdir="build/%s/calendar/test" % pf['platform_objdir'],
                                  timeout=5*60, # 5 minutes.
                                  )

This allows the Lightning builders to just run the Lightning xpcshell-tests. This is in a non-packaged form, so will extend the time of the build by a small amount. This is the easiest way to get them running without adding too much hardware time and needing more hardware.

So far only the Linux 64 bit builder has been running. I've only just enabled the others, but I'm confident that they should run correctly.

Philipp: if you want to run these on aurora, that is easy to do as well.

Comment 24

[Mark Banner (:standard8)]

The Mac tests completely failed, I'm not sure why, but it may be something to do with the universal build. I've disabled them until I get chance to look in more detail.

Comment 25

[Mark Banner (:standard8)]

Windows and Linux stuck and are running nicely green. I'll try and take a look at Mac next week.

Comment 26

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Mark, I'd appreciate if we could transplant these changes to both aurora and beta, that would have calendar in sync across all trees. The additional fix from bug 680742 should also be transplanted. Any objections?

Comment 27

[Mark Banner (:standard8)]

(In reply to Philipp Kewisch [:Fallen] from comment #26)
> Mark, I'd appreciate if we could transplant these changes to both aurora and
> beta, that would have calendar in sync across all trees. The additional fix
> from bug 680742 should also be transplanted. Any objections?

No objections, I can transplant them if you want. Once these are on aurora, I'll also enable xpcshell-tests there.

Comment 28

[Stefan Sitter [:ssitter]]

As long as it doesn't cause the same regression as on the win32 nightly comm-central builder (Bug 682540) ...

Comment 29

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to Mark Banner (:standard8) from comment #27)
> (In reply to Philipp Kewisch [:Fallen] from comment #26)
> > Mark, I'd appreciate if we could transplant these changes to both aurora and
> > beta, that would have calendar in sync across all trees. The additional fix
> > from bug 680742 should also be transplanted. Any objections?
> 
> No objections, I can transplant them if you want. Once these are on aurora,
> I'll also enable xpcshell-tests there.

Go ahead, and please clobber afterwards. That seemed to fix the win32 builder.

Comment 30

[Mark Banner (:standard8)]

Checked into aurora and beta:

http://hg.mozilla.org/releases/comm-aurora/rev/092d1cb3a091
http://hg.mozilla.org/releases/comm-aurora/rev/dda730b7af4a
http://hg.mozilla.org/releases/comm-beta/rev/de55382cc023
http://hg.mozilla.org/releases/comm-beta/rev/7e9fc8c8b8f1

Comment 31

[Mark Banner (:standard8)]

I've now added enable-tests to just the linux & mac builds on aurora:

http://hg.mozilla.org/build/buildbot-configs/rev/d83f4e213cd6

and I've altered buildbot so that unit test should get run on linux 32 & 64 bit builds on aurora as well as on central. Builders are currently busy, so it may be a while before we can tell.

Comment 32

[Mark Banner (:standard8)]

Linux (32 & 64 bit) unit tests are now running fine on comm-aurora.

Comment 33

[Mark Banner (:standard8)]

Now we've got the Windows builder green on trunk, I've turned on enable-tests for comm-aurora and comm-beta for it:

http://hg.mozilla.org/build/buildbot-configs/rev/d76ae177e5b9

If this runs fine over the next day, then I'll enable running the tests.

Comment 34

[Mark Banner (:standard8)]

I've locally modified buildbot master to allow tests to additionally run on Windows comm-aurora and on Windows and Linux comm-beta.

Comment 35

[Mark Banner (:standard8)]

(In reply to Mark Banner (:standard8) from comment #34)
> I've locally modified buildbot master to allow tests to additionally run on
> Windows comm-aurora and on Windows and Linux comm-beta.

Linux comm-beta ran fine. Now just waiting for Windows to run on aurora and beta.

Comment 36

[Mark Banner (:standard8)]

I've spun the Mac issue out into bug 689126.

Comment 37

[Mark Banner (:standard8)]

Windows tests ran fine. So Linux & Windows tests are on all branches:

http://hg.mozilla.org/build/buildbot-configs/rev/585f8dcb13c5

Mac is to be handled by bug 689126.