506202 - Hook up mozmill tests to make package-tests and support running packaged test style

Status: RESOLVED FIXED

(Thunderbird :: Testing Infrastructure, defect)

Description

[Mark Banner (:standard8)]

Attachment: WIP

Bug 500142 added the "make mozmill" and "make mozmill-one" targets to the build system. It would be useful to extend the make structures so that the mozmill test files can be integrated into "make package". This would mean that testers (or other tinderboxes) could download a package and run the mozmill tests against a build. It would also mean testers could extend our tests and have the harness to make it easy to do so.

The general idea is this:

- Hook into the make package target, copy our test files to the dist/stage-package directory.
- Ensure that all tests reference support files relatively so that they can be run from within the package.

I had an initial attempt at this (see attached patch), I got a bit stuck when the tests started referencing the xpcshell support tests. As we didn't need this to get make mozmill & mozmill-one working, I left it.

I'm not planning on working on this in the near future - just filing it here so the patch doesn't get lost.

Comment 1

[Mark Banner (:standard8)]

Attachment: Core WIP

I almost forgot, this patch would be needed for core. I'm not sure it is the best way of doing it, but it worked ;-)

Comment 2

[Mark Banner (:standard8)]

Attachment: Get just mozmill packaging working

Slightly different approach for now - until we get some form of libxul, the MozMill tests are the only ones that we can run in packaged form - xpcshell-tests need libxul and make check run on the same machine anyway.

Therefore this patch ignores the fact that we possibly need to sort out hooks for the mozilla-central stuff, and just enables package-tests for our mozmill tests. This way we can get try server + MozMill up and running quickly, and sort out xpcshell etc later once we've done libxul.

Andrew: I'll get gozer to review the build config bits, but can you review the mozmill changes please?

The basic idea of the mozmill changes is:

- Package tests really need a different structure and to package all the resource files.
- I didn't want to switch MozMill to running from objdir as Windows developers don't have symlinks, and this would be required for ojbdir. We could probably work around this with make mozill etc, but that seems slow and complicated.
- Therefore the tests attempt to load the files from a pre-set list of directories, if they don't find it, then they will report an error so that developers can easily fix it.

Comment 3

[Mark Banner (:standard8)]

Comment on attachment 464039 [details] [diff] [review]
Get just mozmill packaging working

This seems to be working on try, so also requesting gozer's review.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 464039 [details] [diff] [review]
Get just mozmill packaging working

windows users can use symlinks, they just need to escalate privileges and maybe use hardlinks...

in any event, looks good.

Comment 5

[Mark Banner (:standard8)]

I checked this in yesterday and it is working fine:

http://hg.mozilla.org/comm-central/rev/6bf5559f73a7

Comment 6

[Mark Banner (:standard8)]

Comment on attachment 464039 [details] [diff] [review]
Get just mozmill packaging working

a=Standard8 as I want to take onto comm-1.9.2 to support running MozMill tests on try server there.

Comment 7

[Mark Banner (:standard8)]

Checked into 1.9.2: http://hg.mozilla.org/releases/comm-1.9.2/rev/2684b4411b03

Tinderbox still passes.