417516 - add top level Makefile targets to run test suites

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[(not currently active) Ted Mielczarek]

Right now running mochitest or reftest requires knowing an arcane commandline.  We should just have "mochitest" and "reftest" targets in the top level makefile.

Comment 1

[(not currently active) Ted Mielczarek]

Attachment: add "mochitest" target, and targets for each mochitest variant

Comment 2

[(not currently active) Ted Mielczarek]

Comment on attachment 327627 [details] [diff] [review]
add "mochitest" target, and targets for each mochitest variant

Ack, hit enter too early. This is a rough draft of a patch that adds a "mochitest" target, to run all the mochitest variants, as well as a target per-variant. It's not terribly useful yet, since it won't error on failure, I don't think.

Comment 3

[(not currently active) Ted Mielczarek]

Attachment: somewhat more complete

This works for the basic mochitest case. browser-chrome output is different, so it won't work there. I haven't tried the a11y tests yet.

We need to get our test suites on the same page. :-/

Comment 4

[(not currently active) Ted Mielczarek]

Attachment: handles mochitest correctly

Well now that I've done a bunch of work in bug 443090, this is easier! The runtests.py changes are because I didn't fix that problem for browser-chrome when I fixed it for regular mochitest, apparently.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 328898 [details] [diff] [review]
handles mochitest correctly

I think this is good to go now.

Comment 6

[Benjamin Smedberg]

Comment on attachment 328898 [details] [diff] [review]
handles mochitest correctly

Are you sure you want to have all of these targets for all products? (aren't the browser-chrome tests Firefox-specific?) If you are, r=me... otherwise maybe we should put that part in browser/build.mk

Comment 7

[(not currently active) Ted Mielczarek]

Good call, I'll move mochitest-browser to browser/build.mk. The other stuff won't work on a non-browser app, but I do want Firefox and SeaMonkey to both have access.

Comment 8

[(not currently active) Ted Mielczarek]

Attachment: mochitest targets, as checked in (the first time)

Made some tweaks and checked this in. Leaving the bug open to handle reftest.

Comment 9

[Shawn Wilsher :sdwilsh]

Are these documented somewhere (how to use them)?

Comment 10

[(not currently active) Ted Mielczarek]

No, I didn't get around to that yet. Also apparently I got backed out last night.

Comment 11

[(not currently active) Ted Mielczarek]

vlad found a bug in this, if you run runtests.py without --log-file, it breaks mochitest. That might be what was causing problems yesterday. (Oops!)

Comment 12

[(not currently active) Ted Mielczarek]

Attachment: [checked in] fixed the bug in runtests.py

This fixes that stupid oversight in runtests.py. I'll push this when the tree goes green again...

Comment 13

[(not currently active) Ted Mielczarek]

Comment on attachment 331309 [details] [diff] [review]
[checked in] fixed the bug in runtests.py

Pushed to mozilla-central:
http://hg.mozilla.org/mozilla-central/index.cgi/rev/409570e09968

Comment 14

[Justin Wood (:Callek)]

ted, this resolved or should it stay open?

Comment 15

[Arpad Borsos [:Swatinem]]

So far only "make mochitest" is supported. Something like "make reftest" is still missing.

Comment 16

[Nickolay_Ponomarev]

Note, documentation should be updated on at least:
https://developer.mozilla.org/En/Mochitest (I partially updated this, but deki is too frustrating for me)
https://developer.mozilla.org/en/Chrome_tests
https://developer.mozilla.org/en/Browser_chrome_tests

Ideally, there should be a single document on running all the kinds of tests we have (aka what to do before checking in).

Comment 17

[(not currently active) Ted Mielczarek]

Waldo filed bug 468913 on reftest, so might as well close this.

Comment 18

[Eric Shepherd [:sheppy]]

Someone has already updated https://developer.mozilla.org/En/Mochitest to cover this.