Closed Bug 931279 Opened 8 years ago Closed 8 years ago

Allow people to start running at a given test

Categories

(Testing :: Mochitest, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla28

People

(Reporter: billm, Assigned: billm)

Details

Attachments

(2 files, 1 obsolete file)

Attached patch test-subset (obsolete) — Splinter Review
Sometimes a test fails because of something done by a previous test. In that case, you want to run all tests from the previous test to the failing test. If the tests are in the same directory, you can use --test-path to do this. But if they're in different directories, then you often end up running the entire test suite just to reproduce the problem.

This patch adds new options --start-at and --end-at to runtests.py. They allow you to specify which tests to start and end on. For example, if the list of all tests is:

a/b/c
a/b/d
a/b/e
f/g/h
f/g/i

Then specifying --start-at=a/b/d --end-at=f/g/h will run these tests:

a/b/d
a/b/e
f/g/h

Additionally, you can just specify a suffix of the name, so --start-at=d --end-at=h also works.
Attachment #822641 - Flags: review?(jmaher)
thank you for this - will really help people with tracking down intermittents :-)
Comment on attachment 822641 [details] [diff] [review]
test-subset

Review of attachment 822641 [details] [diff] [review]:
-----------------------------------------------------------------

this looks good.  have you tested this in chrome, browser-chrome, and mochitest-plain?  Also with chunking added to each of those?  The code looks as though it will handle all of those scenarios well.

I assume you need to add this to mach: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/mach_commands.py#350
Attachment #822641 - Flags: review?(jmaher) → review+
Attached patch test-subset v2Splinter Review
This adds mach support. It also makes the chunking apply after the skipTests stuff. So if you ask it to run chunk 1/2 and --start-at and --end-at specify a sequence of 20 tests, then you'll run the first 10 of the 20. Previously it did the chunking first, which didn't make much sense.

I tested with all three kinds of mochitests and it seems to work.
Attachment #822641 - Attachment is obsolete: true
Attachment #822854 - Flags: review?(jmaher)
Comment on attachment 822854 [details] [diff] [review]
test-subset v2

Review of attachment 822854 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the mach commands.  Personally I really don't like the switched method of startat/endat then chunking.  I would imagine folks would want to chunk first (to run mochitest 4 as we do on tbpl), then run a subset of those.  There is a use case for doing it the other way as this patch does, but I see that as less common.

r+ as the patch is fine. Please let me know the use case you are solving by slicing first then chunking.
Attachment #822854 - Flags: review?(jmaher) → review+
> I would imagine folks would want to chunk first (to run mochitest 4 as we do on tbpl), then run a subset of those.

That doesn't seem too useful to me. Let's say there are 10 tests, divided into two chunks (1-5 and 6-10). You want to run tests 7 thtough 9. If you just say --start-at=<name-of-test-7> --end-at=<name-of-test-9>, then there's no need to specify the chunk.

The only way it would help is if multiple tests match the argument you give for --start-at, but only one test in the chunk set matches. That doesn't seem too useful, though, because you can always specify more of the pathname to the test in order to disambiguate. Also, our tests tend to have unique names.

> Please let me know the use case you are solving by slicing first then chunking.

It's not really very useful either, but it might help if you're doing some sort of bisection. You could start with a given range of tests with --start/end-at and then narrow it down using chunking.

Anyway, I don't think it really matters much. I doubt anyone will ever combine chunking with these new parameters.
https://hg.mozilla.org/mozilla-central/rev/9c30148393d1
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28
Attached patch mach support fixSplinter Review
small follow-up:

I just noticed that the mach support command definitions ended up defined inside B2GCommand instead of MochitestCommand, making that not usable on desktop. (I imagine it was not on purpose)

the end-at function is defined first because mach help shows them in reversed order, and it looks nicer to have --start-at first in the help
Attachment #824877 - Flags: review?(jmaher)
This is very useful.  Thank you!

Can you provide an example of how to do this?  I've tried all the following with no success:
./mach mochitest-plain toolkit/components/passwordmgr/test --start-at=toolkit/components/passwordmgr/test/test_notifications_popup.html --end-at=toolkit/components/passwordmgr/test/test_prompt_async.html 


./mach mochitest-plain --start-at=toolkit/components/passwordmgr/test/test_notifications_popup.html --end-at=toolkit/components/passwordmgr/test/test_prompt_async.html 

./mach --start-at=toolkit/components/passwordmgr/test/test_notifications_popup.html --end-at=toolkit/components/passwordmgr/test/test_prompt_async.html 

./mach test --start-at=toolkit/components/passwordmgr/test/test_notifications_popup.html --end-at=toolkit/components/passwordmgr/test/test_prompt_async.html 

./mach test mochitest-5 --start-at=toolkit/components/passwordmgr/test/test_notifications_popup.html --end-at=toolkit/components/passwordmgr/test/test_prompt_async.html
Comment on attachment 824877 [details] [diff] [review]
mach support fix

Review of attachment 824877 [details] [diff] [review]:
-----------------------------------------------------------------

we both missed it, good find.
Attachment #824877 - Flags: review?(jmaher) → review+
https://hg.mozilla.org/integration/fx-team/rev/9720ae71914d

Tanvi: with this follow-up fix, your example 2 is the correct entry and should now work (when it makes to m-c). Example 1 should also work but it's unnecessary to specify the folder.

you don't even need the full path in the params, so simplifying:

./mach mochitest-plain --start-at=test_notifications_popup.html --end-at=test_prompt_async.html 

as long as "test_notifications_popup.html" and "test_prompt_async.html" are unique names in the test suite.
You need to log in before you can comment on or make changes to this bug.