Add a test verification mode to the xpcshell test harness

RESOLVED FIXED in Firefox 57


10 months ago
10 months ago


(Reporter: gbrown, Assigned: gbrown)


(Blocks: 1 bug)


Firefox Tracking Flags

(firefox57 fixed)



(3 attachments)



10 months ago in bug 1357520, but for xpcshell instead of mochitest.

This bug aims to add a --verify option for the xpcshell test harness, supporting test verification. Test verification involves running a test, or small group of tests, repeatedly and possibly in different environments, with the intent of finding intermittent test failures quickly and efficiently.

Comment 1

10 months ago
Created attachment 8897652 [details] [diff] [review]
simplify xpcshell interfaces

Sorry, this is a big patch, but there's really not a lot going on -- just re-structuring in preparation for the next 2 patches.
Attachment #8897652 - Flags: review?(jmaher)

Comment 2

10 months ago
Created attachment 8897653 [details] [diff] [review]
split runTests()

runTests() has grown to almost 400 lines of code:

This splits it up a little, with the main split being, runTests constructs a list of tests; runTestList runs the list.

This makes the --verify changes easier...
Attachment #8897653 - Flags: review?(jmaher)

Comment 3

10 months ago
Created attachment 8897658 [details] [diff] [review]
support --verify

I've tried to make the --verify support here very similar to that added for mochitests: Apply a list of verification steps to each test.

Demonstrating no harm done:

I've also completed local testing including:

Desktop (Linux):
mach xpcshell-test <test>
mach xpcshell-test <directory>
mach xpcshell-test <directory> --sequential
mach xpcshell-test <test> --verify
mach xpcshell-test <test1> <test2> --verify

mach xpcshell-test <test>
mach xpcshell-test <directory>

I will also demonstrate --verify mode on treeherder in bug 1380122.
Attachment #8897658 - Flags: review?(jmaher)
Comment on attachment 8897652 [details] [diff] [review]
simplify xpcshell interfaces

Review of attachment 8897652 [details] [diff] [review]:

a lot of changes, but effectively at parity
Attachment #8897652 - Flags: review?(jmaher) → review+
Attachment #8897653 - Flags: review?(jmaher) → review+
Comment on attachment 8897658 [details] [diff] [review]
support --verify

Review of attachment 8897658 [details] [diff] [review]:

nice- I do wonder if we can end up with a common verify routine for mochitest/xpcshell- maybe after reftest is supported we could look for a smarter way to refactor this.
Attachment #8897658 - Flags: review?(jmaher) → review+

Comment 6

10 months ago
Pushed by
1. Simplify argument passing in xpcshell test harness; r=jmaher
2. Break xpcshell test harness runTests() function into smaller pieces; r=jmaher
3. Add --verify and --verify-max-time options to xpcshell test harness; r=jmaher


10 months ago
See Also: → bug 1390889

Comment 7

10 months ago
Last Resolved: 10 months ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.