Add a test verification mode to the xpcshell test harness

RESOLVED FIXED in Firefox 57

Status

Testing
General
RESOLVED FIXED
10 months ago
10 months ago

People

(Reporter: gbrown, Assigned: gbrown)

Tracking

(Blocks: 1 bug)

Trunk
mozilla57
Points:
---

Firefox Tracking Flags

(firefox57 fixed)

Details

Attachments

(3 attachments)

(Assignee)

Description

10 months ago
...as in bug 1357520, but for xpcshell instead of mochitest.

This bug aims to add a --verify option for the xpcshell test harness, supporting test verification. Test verification involves running a test, or small group of tests, repeatedly and possibly in different environments, with the intent of finding intermittent test failures quickly and efficiently.
(Assignee)

Comment 1

10 months ago
Created attachment 8897652 [details] [diff] [review]
simplify xpcshell interfaces

Sorry, this is a big patch, but there's really not a lot going on -- just re-structuring in preparation for the next 2 patches.
Attachment #8897652 - Flags: review?(jmaher)
(Assignee)

Comment 2

10 months ago
Created attachment 8897653 [details] [diff] [review]
split runTests()

runTests() has grown to almost 400 lines of code:

https://dxr.mozilla.org/mozilla-central/rev/b95b1638db48fc3d450b95b98da6bcd2f9326d2f/testing/xpcshell/runxpcshelltests.py#1117-1516

This splits it up a little, with the main split being, runTests constructs a list of tests; runTestList runs the list.

This makes the --verify changes easier...
Attachment #8897653 - Flags: review?(jmaher)
(Assignee)

Comment 3

10 months ago
Created attachment 8897658 [details] [diff] [review]
support --verify

I've tried to make the --verify support here very similar to that added for mochitests: Apply a list of verification steps to each test.

Demonstrating no harm done:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=03c0849f4894af74f27504c899967859f7c5b6fb

I've also completed local testing including:

Desktop (Linux):
mach xpcshell-test <test>
mach xpcshell-test <directory>
mach xpcshell-test <directory> --sequential
mach xpcshell-test <test> --verify
mach xpcshell-test <test1> <test2> --verify

Android:
mach xpcshell-test <test>
mach xpcshell-test <directory>

I will also demonstrate --verify mode on treeherder in bug 1380122.
Attachment #8897658 - Flags: review?(jmaher)
Comment on attachment 8897652 [details] [diff] [review]
simplify xpcshell interfaces

Review of attachment 8897652 [details] [diff] [review]:
-----------------------------------------------------------------

a lot of changes, but effectively at parity
Attachment #8897652 - Flags: review?(jmaher) → review+
Attachment #8897653 - Flags: review?(jmaher) → review+
Comment on attachment 8897658 [details] [diff] [review]
support --verify

Review of attachment 8897658 [details] [diff] [review]:
-----------------------------------------------------------------

nice- I do wonder if we can end up with a common verify routine for mochitest/xpcshell- maybe after reftest is supported we could look for a smarter way to refactor this.
Attachment #8897658 - Flags: review?(jmaher) → review+

Comment 6

10 months ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e95891f5a161
1. Simplify argument passing in xpcshell test harness; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/5be1bdef850a
2. Break xpcshell test harness runTests() function into smaller pieces; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/11f8164a5176
3. Add --verify and --verify-max-time options to xpcshell test harness; r=jmaher
(Assignee)

Updated

10 months ago
See Also: → bug 1390889

Comment 7

10 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/e95891f5a161
https://hg.mozilla.org/mozilla-central/rev/5be1bdef850a
https://hg.mozilla.org/mozilla-central/rev/11f8164a5176
Status: NEW → RESOLVED
Last Resolved: 10 months ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.