Closed Bug 826021 Opened 7 years ago Closed 7 years ago

Ensure skipped tests still show up in reftest logs

Categories

(Testing :: Reftest, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla20

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(1 file)

Bug 818156 added a change which makes it so skipped tests aren't counted when reftest.js does it's chunking. However it also made it so skipped tests don't show up in the logs (see https://bugzilla.mozilla.org/show_bug.cgi?id=818156#c5).

Failing to log skipped tests can cause confusion and false assumptions about the state of our tests.
Pushed a patch that I think will fix it to try: https://tbpl.mozilla.org/?tree=Try&rev=2152c60ee3c8

Will need to do some double checking to make sure it is correct.
Status: NEW → ASSIGNED
Blocks: 818156
I checked the number of tests being run/skipped in each chunk of reftests both with and without this patch on Linux opt, Android 2.2 Armv6 opt and B2G Arm opt. They all had the exact same number of tests being run, except with this patch applied there were always more tests showing up as skipped (as expected).

The fact that some tests are showing up as skipped without this patch suggests we are missing some conditions when building the tURLs array, but it probably doesn't matter too much as the chunks are pretty similar in size to one another.
Attachment #697446 - Flags: review?(jmaher)
Comment on attachment 697446 [details] [diff] [review]
Patch 1.0 - don't mess with gURLs

Review of attachment 697446 [details] [diff] [review]:
-----------------------------------------------------------------

after fiddling around with this for a while and wrapping my brain around it, it makes sense.  If we could just put a line or two as a comment helping describe what it is doing so in the future when folks are editing this start/end logic it might go faster.
Attachment #697446 - Flags: review?(jmaher) → review+
https://hg.mozilla.org/mozilla-central/rev/92c800f3be8b
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
You need to log in before you can comment on or make changes to this bug.