826021 - Ensure skipped tests still show up in reftest logs

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Andrew Halberstadt [:ahal]]

Bug 818156 added a change which makes it so skipped tests aren't counted when reftest.js does it's chunking. However it also made it so skipped tests don't show up in the logs (see https://bugzilla.mozilla.org/show_bug.cgi?id=818156#c5).

Failing to log skipped tests can cause confusion and false assumptions about the state of our tests.

Comment 1

[Andrew Halberstadt [:ahal]]

Pushed a patch that I think will fix it to try: https://tbpl.mozilla.org/?tree=Try&rev=2152c60ee3c8

Will need to do some double checking to make sure it is correct.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - don't mess with gURLs

I checked the number of tests being run/skipped in each chunk of reftests both with and without this patch on Linux opt, Android 2.2 Armv6 opt and B2G Arm opt. They all had the exact same number of tests being run, except with this patch applied there were always more tests showing up as skipped (as expected).

The fact that some tests are showing up as skipped without this patch suggests we are missing some conditions when building the tURLs array, but it probably doesn't matter too much as the chunks are pretty similar in size to one another.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 697446 [details] [diff] [review]
Patch 1.0 - don't mess with gURLs

Review of attachment 697446 [details] [diff] [review]:
-----------------------------------------------------------------

after fiddling around with this for a while and wrapping my brain around it, it makes sense.  If we could just put a line or two as a comment helping describe what it is doing so in the future when folks are editing this start/end logic it might go faster.

Comment 4

[Andrew Halberstadt [:ahal]]

Added a comment for explanation:
https://hg.mozilla.org/integration/mozilla-inbound/rev/92c800f3be8b

Comment 5

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/92c800f3be8b