841780 - Run reftests in one chunk on mozilla-b2g18* branches

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - run reftests in 1 chunk on b2g18*

I noticed we are still only running reftest-sanity tests on the b2g18* branches, yet we still have them in 10 chunks. There aren't any plans to land the extended set of reftests there yet, so lets cut them down to 1 chunk to save on the slave load. Easy short term win.

I *think* this patch will work, though please pay special attention as I'm not 100% sure and don't know how to test it.

I figured that we always have 2 use cases:
1) We want to run reftest-sanity in 1 chunk
2) We want to run the full set in multiple chunks

This patch makes a separate REFTEST_SANITY_ONLY suite_config. A nice side effect is it lets us get rid of all those mozilla-b2g18* extra_args for the manifest special case.

Comment 1

[Aki Sasaki (not active)]

Comment on attachment 714419 [details] [diff] [review]
Patch 1.0 - run reftests in 1 chunk on b2g18*

I thought about renaming it to reftest-sanity, but I think that would be visible to tbpl and the like.

Passes test-masters, and this looks right.  Thanks for the patch!

Comment 2

[Andrew Halberstadt [:ahal]]

http://hg.mozilla.org/build/buildbot-configs/rev/c374c487d52a

Comment 3

[Rail Aliiev [:rail]]

in production

Comment 4

[Andrew Halberstadt [:ahal]]

Crap, this didn't exactly have the desired effect. All the reftest-sanity tests are running under the "R" symbol, but we still have R2 - R10 running the same ~10 tests as they were before.

Comment 5

[Andrew Halberstadt [:ahal]]

I wonder if maybe we should just rename it to reftest-sanity. This has the added benefit of letting people know that "hey! we aren't actually running any meaningful tests here!"

Though, I'm not sure how we are still running R2-R10. I guess buildbot is trying to be clever or something.

Comment 6

[Andrew Halberstadt [:ahal]]

Hm so looking at https://tbpl.mozilla.org/?tree=Mozilla-B2g18&rev=c2dfc2565ae0 it now seems to have resolved itself? I'm confused.. but oh well?

Comment 7

[Phil Ringnalda (:philor)]

If you mean https://tbpl.mozilla.org/?tree=Mozilla-B2g18_v1_0_1&rev=2eb017b9ba1b and https://tbpl.mozilla.org/?tree=Mozilla-B2g18&rev=65998655101c they're showing exactly what they should: the on-push builds finished before the reconfig, so they triggered and ran R1-R10 (except one was still pending after the reconfig when it no longer existed, so I killed that zombie), then the 11pm nightly triggered just R. The tip pushes on each tree have just an R from the on-push build, and an R from the 7am nightly.

Comment 8

[Andrew Halberstadt [:ahal]]

Ah thanks, that makes sense. I didn't notice the nightly build. All is well with the world.