Closed Bug 1393583 Opened 7 years ago Closed 7 years ago

reduce extra jobs for osx

Categories

(Taskcluster :: General, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla57

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

Attachments

(1 file, 1 obsolete file)

I noticed osx debug has reftest in 2 chunks and that is it.

for osx-stylo we have:
* 8 chunks of reftest
* 16 chunks of reftest-stylo (added in 1318200)


we have a 2-3 minute overhead hit per job typically, so the chunks are adding a lot of extra load.

Also we are running reftest + reftest-stylo on stylo builds, if we have a stylo build, do we need to run reftest-stylo?

I would like to reduce the chunks and get rid of the duplicated reftest jobs
:jryans, do you know if we need both reftest and reftest-stylo on a osx-stylo build?
Flags: needinfo?(jryans)
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #1)
> :jryans, do you know if we need both reftest and reftest-stylo on a
> osx-stylo build?

Well, they are testing distinct things at least:

reftest with Stylo enabled means: compare test file in Stylo mode vs. reference file
reftest-stylo means: compare test file in Stylo mode vs. test file in Gecko mode

You can see here that reftest-stylo also runs on Linux Stylo platforms as well:

https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=reftest-stylo

Maybe we can get by with less chunks?  Since they run the same test files as reftest (just in a different mode), maybe we can use the reftest job's chunk settings?  Looks like that is currently 1 instead of 16.
Flags: needinfo?(jryans)
will we need this duplicate set of stylo when it comes to stylo on by default coming up next week?

another question is do we need this for osx?  Right now osx seems to be struggling and it has struggled more frequently than any other platform.  If this is needed, that is ok- but every little bit helps here.
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #3)
> will we need this duplicate set of stylo when it comes to stylo on by
> default coming up next week?

Good question.  :bholley, do you think we need to keep around Stylo vs. Gecko reftests (for any platform) after flipping the default to Stylo (like until Stylo ships to release), or do you think we can drop these jobs soon once Stylo is the default?

> another question is do we need this for osx?  Right now osx seems to be
> struggling and it has struggled more frequently than any other platform.  If
> this is needed, that is ok- but every little bit helps here.

Well, they have been useful to notice some differences that only happen on specific platforms, so it'd be nice to keep them on macOS (assuming we are keeping them in general).

Tweaking the chunks seems like it's a good start.  We could also make them central and try only (removing autoland) if we really need to.
Flags: needinfo?(bobbyholley)
I think it would be reasonable to turn off styloVsGecko mode for mac when we switch the default config to stylo. I think it's probably still worth keeping styloVsGecko for at least one platform (say, linux64) to provide additional sanity checking.

Manish, you've been tracking our reftest failures. Do we have any failures we care about that currently show up in styloVsGecko mode but not regular mode? If so, what platforms do they affect?
Flags: needinfo?(bobbyholley) → needinfo?(manishearth)
I do not recall any platform-specific styloVsGecko-only failures. Can't find any with a quick grep either.
Flags: needinfo?(manishearth)
Okay, sounds like we should:

* Make reftest-stylo linux64 only
* Copy the chunk settings from reftest
Blocks: 1386625
Comment on attachment 8901180 [details] [diff] [review]
reduce osx load by removing reftest-stylo and using 1/2 chunks for reftest instead of 8/16

Review of attachment 8901180 [details] [diff] [review]:
-----------------------------------------------------------------

Your try run confirms they aren't scheduled for macOS anymore, great!  However, this patch also alters the Linux settings, so I'd like to see a successful Linux reftest and reftest-stylo run with these settings first.

::: taskcluster/ci/test/tests.yml
@@ +1215,5 @@
>      description: "Reftest run for Stylo"
>      suite: reftest/reftest-stylo
>      treeherder-symbol: tc-R(Rs)
>      virtualization: virtual-with-gpu
> +    chunks: 2

Is this enough chunks for Linux runs of `reftest-stylo`?  I'd like to see a try run that includes it first.

Previously, Linux runs of `reftest-stylo` were getting 16 chunks, and even regular `reftest` uses 8 chunks on Linux, so I'm wary of using 2 unless we have a successful run to confirm it.
Attachment #8901180 - Flags: review?(jryans) → review-
this time with a try run and back to 8 chunks for Rs (as we hit timeouts on Rs2 at 1 hour):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c0153224762976952ed7e4d8f1ade40109b2576
Attachment #8901180 - Attachment is obsolete: true
Attachment #8901436 - Flags: review?(jryans)
Comment on attachment 8901436 [details] [diff] [review]
reduce osx load by removing reftest-stylo and using 1/2 chunks for reftest instead of 8/16

Review of attachment 8901436 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, thanks!
Attachment #8901436 - Flags: review?(jryans) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a5f36a073139
reduce chunks of reftest on osx; remove reftest-stylo for osx; r=jryans
https://hg.mozilla.org/mozilla-central/rev/a5f36a073139
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.