855282 - Change scheduling model to only trigger one OS version out of 10.6/10.7/10.8 and WinXP/Win7/Win8 except on PGO triggered changesets (was "Stop scheduling 10.6 and 10.7 tests on m-i and try")

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

aki let me know if you would like me help with this.

Comment 1

[Aki Sasaki (not active)]

Hunh, I didn't even see this bug.
Maybe because I was focused on the unowned-buildduty queue.

Comment 2

[Armen [:armenzg]]

I had no time to look into this week but I should be able to for next week.

Comment 3

[Armen [:armenzg]]

I have tried to bring machines back into the pool but a lot of them have the need for memory to be replaced.

We also don't have enough machines to keep up for 10.6 & 10.7.
I'm going to push hard on reducing number of jobs on 10.6.

Comment 4

[Armen [:armenzg]]

I will paste the changes needed to reduce the testing of 10.7.

Comment 5

[Armen [:armenzg]]

Attachment: [lion] disable debug and talos jobs

This should reduce almost 2/3 of the load on 10.7 machines.

Comment 6

[Armen [:armenzg]]

I'm going to take the approach mentioned by jlebar on the thread but I have a couple of questions.

This is what my proposal plus jlebar's would be:
* on m-i:
1) do not trigger 10.6 and 10.7 opt/debug test jobs on every check-in
2) when the time of PGO jobs comes around we test on 10.6, 10.7 & 10.8
** 10.8 is untouched
* on try:
3) do not trigger 10.7 by default

I think I can do #2 but I am fearing a lot of hackery.
I have no clue if we can actually do #3. I have been able to disable win64 *builds* by default but I don't know if we can have 10.7 disabled by default yet be able to trigger it with -u none[10.7]

catlee: what would you suggest me to do?

Comment 7

[Phil Ringnalda (:philor)]

"the time of PGO jobs" is actually to a very large extent "when we trigger PGO through self-serve" since the scheduler fails to know that a push it chooses will wind up in the middle of a huge bustage and multiple overlapping backouts range, so we trigger quite a bit of PGO on csets that we believe will wind up being our good merge points. Having both periodic PGO and self-serve PGO also trigger the missing Mac tests would probably work out well (other than the headaches for you, since there's no promise that periodic PGO will be triggered on a push that even has Mac builds); having periodic PGO trigger them but self-serve not trigger them is unlikely to work for how we manage inbound.

Comment 8

[Armen [:armenzg]]

I have a patch that does the following:
* stops testing 10.7 on Jetpack
* stops running 10.7 talos jobs (as suggested by mbrubeck)
* stops running unit tests for 10.7 on m-i and try

Adding periodic tests for 10.7 on m-i is going to be an *interesting* code refactoring but I will give it a shot.

I would like to drop 10.7's try support as it would be way too much work for the gain that we would get (do not run by default but find a way to trigger it when needed).

Comment 9

[Armen [:armenzg]]

Attachment: [buildbotcustom] behave correctly when a branch does not have one of the supported platforms

Comment 10

[Justin Lebar (not reading bugmail)]

> I would like to drop 10.7's try support as it would be way too much work for the gain that we would 
> get (do not run by default but find a way to trigger it when needed).

So if a developer has a failure on 10.7 only on m-c, they're up a creek without a paddle?

I don't think that's OK unless you can prove that this never happens.  The logic for not dropping 10.7 altogether was that we couldn't prove that 10.6 + 10.8 is sufficient to catch all failures we care about, so it seems that we're expecting some failures on 10.7.

Comment 11

[Armen [:armenzg]]

(In reply to Justin Lebar [:jlebar] from comment #10)
> > I would like to drop 10.7's try support as it would be way too much work for the gain that we would 
> > get (do not run by default but find a way to trigger it when needed).
> 
> So if a developer has a failure on 10.7 only on m-c, they're up a creek
> without a paddle?
> 
> I don't think that's OK unless you can prove that this never happens.  The
> logic for not dropping 10.7 altogether was that we couldn't prove that 10.6
> + 10.8 is sufficient to catch all failures we care about, so it seems that
> we're expecting some failures on 10.7.

If I knew how to do it I would do it but because I know to the most part how buildbot is designed I would have to redesign it just to support a platform that has less users that Windows Vista does (Vista has more Firefox users than *all* Mac versions together).

The proposal was to make 10.7 a tier2 platform (which Asa and akeybl approved) to improve our testing on 10.6 *or* 10.7. We'll have to put a lot of effort into it and might not be sufficient. It is not ideal; we know. We have to be able to come to a middle ground. We can't support everything just because we have to.

On another note, is there anyone from development that could help improve our running times for some of the jobs?
Not sure if this link is reachable by everybody. For instance, I've noticed that "mochitest-browser-chrome" debug jobs are extremely bad:
https://secure.pub.build.mozilla.org/buildapi/reports/builders?detail_level=builder&platform=lion,mountainlion,snowleopard,snowleopard-r4&build_type=debug,opt&job_type=repack,talos,unittest

The average running time for this are:
10.7 mozilla-central debug test mochitest-browser-chrome 1h 41m
10.6 mozilla-central debug test mochitest-browser-chrome 1h 30m
10.8 mozilla-central debug test mochitest-browser-chrome 1h 10m

Comment 12

[Ed Morley [:emorley]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #11)
> For instance, I've noticed
> that "mochitest-browser-chrome" debug jobs are extremely bad

Bug 864085

Comment 13

[Armen [:armenzg]]

On another note, we do zero testing on Windows 7 64-bit yet we are OK to not test it there and be convinced that we ship an OK browser there.

Comment 14

[Justin Lebar (not reading bugmail)]

> On another note, we do zero testing on Windows 7 64-bit yet we are OK to not test it there and be 
> convinced that we ship an OK browser there.

Two wrongs don't make a right...

If we're going to make Mac 10.7 a tier-two platform, then we should go all the way and shut everything off.  Running tests only on m-c and having no capability to run them on both try and m-i is a recipe for disaster.

It was not clear to me that your proposal was to make 10.7 a tier-two platform, so if this is actually what we're going to do, I think we need to start a new newsgroup thread with "Making Mac OS 10.7 a tier-two platform" as the subject.

Please understand that it's frustrating from my perspective to de-support a platform because it's too hard to modify our infrastructure to support it.  We do hard things all the time at Mozilla; that's the nature of shipping a web browser to millions of people on many different platforms.  It's not your fault if you don't have time to do this, but I think it is our fault, as an organization, that we're making decisions about which platforms to support based solely on how difficult it is to modify buildbot.

Comment 15

[Armen [:armenzg]]

I will re-post the thread with a new title sometime this week to make it clearer.

On another note, I don't know how to be clear on what type of tier this would be [1].

Meanwhile, I'm going to try to figure out if some jobs could be merged and how to do so before disabling anything.

[1] https://developer.mozilla.org/en-US/docs/Supported_build_configurations

Comment 16

[Justin Lebar (not reading bugmail)]

Why would it be something other than tier-two?

Comment 17

[Armen [:armenzg]]

(In reply to Justin Lebar [:jlebar] from comment #16)
> Why would it be something other than tier-two?

The document talks about build configurations rather than test configurations (even though it points to tbpl). It also merges OS specific versions (10.6, 10.7, 10.8) into architecture buckets (OS X/x86 and x86-64). That is why I wasn't sure if it would become a tier-3.

Comment 18

[Justin Lebar (not reading bugmail)]

What we decided at the build infra meeting was that we'd like to try:

1. Run 10.6 / 10.7 tests on m-c only.
2. Provide a way to opt in to 10.6 / 10.7 tests on try.

(1) can be this bug, and
(2) can be bug 793989.  It's not ideal, because you might have to wait until the mac build finishes before requesting the tests, but it might be good enough to start with.

If this results in lots of difficult m-c closures due to orange on 10.6 / 10.7, we could revisit this decision.  That call will be made by the sheriffs.

Comment 19

[Justin Lebar (not reading bugmail)]

It would be great if we could get a rough estimate of when we might expect this work to be done, so that this bug doesn't languish interminably.

Comment 20

[Justin Lebar (not reading bugmail)]

> 1. Run 10.6 / 10.7 tests on m-c only.

Perhaps this should be instead

  1. Don't run 10.6 and 10.7 tests on m-i or try.

We still want them on e.g. b2g18 and probably other trees.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Though I will point out that bug 851066 has been on file for quite some time about the number of builds/tests we run on b2g18.

Comment 22

[Armen [:armenzg]]

I'm spending a bunch of stuff on infrastructure planning this week and I can have a look at the bug that Ryan points out first (since it is an immediate gain).

Comment 23

[Rail Aliiev [:rail]]

in production

Comment 24

[Armen [:armenzg]]

I'm looking at my queue and I don't have time for this for a while.

Comment 25

[Steve Fink [:sfink] [:s:]]

(In reply to Justin Lebar [:jlebar] from comment #18)
> 2. Provide a way to opt in to 10.6 / 10.7 tests on try.
> 
> (1) can be this bug, and
> (2) can be bug 793989.  It's not ideal, because you might have to wait until
> the mac build finishes before requesting the tests, but it might be good
> enough to start with.

Unless I'm missing something, #2 just means making them non-default on try. Request those specifically with testname[10.6,10.7]. Or get everything ever defined in the world, including everybody else's funky nondefault tests, with testname[]. I guess I should make a way of selecting default plus selected other tests, but for now this should be good enough to handle your #2 need.

Comment 26

[Chris Cooper [:coop] (he/him)]

This is a scheduling change so it belongs in the Automation component.