864517 - on birch branch, stop running WinXP,Win7,osx10.6,osx10.7 tests (and continue running Win8, osx10.8 tests)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

(CCing some B2G folks here too so they can weigh in)

In order to speed up turnaround time and minimize the infrastructure hit of the birch branch during the B2G work week, many builds and tests were eliminated to leave us with a suite that would give us a level of confidence about what we were merging to m-c without being redundant. For example, debug builds were only created on Linux/Linux64 and checktests were only run on Linux/Linux64 builds.

One thing that was not done was to eliminate testing on OSX 10.6/10.7 and WinXP/Win7, leaving only OSX 10.8 and Win8. However, now that birch is going to be in use for a longer period of time, I think we should revisit that proposal.

Running only OSX 10.8 and Win8 tests would cut the number of test jobs by 2/3 on those two platforms and would mean that the tests being run are on our fastest hardware. In my experience, the odds of a version-specific bustage are remote enough that the risk of such an occurrence is outweighed by the reward of reduced infrastructure load.

Keep in mind that this proposal does not in any way affect what tests are run on any other branch - only birch. Test coverage on m-c and b2g18 will remain the same as ever (which by my understanding is what matters from a QA perspective).

Based on the above, does anyone see a reason this shouldn't be done?

Comment 1

[Armen [:armenzg]]

I don't see one. This should be helpful.

Comment 2

[Justin Lebar (not reading bugmail)]

Perhaps we should merge birch into m-i instead of m-c, so we get a bit of testing here?  I know some people on IRC have suggested that merging into m-c without a full set of tests is a bad idea.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

(In reply to Justin Lebar [:jlebar] from comment #2)
> Perhaps we should merge birch into m-i instead of m-c, so we get a bit of
> testing here?  I know some people on IRC have suggested that merging into
> m-c without a full set of tests is a bad idea.

IMO, if we're merging into inbound, then we lose the advantages of having birch in the first place.

That said, I think we've waited long enough for reasons not to do this.

Comment 4

[Justin Lebar (not reading bugmail)]

Just to be clear, those browser-element leaks I was fighting against recently were OSX 10.6 / 10.7 debug only.

> IMO, if we're merging into inbound, then we lose the advantages of having birch in the first place.

It's not clear that this is true, since you can choose when to merge birch into inbound.

But I'm OK burning central if you are.  :)

Comment 5

[Ryan VanderMeulen [:RyanVM]]

(In reply to Justin Lebar [:jlebar] from comment #4)
> It's not clear that this is true, since you can choose when to merge birch
> into inbound.

The point of birch is to get B2G patches on m-c (and uplifted to b2g18) faster. If we're still going to have to merge to inbound and wait for pgo-green, why not just directly land there? All it does is add indirection for no gain.

> But I'm OK burning central if you are.  :)

Haven't yet, but we're obviously taking a calculated risk with our current birch setup. That said, the number of patches landing in a typical birch merge is small enough that bisecting a merge bustage probably wouldn't be too hard if it came to that IMO.

Comment 6

[Justin Lebar (not reading bugmail)]

> If we're still going to have to merge to inbound and wait for pgo-green, why not just directly 
> land there?

Because inbound is frequently busted, and it's nice to be insulated from that?  I'd thought that was the reason we kept birch around after the last work week.

I agree that if the purpose is to land to central more quickly, then merging to inbound doesn't make much sense.  Anyway you watch the tree, so I have no objections to your plan.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per infra load meeting just now:

1) seems like no objections to keeping one variant of OSX+Windows, and trim the others to reduce load. Tweaking summary to match.
2) fabrice, jonas:  RyanVM has posted to newsgroups on this, and we think its ok, but wanted to make sure you saw and had no objections from b2g side? If you have questions not already answered here in bug, please contact RyanVM or myself.

Comment 8

[[:fabrice] Fabrice Desré]

That's fine for me!

Comment 9

[Chris AtLee [:catlee]]

Attachment: only have win8, osx10.8 enabled for birch

Comment 10

[Armen [:armenzg]]

Live in production.