1139006 - decide what to do with MacOSX desktop tests on b2g-inbound

Status: RESOLVED DUPLICATE of bug 1139029

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

We are currently in the process of moving 10.8 tests to run on 10.10.

This impacts the 10.8 tests that run on b2g-inbound.  Before I move them, are they still needed?

Comment 1

[Kim Moir [:kmoir] ET]

:gwagner - RyanVM mentioned that you might have insight into whether MacOSX desktop tests on b2g-inbound are still useful?

Comment 2

[Gregor Wagner [:gwagner]]

(In reply to Kim Moir [:kmoir] from comment #1)
> :gwagner - RyanVM mentioned that you might have insight into whether MacOSX
> desktop tests on b2g-inbound are still useful?

We discussed this on IRC. It seems that we are only running Gip there? The QA team maintains them and they need to figure out what to do.

Comment 3

[Johan Lorenzo [:jlorenzo]]

Adding Geo in the loop.

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

We inherited that setup, so I'm going to have to check in with FxOS A-Team to see if there's a historical reason. I'm NI'ing Dave Hunt for his take.

My off-the-cuff take is that for the purposes of qualifying Gaia and whatever extent of Gecko is valid to test in B2G Desktop, it's perfectly OK to only run them on Linux64. 

However, to back that up I'd like to look at some test failure history to understand if we ever get Gaia failures on OS X that don't reproduce on Linux64. I'm not sure of the easiest way to do that yet, but it's probably the followup task here.

The only reason I can think of to test this on two different platforms is to qualify B2G Desktop itself on OS X vs. a reference Linux64 instance (or vice versa), but if the tests are a hassle to run that's not a good enough reason for me to think we should keep them.

I'm keeping my NI open for followup and clearing Johan's.

Comment 5

[Martijn Wargers (dead)]

(In reply to Geo Mealer [:geo] from comment #4)
> However, to back that up I'd like to look at some test failure history to
> understand if we ever get Gaia failures on OS X that don't reproduce on
> Linux64. I'm not sure of the easiest way to do that yet, but it's probably
> the followup task here.

We currently have at least one case where that seems to be happening, see bug 1126080, comment 56.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We've hit a couple of OS X specific issues in the past but as far as I can recall these were all determined to be unresolvable environmental issues. I suspect the main reason we're running on Mac OS X is for additional coverage - if there's little or no cost to continuing this then I don't see a reason to terminate them. Potential impact of removing these is that the nightly desktop simulator builds offered at nightly.mozilla.org could be broken without us knowing about it.

Whatever we decide, we may want to consider applying the same decisions to Mulet in bug 1110839.

Comment 7

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Well, I think bug 1139029 has made much of this discussion moot, as I believe the OS X Gip tests have now been turned off on all branches.

At this point, looking at Treeherder, no tests run on OS X B2G Desktop. Dave's comment 6 reinforces my impression that we were running on OS X to qualify the B2G Desktop release itself, but the Gip tests wouldn't have been nearly enough to do that in isolation. 

I am concerned at the lack of validation there, but doing so effectively would really require turning on our full suite of tests again, not just Gip.