1127916 - Un-hide Android x86 S4 when it meets visibility requirements

Status: RESOLVED FIXED

(Tree Management Graveyard :: Visibility Requests, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

By my less-than-informal poll of mozilla-inbound today, Android x86 S4 is sitting on a failure rate of ~13% at the moment, well in excess of 5%. I've hidden it across all trees until it is able to meet that requirement.

AFAICT, fixing the two deps on this bug would largely eradicate the intermittent failures we're seeing in the suite.

Comment 1

[Geoff Brown [:gbrown]]

We decided to "cheat" by not running robocop on x86 at all, and that is now deployed, in bug 1132209.

That leaves Android x86 S4 running only xpcshell, and that seems quite reliable -- I think we can unhide S4 now.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Per https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy:

1.1 Has an active owner
1.2 Usable job logs
1.3 Low intermittent failure rate
1.4 Must avoid patterns known to cause non deterministic failures
1.5 Has sufficient documentation
2.1 Breakage is expected to be followed by tree closure or backout
2.2 Runs on mozilla-central and all trees that merge into it
2.3 Scheduled on every push
2.4 Easily run on try server

And for the record, "Maybe gbrown can take a look if/when he has some spare cycles" isn't active ownership IMO. Android x86 tests have chronically suffered from lack of ownership, so I'm not feeling very inclined to reopen this door unless there's a real commitment from the mobile team to treat them like a Tier 1 platform.

Comment 3

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #2)
I could say we're going to "actively own" these tests, but the reality is that we just don't really have resources to do that properly. We will, however, do our best to make sure they are reliable.

If we don't at least have these tests running, it means there is nothing (in automation) stopping a breaking change from making it into a product that we ship to users. The xpcshell tests at least give us *something*, so please consider unhiding these.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Per the IRC discussion we had, I've unhidden the suite. We'll see how it goes.