1204032 - Android 4.0 API11+ debug reftests not running on try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

In https://treeherder.mozilla.org/#/jobs?repo=try&revision=be5f69a52f8e&exclusion_profile=false, no Android 4.0 debug tests were run, even though Android 4.0 API11+ debug reftests still run on mozilla-central.

Comment 1

[James Graham [:jgraham]]

It turns out that I've been backed out twice for not spotting that my perfectly green* try run was insufficient to tell that I had broken these tests and therefore inbound.

* Well, as green as one can expect.

Comment 2

[Kim Moir [:kmoir] ET]

The builders are there on the masters.  And I can see that jobs ran yesterday on try for Android 4.0 debug reftests. For example 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e8cdb09267b6

Perhaps there is an issue with the the try syntax.  In the previous push, -u all[Ubuntu,10.6,10.10,Windows XP,Windows 7,Windows 8,Android]  was used

while your push used -u all

Comment 3

[Kim Moir [:kmoir] ET]

Also, maybe this is related to bug 1184117 too, where android 4.0 is disabled by default on try

Comment 4

[Geoff Brown [:gbrown]]

I think that's right: bug 1184117 disabled 4.0 by default on try. That was probably not the best thing to do. Now anyone who pushes to try with something like -p all -u all may be surprised by 4.0 Debug reftest failures on landing. 

If the try_by_default: False is removed, will anything other than 4.0 Debug reftests/jsreftests/crashtests run?

Comment 5

[Kim Moir [:kmoir] ET]

Android 4.0 opt builders are still enabled.  If we enable 4.0 by default on try again will schedule them unless you want me to disable these builders. Let me know what you would prefer.

Comment 6

[Geoff Brown [:gbrown]]

It looks to me like all of the 4.0 -> 4.3 test migration -- other than debug reftests/jsreftests/crashtests -- has ridden the trains all the way to mozilla-release. I don't think there are any esr concerns (are there? I'm not very well-informed about esr stuff). So then we should never need to run 4.0 opt tests or 4.0 debug cpp/xpcshell/mochi tests again and should be able to disable those builders (we still need 4.0 opt and 4.0 debug *builds* and 4.0 debug reftests/jsreftests/crashtests of course) and turn on try_by_default so that 4.0 debug reftests start running for try -p all -u all.

When we start running 4.3 debug reftests on mozilla-central, we will again want try_by_default: False, so that default try pushes reflect mozilla-central but specific try pushes can still trigger 4.0 debug reftests on try, for aurora+ testing.

Does that sound right to you? I have a nagging feeling I am forgetting something...

If 4.3 debug reftests are coming very soon, I think the argument could be made that this is too much trouble for a temporary problem. But we should keep in mind that both snorp and jgraham have been affected by this just in the last week.

Comment 7

[Kim Moir [:kmoir] ET]

Okay, I'll disable those opt builders and then remove try_by_default for this platform

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: bug1204032.patch

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: bug1204032builder.diff

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1204032.patch

Comment 11

[Justin Wood (:Callek)]

Comment on attachment 8661942 [details] [diff] [review]
bug1204032.patch

Review of attachment 8661942 [details] [diff] [review]:
-----------------------------------------------------------------

If the intent is to remove these builders from try, and simultaneously run the remaining 4.0 builders by default on try then this does as advertised

Comment 12

[Kim Moir [:kmoir] ET]

Comment on attachment 8661942 [details] [diff] [review]
bug1204032.patch

thanks Callek!

Comment 13

[Kim Moir [:kmoir] ET]

in production