693015 - Stop running tests on debug Android builds until we can get some use out of them

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Phil Ringnalda (:philor)]

Until we get some solution to bug 682129, we cannot make them visible or even temporarily green them up in a reasonable way: we'd have to file unfixable bugs with summaries like "8 totally unknown and unknowable assertions in min-width-1b.html on Android" and then when we got them green for a day or two, the very first time someone landed a patch that asserted in a different test, we would need to hide them again, since we can't very well say "you have to back out, you assert on Android, no I can't tell you what assertion so you can fix it, only that it exists." There's nothing to be gained from a policy of "you can land anything even if it asserts on Android, you just have to blindly file an unfixable bug and blindly annotate your assertions, unless you don't know that's the policy, in which case you'll just abandon your patch after being unable to green it up on try."

I've only been bitching about them, not actually filing to get them turned off, because I assumed that they must be doing mobile devs *some* good while they annoy the piss out of everyone else, but according to bug 669953 the collective opinion of mobile devs was that they crashed on startup, and that their logs shouldn't even be looked at to see whether or not that was what had happened, because a startup crash was certain.

Let's just shut them off until we can either run them usefully, or in a post bug 691177 world, only run them for people who actually asked for them, instead of what we're doing, everything we can to persuade anyone who uses try that all Android logs are orange and should be ignored.

Comment 1

[Brad Lassey [:blassey] (use needinfo?)]

My assumption is that if the debug builds were to crash on start up (which has happened in the recent past) these would change from orange to red. If that is the case, these tests do provide some value. Obviously we'd like them green though.

Comment 2

[Phil Ringnalda (:philor)]

I don't think so - the last push (to inbound, where we don't do debug tests) that I remember that looked like it was a startup crash doesn't seem to have done either crashtests or reftests, but for mochitests and talos, it just looked like dozens of other sorts of orange, since the tegra doesn't do anything useful, and the foopy keeps trying to pull a log and trying to pull a log and trying to pull a log until it gets bored and gives up. bug 686091 agrees with me (well, it's almost entirely me agreeing with me) that the only way you can tell a startup crash is by being able to detect the similarity of the times in "automation.py | Application ran for:" lines. From there, I'd say that https://tbpl.mozilla.org/php/getParsedLog.php?id=6383530&tree=Mozilla-Inbound is what your debug crashtest would look like with a startup crash, orange from warnings with a fairly consistent run time, but if anyone looked at it, they'd probably blindly star it as bug 689839 (which may well actually be the combination of two different startup crashes and another non-startup crash).

Is it possible to run browser-chrome on a debug build? That would be easier than getting someone to write a separate alive test suite that does what the other platforms do on the builder, and would duck the issue of fatal assertions.

Comment 3

[Mike Taylor [:bear]]

Moving this over to ateam until the question of what test is needed where and when is resolved.  right now the discussion above isn't leading me to any action items.

if that is not correct then please do bounce it back

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

ateam is fine, not a priority though.  As a said before, no new tests to come online until we have <=5% orange rate on the tegras.  To me debug tests are a new test.  Once we get there, we can prioritize which tests come online and when.

Comment 5

[Phil Ringnalda (:philor)]

Aaaaaargh.

Comment 6

[Phil Ringnalda (:philor)]

Please type one of two things, either:

"Please turn off these three debug tests which cannot be made green, until they can be made green."

or

"Please leave these three permaorange tests which cannot be made green running."

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

"Please turn off these three debug tests which cannot be made green, until they can be made green."

Comment 8

[Phil Ringnalda (:philor)]

Thanks, that looks like a pretty clear action item.

Comment 9

[Chris Cooper [:coop] (he/him)]

Per triage, re-assigning to bear so he can turn these tests off prior to the next reconfig.

Comment 10

[Mike Taylor [:bear]]

Attachment: disable debug android tests

Comment 11

[Mike Taylor [:bear]]

Comment on attachment 568518 [details] [diff] [review]
disable debug android tests

committed changeset 4921:f5f9880793de

Comment 12

[Mike Taylor [:bear]]

change has landed in production

Comment 13

[Chris Cooper [:coop] (he/him)]

Comment on attachment 568518 [details] [diff] [review]
disable debug android tests

A reconfig that included this happened today.