762179 - Landing by ehsan caused Android reftests-3 to fail without much information

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

Ehsan landed a bunch of reftests that are turning the Android reftests-3 as orange. [1][2]
Unfortunately, there is no test failures on the log.
I believe this is a buildbot artifact.

The log it fails with:
TinderboxPrint: reftest-3<br/><em class="testfail">T-FAIL</em>

This is outputted by buildbot if (passCount > 0) is not valid:
summary = emphasizeFailureText("T-FAIL")

I think the buildbot code is incorrect. I am looking into it.


[1] https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Android Tegra 250 mozilla-inbound opt test reftest-3&noignore=1&rev=df6702c41ddd
[2] https://tbpl.mozilla.org/php/getParsedLog.php?id=12411687&tree=Mozilla-Inbound&full=1
[3] http://mxr.mozilla.org/build/source/buildbotcustom/steps/mobile.py#83

Comment 1

[(no longer active)]

FWIW, we're looking into disabling the tests for now in bug 762180.  Bug 762181 is filed to re-enable them when this is fixed.

Comment 2

[Armen [:armenzg]]

Attachment: analyze log

This shows that the log can be parsed:
Armens-MacBook-Air:~ armenzg$ python analyze.py 
TinderboxPrint: reftest-3<br/>1638/0/55 [0]

The problem comes if the file is an empty one:
Armens-MacBook-Air:~ armenzg$ python analyze.py 
Traceback (most recent call last):
  File "analyze.py", line 86, in <module>
    summary = emphasizeFailureText("T-FAIL")
NameError: name 'emphasizeFailureText' is not defined

I don't like that "if passCount > 0" statement.

Comment 3

[Armen [:armenzg]]

Attachment: be more clear about issue

I believe that the stdio log file does not exist when we call it and that is why we fall on the else statement.

I am trying to run this on staging.

catlee, what do you think so far?

Comment 4

[Armen [:armenzg]]

Adding dustin in case he notices something obvious.

I don't know enough about buildbot.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 630693 [details] [diff] [review]
be more clear about issue

Review of attachment 630693 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think log is the filename on disk, so os.stat won't work here. I also doubt it's empty. We need to figure out why the parser isn't working on this file.

I also believe that the log parser to look at is summarizeLogReftest

Comment 6

[(no longer active)]

FWIW, I ended up disabling the newly added tests for native Fennec on inbound yesterday (bug 762180) and that fixed the problem.  So this was merely an issue with some additional tests being run.

Comment 7

[(no longer active)]

I did a try push which re-enables these tests on native Fennec, and I triggered R3 on it: https://tbpl.mozilla.org/?tree=Try&rev=8cb42a493f6f

Armen said that he's going to use it to test to make sure that whatever fix he comes up with for this actually works.

Comment 8

[(no longer active)]

Any updates on this?

Comment 9

[Chris Cooper [:coop] (he/him)]

Armen: are you still looking at this?

Comment 10

[Armen [:armenzg]]

I tried getting buildduty to work on it but he didn't want to.
I am not looking at this and I should not touch it until I get the win8 builds working again.

Comment 11

[Armen [:armenzg]]

FTR I spoke with Ehsan on IRC letting him know that I would be busy.
Armv6 took a chunk of my week as well.

Comment 12

[(no longer active)]

Any updates on this?

Comment 13

[Armen [:armenzg]]

I don't have time to work on this.

Asking triagefollowup in hopes that someone can pick it up.

Comment 14

[Justin Wood (:Callek)]

Clint, is this something we think someone on your end of the yard would want to look into. I don't know of any specifics in releng side that would warrant causing this.

Primarily I'm not comprehending how running a few reftests can cause the problem described. And we've been getting better infra wise for mobile and reftests in particular that we might be able to see a specific issue.

If you don't think it belongs on your team yet, or specifically that there's no-one to work on it, we can leave it in our queue for now.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

Unfortunately I cannot see any of the logs referenced in this bug as too much time has passed.  I don't really remember this bug when it was filed.  Is there a chance that we can reproduce it now?  

From the sounds of it this was a buildbot specific thing, I would say the next steps are to reproduce this.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

I landed this on try server and the tests were green:
https://tbpl.mozilla.org/?tree=Try&rev=a75ab5f1b3de

but r3 took a long time which is what the comment indicated in the manifest file.  I am not sure we need this bug anymore?

Comment 17

[(no longer active)]

OK, if you believe this is no longer an issue, I guess we can re-enable these tests:

https://hg.mozilla.org/integration/mozilla-inbound/rev/13061a0c2e7a

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/13061a0c2e7a