962676 - Assertion count mismatches during various crashtests on Android 4.0 Debug

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

Android 4.0 *Debug* tests currently only run on the Cedar tree. We are trying to get these tests to pass so that they can be run on trunk trees.

On Cedar, on Android 4.0 Debug, crashtests tests consistently fail with various assertion count mismatches:

https://tbpl.mozilla.org/php/getParsedLog.php?id=33337466&tree=Cedar&full=1

REFTEST TEST-UNEXPECTED-PASS | data:text/html, | assertion count 0 is less than expected 8 to 10 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/docshell/base/crashtests/436900-1.html | assertion count 0 is less than expected 2 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/dom/base/crashtests/90613-1.html | assertion count 0 is less than expected 1 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/dom/xbl/crashtests/493123-1.xhtml | assertion count 0 is less than expected 2 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/editor/composer/src/crashtests/351236-1.html | assertion count 0 is less than expected 2 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/base/crashtests/265986-1.html | assertion count 3 is less than expected 6 to 12 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/base/crashtests/266360-1.html | assertion count 1 is less than expected 2 to 7 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/base/crashtests/477731-1.html | assertion count 0 is less than expected 6 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/base/crashtests/480686-1.html | assertion count 0 is less than expected 6 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/base/crashtests/898913.html | assertion count 2 is less than expected 4 to 6 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/forms/crashtests/378413-1.xhtml | assertion count 4 is less than expected 5 to 10 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/forms/crashtests/383887-1.html | assertion count 0 is less than expected 1 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/generic/crashtests/394818-2.html | assertion count 0 is less than expected 1 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/generic/crashtests/414061-1.html | assertion count 0 is less than expected 6 assertions
REFTEST TEST-UNEXPECTED-FAIL | http://10.12.133.21:30582/tests/layout/generic/crashtests/499885-1.xhtml | assertion count 4 is more than expected 0 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/generic/crashtests/673770.html | assertion count 0 is less than expected 8 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/generic/crashtests/798020-1.html | assertion count 0 is less than expected 8 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/generic/crashtests/847209.html | assertion count 2 is less than expected 4 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/layout/xul/tree/crashtests/366583-1.xul | assertion count 0 is less than expected 3 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.21:30582/tests/parser/htmlparser/tests/crashtests/328751-1.html | assertion count 0 is less than expected 1 assertions

(There is also a crash in this log -- see bug 940087.)

Unfortunately, not enough logcat is reported to see the failed assertions.

One approach to resolving this would be to reproduce the failures on a local device with a debug build while observing logcat.

There are on-going efforts to report complete logcats on tbpl, but I am not sure how long it will be before a working solution is available.

See https://tbpl.mozilla.org/?tree=Cedar&showall=1&jobname=android.*debug for more logs.

Comment 1

[Geoff Brown [:gbrown]]

Complete logcats are available now.

https://tbpl.mozilla.org/php/getParsedLog.php?id=34563411&tree=Cedar&full=1

http://mozilla-releng-blobs.s3.amazonaws.com/blobs/cedar/sha512/0e7a497b2fbcbd250cd8b82f2f3a6cddba57f3ca6b4f2ad2df55a48b3baf362ff65325d10c31cea37cf13a82d4c0b51c86e785af5f8ef5f69ad63a959fe779c3

Comment 2

[Geoff Brown [:gbrown]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=35890174&tree=Cedar&full=1

Comment 3

[Geoff Brown [:gbrown]]

:blassey -- Please find an owner. Note that most of these are unexpected passes -- this might be as easy as updating the tests to expect 0 assertions.

Comment 4

[Geoff Brown [:gbrown]]

Each failure is now reported in its own dependent bug -- they all need owners.

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

ms2ger, it looks like you added all (or at least most) of these assertions in bug 684807. Were we running debug tests back then?? If not, how were you hitting these assertions in the first place?

Comment 6

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #5)
> ms2ger, it looks like you added all (or at least most) of these assertions
> in bug 684807. Were we running debug tests back then?? If not, how were you
> hitting these assertions in the first place?

s/assertions/annotations/

Comment 7

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

I think we ran them, but we shut them off because they didn't have stacks, and then presumably a bunch were fixed since.

Comment 8

[Geoff Brown [:gbrown]]

Attachment: remove asserts-if(Android) from crashtest manifests

This patch deals with the lowest hanging fruit here: Removes the asserts-if(Android) manifest notations that are causing failures. 

This reduces the set of failures to:

REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.23:30599/tests/layout/base/crashtests/266360-1.html | assertion count 1 is less than expected 2 to 7 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.23:30599/tests/layout/base/crashtests/898913.html | assertion count 2 is less than expected 4 to 6 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.23:30599/tests/layout/forms/crashtests/378413-1.xhtml | assertion count 4 is less than expected 5 to 10 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.23:30599/tests/layout/generic/crashtests/798020-1.html | assertion count 0 is less than expected 8 assertions
REFTEST TEST-UNEXPECTED-PASS | http://10.12.133.23:30599/tests/layout/generic/crashtests/847209.html | assertion count 2 is less than expected 4 assertions

These remaining cases are caused by asserts() or more generic asserts-if() (not specific to Android).

Comment 9

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/817929e9eb00

Comment 10

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/817929e9eb00

Comment 11

[Geoff Brown [:gbrown]]

All dependent bugs fixed.