860748 - Cleanup exclude list for b2g mochitests on the emulator

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Martijn Wargers (dead)]

Attachment: patch

I've gone through all the tests in the existing exclude list to see if they are passing in current b2g emulator, especially since bug 788866 is fixed.
And this is on top of the patch of bug 850302, btw, so it depends on that fix.
Also, "content/html/content/test/" directory is not in the exclude list anymore (this was done because of timing out of tbpl, while trying to parse the b2g.json file)

I did a local emulator run and this was the end result:
229819 INFO Passed: 344831
229820 INFO Failed: 0
229821 INFO Todo:   7182
229822 INFO SimpleTest FINISHED

I'll also do a try-server run.

b2g.json file size before is 30.2kb and after, it's 22.3kb, so this shouldn't give problems on tbpl regarding timing out because parsing of the b2g.json file takes too long (it should become faster, even).

Comment 1

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Don't forget to remove the trailing whitespace you've introduced.

Comment 2

[Martijn Wargers (dead)]

Attachment: patch

Something that should apply against current trunk.

(In reply to :Ms2ger from comment #1)
> Don't forget to remove the trailing whitespace you've introduced.

Not sure where, but hopefully it is resolved in this patch.

Comment 3

[Martijn Wargers (dead)]

Pushed to try-server:
mwargers-10103:mozilla-central mwargers$ hg qnew cleanup_exclude
mwargers-10103:mozilla-central mwargers$ patch -bp1 < 850302.diff
patching file testing/mochitest/runtestsb2g.py
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none" try
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 4 changesets with 2 changes to 2 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=c98da3ad5033
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
mwargers-10103:mozilla-central mwargers$

Comment 4

[Martijn Wargers (dead)]

With that latest patch, I got this as result, btw:
230185 INFO Passed: 344963
230186 INFO Failed: 0
230187 INFO Todo:   7246
230188 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 2:10:02.540865

Comment 5

[Jonathan Griffin (:jgriffin)]

Can you make another try run, this one with -p ics_armv7a_gecko?  That's the only platform we're currently running mochitests against.

Comment 6

[Martijn Wargers (dead)]

Ok, done:
mwargers-10103:mozilla-central mwargers$ hg qgoto excludelist
applying excludelist
patch excludelist is empty
now at: excludelist
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p ics_armv7a_gecko -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=330f1a5cb0eb
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

Comment 7

[Martijn Wargers (dead)]

Ok, I'm seeing failures in chunk 2:
content/events/test/test_bug615597.html 
content/html/content/test/forms/test_input_sanitization.html
I can re-add those tests to the exclude list for now.

In chunk 3, some other problems are seen:
13:47:52 test_addRange.html test starts, then:
13:55:47     INFO -  27495 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
13:55:47     INFO -  27496 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test runner timed us out.
13:55:47     INFO -  27497 INFO TEST-END | /tests/dom/imptests/editing/selecttest/test_addRange.html | finished in 455694ms
13:55:47     INFO -  27498 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
I guess this test just takes too long?

A similar thing seems to happen with test_Range-insertNode.html

test_Range-cloneContents.html, test_Range-deleteContents.html, test_Range-extractContents.html seem to have more regular test failures.

I'll add those test files to the exclude list and do a new tryserver test run.

Comment 8

[Martijn Wargers (dead)]

Hmm, according to bug 775227, this might be some oom problem for those time outs.

Comment 9

[Martijn Wargers (dead)]

I'm also re-adding these to the exclude list, since they don't show up in the tryserver log, so weren't tested (because the run didn't complete), so I'm assuming they still fail for now:
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-set.html":"",
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-surroundContents.html":"",
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-mutations.html":"Test timed out.",

Comment 10

[Martijn Wargers (dead)]

Attachment: Updated patch

test_input_sanitization.html is also timing out related.
I added some comments for those tests that are timing out to the exclude list.

Comment 11

[Martijn Wargers (dead)]

Attachment: Updated patch

The previous patch (which was made in the Linux VM) didn't apply for some unknown reason on my Mac tree, where I do the pushes from. So I had to reconstruct the patch from my Mac tree (which was annoying).
So I'm pushing this patch to the try server.

Comment 12

[Martijn Wargers (dead)]

Pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=9025aba386cb

Comment 13

[Martijn Wargers (dead)]

With that try run, I get in chunk 1:
TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug585978.html | application timed out after 330 seconds with no output
Not sure why this failure is happening, there is just a sudden long delay in the test. This test wasn't disabled in current b2g.json file, either. So I guess this is a random failure and I can ignore it for now.
I'm only seeing bug 818190 related to this test.

chunk 4:
15:36:34 INFO - 10670 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/jquery/test_jQuery.html | Test timed out.
This is in the original b2g.json file, so I'll just re-add this entry for now. 

chunk 8:
15:37:27 INFO - 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:46:07 WARNING - TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | application timed out after 330 seconds with no output
15:47:16 INFO - I/GeckoDump( 797): 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:47:17 INFO - I/GeckoDump( 797): 3304 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | Test timed out.
I accidentally removed test_cachedPosition.html and test_closeOnGC.html in the revised patch, so those have to be re-added.

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #11)
[10:34]    <jgriffin>    mwargers: a default B2G build is made using mozilla-b2g18, which is quite different from mozilla-central
[10:35]    <jgriffin>    you can make a B2G build using mozilla-central if you do BRANCH=master ./config.sh emulator

Comment 14

[Martijn Wargers (dead)]

Attachment: Updated patch_v2

Pushed this patch to try server: https://tbpl.mozilla.org/?tree=Try&rev=077b9e7f7d51

Comment 15

[Martijn Wargers (dead)]

Attachment: Updated patch_v3

Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
This one timed out.
I re-added it to the exclude list again, it passed in the previous tryserver run, but I noticed it takes a very long time to run that test file, so I guess it runs the risk of timing out.

Let me know if you want to do another tryserver run on this patch (which I hope would be the last time).

Comment 16

[Jonathan Griffin (:jgriffin)]

Comment on attachment 737465 [details] [diff] [review]
Updated patch_v3

Review of attachment 737465 [details] [diff] [review]:
-----------------------------------------------------------------

Seems good; I don't think we need another try run.

::: testing/mochitest/b2g.json
@@ -3,5 @@
>      "caps": "",
>      "content": "",
>      "docshell": "",
>      "dom": "",
> -    "dom/apps": "",

Is removing 'dom/apps' intentional?

Comment 17

[Martijn Wargers (dead)]

Yes, because there is already "dom" mentioned, so "dom/apps" is superfluous.

I'm doing a test run locally, and if that one passes, I'll add the check-in keyword to this bug.

Comment 18

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #15)
> Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
> This one timed out.
> I re-added it to the exclude list again, it passed in the previous tryserver
> run, but I noticed it takes a very long time to run that test file, so I
> guess it runs the risk of timing out.

I filed bug 861850 for increasing the per-test timeout for b2g mochitests.

Comment 19

[Martijn Wargers (dead)]

Attachment: Updated patch_v3 for check-in

Local test run went fine:
169910 INFO Passed: 215916
169911 INFO Failed: 0
169912 INFO Todo:   2974
169913 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 1:46:17.218677

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/projects/birch/rev/27fd58da1d36

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/27fd58da1d36

Comment 22

[Martijn Wargers (dead)]

Attachment: patch pushed to try

This is the patch that I pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=4d2b426d5db4
A lot of the tests that were giving me problems in order to bug 860748, I removed in the b2g.json list. So with this patch, they shouldn't time out anymore.

Comment 23

[Martijn Wargers (dead)]

Sorry, comment in the wrong bug, this was meant for bug 861850