Cleanup exclude list for b2g mochitests on the emulator

RESOLVED FIXED in mozilla23

Status

Testing
Mochitest
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: Martijn Wargers (zombie), Assigned: Martijn Wargers (zombie))

Tracking

Trunk
mozilla23
x86_64
Linux
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fixed-in-birch])

Attachments

(1 attachment, 7 obsolete attachments)

(Assignee)

Description

5 years ago
Created attachment 736281 [details] [diff] [review]
patch

I've gone through all the tests in the existing exclude list to see if they are passing in current b2g emulator, especially since bug 788866 is fixed.
And this is on top of the patch of bug 850302, btw, so it depends on that fix.
Also, "content/html/content/test/" directory is not in the exclude list anymore (this was done because of timing out of tbpl, while trying to parse the b2g.json file)

I did a local emulator run and this was the end result:
229819 INFO Passed: 344831
229820 INFO Failed: 0
229821 INFO Todo:   7182
229822 INFO SimpleTest FINISHED

I'll also do a try-server run.

b2g.json file size before is 30.2kb and after, it's 22.3kb, so this shouldn't give problems on tbpl regarding timing out because parsing of the b2g.json file takes too long (it should become faster, even).
Don't forget to remove the trailing whitespace you've introduced.
(Assignee)

Comment 2

5 years ago
Created attachment 736299 [details] [diff] [review]
patch

Something that should apply against current trunk.

(In reply to :Ms2ger from comment #1)
> Don't forget to remove the trailing whitespace you've introduced.

Not sure where, but hopefully it is resolved in this patch.
Attachment #736281 - Attachment is obsolete: true
(Assignee)

Comment 3

5 years ago
Pushed to try-server:
mwargers-10103:mozilla-central mwargers$ hg qnew cleanup_exclude
mwargers-10103:mozilla-central mwargers$ patch -bp1 < 850302.diff
patching file testing/mochitest/runtestsb2g.py
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none" try
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 4 changesets with 2 changes to 2 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=c98da3ad5033
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
mwargers-10103:mozilla-central mwargers$
(Assignee)

Comment 4

5 years ago
With that latest patch, I got this as result, btw:
230185 INFO Passed: 344963
230186 INFO Failed: 0
230187 INFO Todo:   7246
230188 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 2:10:02.540865
Can you make another try run, this one with -p ics_armv7a_gecko?  That's the only platform we're currently running mochitests against.
(Assignee)

Comment 6

5 years ago
Ok, done:
mwargers-10103:mozilla-central mwargers$ hg qgoto excludelist
applying excludelist
patch excludelist is empty
now at: excludelist
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p ics_armv7a_gecko -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=330f1a5cb0eb
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
(Assignee)

Comment 7

5 years ago
Ok, I'm seeing failures in chunk 2:
content/events/test/test_bug615597.html 
content/html/content/test/forms/test_input_sanitization.html
I can re-add those tests to the exclude list for now.

In chunk 3, some other problems are seen:
13:47:52 test_addRange.html test starts, then:
13:55:47     INFO -  27495 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
13:55:47     INFO -  27496 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test runner timed us out.
13:55:47     INFO -  27497 INFO TEST-END | /tests/dom/imptests/editing/selecttest/test_addRange.html | finished in 455694ms
13:55:47     INFO -  27498 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
I guess this test just takes too long?

A similar thing seems to happen with test_Range-insertNode.html

test_Range-cloneContents.html, test_Range-deleteContents.html, test_Range-extractContents.html seem to have more regular test failures.

I'll add those test files to the exclude list and do a new tryserver test run.
(Assignee)

Comment 8

5 years ago
Hmm, according to bug 775227, this might be some oom problem for those time outs.
(Assignee)

Comment 9

5 years ago
I'm also re-adding these to the exclude list, since they don't show up in the tryserver log, so weren't tested (because the run didn't complete), so I'm assuming they still fail for now:
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-set.html":"",
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-surroundContents.html":"",
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-mutations.html":"Test timed out.",
(Assignee)

Comment 10

5 years ago
Created attachment 737303 [details] [diff] [review]
Updated patch

test_input_sanitization.html is also timing out related.
I added some comments for those tests that are timing out to the exclude list.
Attachment #736299 - Attachment is obsolete: true
(Assignee)

Comment 11

5 years ago
Created attachment 737306 [details] [diff] [review]
Updated patch

The previous patch (which was made in the Linux VM) didn't apply for some unknown reason on my Mac tree, where I do the pushes from. So I had to reconstruct the patch from my Mac tree (which was annoying).
So I'm pushing this patch to the try server.
Attachment #737303 - Attachment is obsolete: true
(Assignee)

Comment 13

5 years ago
With that try run, I get in chunk 1:
TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug585978.html | application timed out after 330 seconds with no output
Not sure why this failure is happening, there is just a sudden long delay in the test. This test wasn't disabled in current b2g.json file, either. So I guess this is a random failure and I can ignore it for now.
I'm only seeing bug 818190 related to this test.

chunk 4:
15:36:34 INFO - 10670 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/jquery/test_jQuery.html | Test timed out.
This is in the original b2g.json file, so I'll just re-add this entry for now. 

chunk 8:
15:37:27 INFO - 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:46:07 WARNING - TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | application timed out after 330 seconds with no output
15:47:16 INFO - I/GeckoDump( 797): 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:47:17 INFO - I/GeckoDump( 797): 3304 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | Test timed out.
I accidentally removed test_cachedPosition.html and test_closeOnGC.html in the revised patch, so those have to be re-added.

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #11)
[10:34]    <jgriffin>    mwargers: a default B2G build is made using mozilla-b2g18, which is quite different from mozilla-central
[10:35]    <jgriffin>    you can make a B2G build using mozilla-central if you do BRANCH=master ./config.sh emulator
(Assignee)

Comment 14

5 years ago
Created attachment 737419 [details] [diff] [review]
Updated patch_v2

Pushed this patch to try server: https://tbpl.mozilla.org/?tree=Try&rev=077b9e7f7d51
Attachment #737306 - Attachment is obsolete: true
(Assignee)

Comment 15

5 years ago
Created attachment 737465 [details] [diff] [review]
Updated patch_v3

Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
This one timed out.
I re-added it to the exclude list again, it passed in the previous tryserver run, but I noticed it takes a very long time to run that test file, so I guess it runs the risk of timing out.

Let me know if you want to do another tryserver run on this patch (which I hope would be the last time).
Attachment #737419 - Attachment is obsolete: true
Attachment #737465 - Flags: review?(jgriffin)
Comment on attachment 737465 [details] [diff] [review]
Updated patch_v3

Review of attachment 737465 [details] [diff] [review]:
-----------------------------------------------------------------

Seems good; I don't think we need another try run.

::: testing/mochitest/b2g.json
@@ -3,5 @@
>      "caps": "",
>      "content": "",
>      "docshell": "",
>      "dom": "",
> -    "dom/apps": "",

Is removing 'dom/apps' intentional?
Attachment #737465 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 17

5 years ago
Yes, because there is already "dom" mentioned, so "dom/apps" is superfluous.

I'm doing a test run locally, and if that one passes, I'll add the check-in keyword to this bug.
(Assignee)

Comment 18

5 years ago
(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #15)
> Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
> This one timed out.
> I re-added it to the exclude list again, it passed in the previous tryserver
> run, but I noticed it takes a very long time to run that test file, so I
> guess it runs the risk of timing out.

I filed bug 861850 for increasing the per-test timeout for b2g mochitests.
(Assignee)

Comment 19

5 years ago
Created attachment 737597 [details] [diff] [review]
Updated patch_v3 for check-in

Local test run went fine:
169910 INFO Passed: 215916
169911 INFO Failed: 0
169912 INFO Todo:   2974
169913 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 1:46:17.218677
Assignee: nobody → martijn.martijn
Status: NEW → ASSIGNED
(Assignee)

Updated

5 years ago
Keywords: checkin-needed
Attachment #737465 - Attachment is obsolete: true
https://hg.mozilla.org/projects/birch/rev/27fd58da1d36
Keywords: checkin-needed
Whiteboard: [fixed-in-birch]
https://hg.mozilla.org/mozilla-central/rev/27fd58da1d36
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
(Assignee)

Comment 22

5 years ago
Created attachment 738581 [details] [diff] [review]
patch pushed to try

This is the patch that I pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=4d2b426d5db4
A lot of the tests that were giving me problems in order to bug 860748, I removed in the b2g.json list. So with this patch, they shouldn't time out anymore.
(Assignee)

Comment 23

5 years ago
Sorry, comment in the wrong bug, this was meant for bug 861850
(Assignee)

Updated

5 years ago
Attachment #738581 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.