Closed Bug 860748 Opened 7 years ago Closed 7 years ago

Cleanup exclude list for b2g mochitests on the emulator

Categories

(Testing :: Mochitest, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla23

People

(Reporter: martijn.martijn, Assigned: martijn.martijn)

References

Details

(Whiteboard: [fixed-in-birch])

Attachments

(1 file, 7 obsolete files)

Attached patch patch (obsolete) — Splinter Review
I've gone through all the tests in the existing exclude list to see if they are passing in current b2g emulator, especially since bug 788866 is fixed.
And this is on top of the patch of bug 850302, btw, so it depends on that fix.
Also, "content/html/content/test/" directory is not in the exclude list anymore (this was done because of timing out of tbpl, while trying to parse the b2g.json file)

I did a local emulator run and this was the end result:
229819 INFO Passed: 344831
229820 INFO Failed: 0
229821 INFO Todo:   7182
229822 INFO SimpleTest FINISHED

I'll also do a try-server run.

b2g.json file size before is 30.2kb and after, it's 22.3kb, so this shouldn't give problems on tbpl regarding timing out because parsing of the b2g.json file takes too long (it should become faster, even).
Don't forget to remove the trailing whitespace you've introduced.
Attached patch patch (obsolete) — Splinter Review
Something that should apply against current trunk.

(In reply to :Ms2ger from comment #1)
> Don't forget to remove the trailing whitespace you've introduced.

Not sure where, but hopefully it is resolved in this patch.
Attachment #736281 - Attachment is obsolete: true
Pushed to try-server:
mwargers-10103:mozilla-central mwargers$ hg qnew cleanup_exclude
mwargers-10103:mozilla-central mwargers$ patch -bp1 < 850302.diff
patching file testing/mochitest/runtestsb2g.py
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none" try
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 4 changesets with 2 changes to 2 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=c98da3ad5033
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
mwargers-10103:mozilla-central mwargers$
With that latest patch, I got this as result, btw:
230185 INFO Passed: 344963
230186 INFO Failed: 0
230187 INFO Todo:   7246
230188 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 2:10:02.540865
Can you make another try run, this one with -p ics_armv7a_gecko?  That's the only platform we're currently running mochitests against.
Ok, done:
mwargers-10103:mozilla-central mwargers$ hg qgoto excludelist
applying excludelist
patch excludelist is empty
now at: excludelist
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p ics_armv7a_gecko -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=330f1a5cb0eb
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
Ok, I'm seeing failures in chunk 2:
content/events/test/test_bug615597.html 
content/html/content/test/forms/test_input_sanitization.html
I can re-add those tests to the exclude list for now.

In chunk 3, some other problems are seen:
13:47:52 test_addRange.html test starts, then:
13:55:47     INFO -  27495 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
13:55:47     INFO -  27496 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test runner timed us out.
13:55:47     INFO -  27497 INFO TEST-END | /tests/dom/imptests/editing/selecttest/test_addRange.html | finished in 455694ms
13:55:47     INFO -  27498 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
I guess this test just takes too long?

A similar thing seems to happen with test_Range-insertNode.html

test_Range-cloneContents.html, test_Range-deleteContents.html, test_Range-extractContents.html seem to have more regular test failures.

I'll add those test files to the exclude list and do a new tryserver test run.
Hmm, according to bug 775227, this might be some oom problem for those time outs.
I'm also re-adding these to the exclude list, since they don't show up in the tryserver log, so weren't tested (because the run didn't complete), so I'm assuming they still fail for now:
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-set.html":"",
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-surroundContents.html":"",
    "dom/imptests/webapps/DOMCore/tests/approved/test_Range-mutations.html":"Test timed out.",
Attached patch Updated patch (obsolete) — Splinter Review
test_input_sanitization.html is also timing out related.
I added some comments for those tests that are timing out to the exclude list.
Attachment #736299 - Attachment is obsolete: true
Attached patch Updated patch (obsolete) — Splinter Review
The previous patch (which was made in the Linux VM) didn't apply for some unknown reason on my Mac tree, where I do the pushes from. So I had to reconstruct the patch from my Mac tree (which was annoying).
So I'm pushing this patch to the try server.
Attachment #737303 - Attachment is obsolete: true
With that try run, I get in chunk 1:
TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug585978.html | application timed out after 330 seconds with no output
Not sure why this failure is happening, there is just a sudden long delay in the test. This test wasn't disabled in current b2g.json file, either. So I guess this is a random failure and I can ignore it for now.
I'm only seeing bug 818190 related to this test.

chunk 4:
15:36:34 INFO - 10670 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/jquery/test_jQuery.html | Test timed out.
This is in the original b2g.json file, so I'll just re-add this entry for now. 

chunk 8:
15:37:27 INFO - 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:46:07 WARNING - TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | application timed out after 330 seconds with no output
15:47:16 INFO - I/GeckoDump( 797): 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:47:17 INFO - I/GeckoDump( 797): 3304 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | Test timed out.
I accidentally removed test_cachedPosition.html and test_closeOnGC.html in the revised patch, so those have to be re-added.

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #11)
[10:34]    <jgriffin>    mwargers: a default B2G build is made using mozilla-b2g18, which is quite different from mozilla-central
[10:35]    <jgriffin>    you can make a B2G build using mozilla-central if you do BRANCH=master ./config.sh emulator
Attached patch Updated patch_v2 (obsolete) — Splinter Review
Pushed this patch to try server: https://tbpl.mozilla.org/?tree=Try&rev=077b9e7f7d51
Attachment #737306 - Attachment is obsolete: true
Attached patch Updated patch_v3 (obsolete) — Splinter Review
Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
This one timed out.
I re-added it to the exclude list again, it passed in the previous tryserver run, but I noticed it takes a very long time to run that test file, so I guess it runs the risk of timing out.

Let me know if you want to do another tryserver run on this patch (which I hope would be the last time).
Attachment #737419 - Attachment is obsolete: true
Attachment #737465 - Flags: review?(jgriffin)
Comment on attachment 737465 [details] [diff] [review]
Updated patch_v3

Review of attachment 737465 [details] [diff] [review]:
-----------------------------------------------------------------

Seems good; I don't think we need another try run.

::: testing/mochitest/b2g.json
@@ -3,5 @@
>      "caps": "",
>      "content": "",
>      "docshell": "",
>      "dom": "",
> -    "dom/apps": "",

Is removing 'dom/apps' intentional?
Attachment #737465 - Flags: review?(jgriffin) → review+
Yes, because there is already "dom" mentioned, so "dom/apps" is superfluous.

I'm doing a test run locally, and if that one passes, I'll add the check-in keyword to this bug.
(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #15)
> Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
> This one timed out.
> I re-added it to the exclude list again, it passed in the previous tryserver
> run, but I noticed it takes a very long time to run that test file, so I
> guess it runs the risk of timing out.

I filed bug 861850 for increasing the per-test timeout for b2g mochitests.
Local test run went fine:
169910 INFO Passed: 215916
169911 INFO Failed: 0
169912 INFO Todo:   2974
169913 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 1:46:17.218677
Assignee: nobody → martijn.martijn
Status: NEW → ASSIGNED
Keywords: checkin-needed
Attachment #737465 - Attachment is obsolete: true
https://hg.mozilla.org/mozilla-central/rev/27fd58da1d36
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
Attached patch patch pushed to try (obsolete) — Splinter Review
This is the patch that I pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=4d2b426d5db4
A lot of the tests that were giving me problems in order to bug 860748, I removed in the b2g.json list. So with this patch, they shouldn't time out anymore.
Sorry, comment in the wrong bug, this was meant for bug 861850
Attachment #738581 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.