Closed Bug 759221 Opened 12 years ago Closed 12 years ago

Enable content/canvas and content/media tests on Android

Categories

(Firefox for Android Graveyard :: General, defect)

ARM
Android
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Firefox 16

People

(Reporter: bjacob, Assigned: bjacob)

References

Details

Attachments

(7 files, 4 obsolete files)

893 bytes, patch
jmaher
: review+
Details | Diff | Splinter Review
1.04 KB, patch
jmaher
: review+
Details | Diff | Splinter Review
921 bytes, patch
Details | Diff | Splinter Review
4.77 KB, patch
jgilbert
: review+
Details | Diff | Splinter Review
1.06 KB, patch
jgilbert
: review+
Details | Diff | Splinter Review
1.51 KB, patch
jmaher
: review+
Details | Diff | Splinter Review
5.36 KB, patch
bjacob
: review+
Details | Diff | Splinter Review
Canvas tests (content/canvas/test) are currently not run on Android. We should enable them.

I saw this file and vaguely remember being pointed to it:

  testing/mochitest/android.json

Is it the right file to edit?
Attached patch enable canvas tests on android (obsolete) — Splinter Review
Not tested yet.
The mochitest log shows that the android test slaves fail to create WebGL contexts. New try with debug output to try understanding why:

https://tbpl.mozilla.org/?tree=Try&rev=c978ff534a77
Logcat isn't recorded in mochitest logs, which are sourced from a separate file that mochitest writes directly to, so there's nothing I can do here with the test slaves. Will try to get a Tegra device to reproduce.
Reproduced on Tegra board. The problem is that the first MakeCurrent on the GL context newly created for the WebGL context being created, generates a EGL_BAD_ACCESS error.
This is probably the same issue as Bug 758323 Comment 8.
Armen, is there a bug on file to track getting the panda boards into production?
(In reply to Brad Lassey [:blassey] from comment #7)
> Armen, is there a bug on file to track getting the panda boards into
> production?

bug 725844 or bug 725845.
There is also the tracking bug 725544.

Why? Does bjacob need a panda board to tinker with?
(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #8)
> (In reply to Brad Lassey [:blassey] from comment #7)
> > Armen, is there a bug on file to track getting the panda boards into
> > production?
> 
> bug 725844 or bug 725845.
> There is also the tracking bug 725544.
> 
> Why? Does bjacob need a panda board to tinker with?

Nope, just need ICS running tests in production to enable canvas tests on android
Some notes:

1. Some ICS Tegra 2 devices have this problem. For example Motorola Xoom ICS, see bug 759225.

2. Jeff found a link that confirms the theory that these devices can't have two OpenGL contexts bound on two different threads: see bug 758323 comment 12.

3. The way to tell is go to a WebGL page and check if logcat says:

E/libEGL  (25716): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)

Nevermind the 25716 and 674 here, what matters is 3002 / EGL_BAD_ACCESS.
I'm currently looking at enabling at least the 2D canvas tests. https://tbpl.mozilla.org/php/getParsedLog.php?id=12157024&tree=Try&full=1 gives these errors:

1155 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/crossorigin/test_video_crossorigin.html | We expected 'tests/content/media/test/320x240.ogv' to exist, but it doesn't!
1156 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/crossorigin/test_video_crossorigin.html | We expected 'tests/content/media/test/r11025_s16_c1.wav' to exist, but it doesn't!
1157 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/crossorigin/test_video_crossorigin.html | We expected 'tests/content/media/test/bug461281.ogg' to exist, but it doesn't!
1158 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/crossorigin/test_video_crossorigin.html | We expected 'tests/content/media/test/seek.webm' to exist, but it doesn't!

This is the crossorigin video test complaining about its dependency on content/media tests which aren't built. We need to either also enable the content/media tests (my preferred solution) or disable the crossorigin video canvas test.


3379 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 0,0 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3381 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 99,0 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3385 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 0,49 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3387 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 99,49 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0

Seems like a typical expected-32bpp-got-16bpp failure i.e. a test that's assuming desktop i.e. bug in the test.
Try run enabling both canvas and media tests:
https://tbpl.mozilla.org/?tree=Try&rev=fb1dd9dc42e0
Attachment 628916 [details] [diff] makes WebGL work on the Tegra 2!
Woohoo, the try run in comment 14 has actually been running the WebGL conformance tests. However it's crashed running them, but the crash is well-known, it's bug 746794 which is getting a fix right now.
Summary: Enable canvas tests on Android → Enable content/canvas and content/media tests on Android
New try, with hopefully:
 - detection of Android
 - custom list of failing WebGL tests on Android
 - avoiding the file:// tests in media on Android

https://tbpl.mozilla.org/?tree=Try&rev=6efbc63257af
Depends on: 760675
Regarding the above try push:
Good news: the file:// issue is fixed
Bad news: the list of known-failing WebGL test pages hasn't been picked up
Some logging info to understand the WebGL failing tests list problem:
https://tbpl.mozilla.org/?tree=Try&rev=78d9dab684d4
The issue was I forgot to add the new failing_tests_android.txt to the makefile. New try:
https://tbpl.mozilla.org/?tree=Try&rev=b81d92f4acbc
Now we are down to a few failures:

3406 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 0,0 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3408 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 99,0 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3412 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 0,49 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3414 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 99,49 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
15449 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_autoplay_contentEditable.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15451 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_autoplay_contentEditable.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15454 INFO TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_buffered.html | finished in a non-clean fashion (in /tests/content/media/test/test_autoplay_contentEditable.html)
15459 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug448534.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15462 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug448534.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15465 INFO TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug463162.xhtml | finished in a non-clean fashion (in /tests/content/media/test/test_bug448534.html)
15600 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug686137.html | Test timed out.
15685 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_constants.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15897 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_contentDuration1.html | Test timed out.
15900 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_contentDuration2.html | Test timed out.
15903 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_contentDuration3.html | Test timed out.
15904 ERROR TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | 4 test timeouts, giving up.
15905 ERROR TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | Skipping 139 remaining tests.
15906 INFO TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | finished in a non-clean fashion (in /tests/content/media/test/test_contentDuration3.html)



But these timeouts are preventing the rest from continuing so there might be more. Need to reproduce locally on tegra board on Monday.
Here's  the first test that times out, after 5 minutes: content/media/test/test_contentDuration1.html

<!DOCTYPE HTML>
<html>
<head>
  <title>Media test: seek test 1</title>
  <script type="text/javascript" src="/MochiKit/Base.js"></script>
  <script type="text/javascript" src="/MochiKit/DOM.js"></script>
  <script type="text/javascript" src="/MochiKit/Style.js"></script>
  <script type="text/javascript" src="/MochiKit/Signal.js"></script>
  <script type="text/javascript" src="/tests/SimpleTest/SimpleTest.js"></script>
  <link rel="stylesheet" type="text/css" href="/tests/SimpleTest/test.css" />
</head>
<body onunload="mediaTestCleanup();">
<pre id="test">
<script class="testbody" type="text/javascript">

function on_metadataloaded() {
  var v = document.getElementById('v');
  var d = Math.round(v.duration*1000);
  ok(d == 233, "Checking duration: " + d);
  SimpleTest.finish();
}

SimpleTest.waitForExplicitFinish();
</script>
</pre>
<video id='v'
       src='contentDuration1.sjs'
       onloadedmetadata='on_metadataloaded();'></video>
</body>
</html>


Any idea why it's timing out on Android?
Hm, the log doesn't show "Checking duration: " which means that on_metadataloaded() is never called.
could the .sjs file be problematic?  It appears it is reading [CurWorkD]/tests/content/media/test/320x240.ogv.  I bet we could pass in a rootdir as an argument to the .sjs file or something like that.

Here is the contentDuration1.sjs file:

function handleRequest(request, response)
{
  var file = Components.classes["@mozilla.org/file/directory_service;1"].
                        getService(Components.interfaces.nsIProperties).
                        get("CurWorkD", Components.interfaces.nsILocalFile);
  var fis  = Components.classes['@mozilla.org/network/file-input-stream;1'].
                        createInstance(Components.interfaces.nsIFileInputStream);
  var bis  = Components.classes["@mozilla.org/binaryinputstream;1"].
                        createInstance(Components.interfaces.nsIBinaryInputStream);
  var paths = "tests/content/media/test/320x240.ogv";
  var split = paths.split("/");
  for(var i = 0; i < split.length; ++i) {
    file.append(split[i]);
  }
  fis.init(file, -1, -1, false);
  bis.setInputStream(fis);
  var bytes = bis.readBytes(bis.available());
  response.setHeader("Content-Duration", "0.233", false);
  response.setHeader("Content-Length", ""+bytes.length, false);
  response.setHeader("Content-Type", "video/ogg", false);
  response.write(bytes, bytes.length);
  bis.close();
}
New try:
https://tbpl.mozilla.org/?tree=Try&rev=6d96c8b66d26

Indeed, thanks, that seems like it. I didn't know anything about nsIFileInputStream so searched for some documentation and found this:
https://developer.mozilla.org/en/Mochitest#How_do_I_get_access_to_the_files_on_the_server_as_XPCOM.C2.A0objects_from_an_SJS_script.3F_%281.9.3_or_later%29

This is talking about exactly the same "tests/content/media/test/320x240.ogv" file!
It seems that the handleRequest function in the sjs file hasn't been called. Indeed I put

  ok(true, "handleRequest 0");

at the beginning of handleRequest and it doesn't show up in the log.

Maybe the simplest is to just skip these contentDuration tests for now on Android.
Now we're really getting close:

3406 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 0,0 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3408 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 99,0 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3412 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 0,49 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
3414 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_canvas.html | pixel 99,49 of c343 is 4,251,0,255; expected 0,255,0,255 +/- 0
15449 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_autoplay_contentEditable.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15451 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_autoplay_contentEditable.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15454 INFO TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_buffered.html | finished in a non-clean fashion (in /tests/content/media/test/test_autoplay_contentEditable.html)
15461 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug448534.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15464 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug448534.html | an unexpected uncaught JS exception reported through window.onerror - TypeError: label is null at chrome://global/content/bindings/videocontrols.xml:688
15469 INFO TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_bug463162.xhtml | finished in a non-clean fashion (in /tests/content/media/test/test_bug448534.html)



I think I'll just disable on Android the remaining failing media tests (with follow-up bug), and fix the failing canvas test.
I am currently working on an android-failures.json which would specify tests to skip and run all by default.  This would work better for content/media tests to skip instead of manifest files.  I have it created, just running into oddities while doing additional testing.
(In reply to Joel Maher (:jmaher) from comment #29)
> I am currently working on an android-failures.json which would specify tests
> to skip and run all by default.  This would work better for content/media
> tests to skip instead of manifest files.  I have it created, just running
> into oddities while doing additional testing.

Great!

Notice that for the tests that are timing out, it's important to completely skip them as opposed to just mark them as known-to-fail and still run them.
No longer depends on: android_4.0_testing
No longer blocks: android_4.0_testing
New try with remaining media tests disabled:
https://tbpl.mozilla.org/?tree=Try&rev=48bc57638771

Now looking into the failing canvas2d test.
OK, it was a bug in the test_canvas test, it required an unrealistic canvas drawing tolerance of 0.098 (lower is higher-quality but slower).

New try with this fixed to only require a tolerance of 0.5:
https://tbpl.mozilla.org/?tree=Try&rev=aeff348c8479

(Will file separate bug about that test)
Assignee: nobody → bjacob
Attachment #627823 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #629917 - Flags: review?(jmaher)
We can't enable tests that time out, it's a waste of test slave time and we bail out after 4 of these timeouts.

Will file follow-up bug to enable them.
Attachment #629921 - Flags: review?(jmaher)
This probably conflicts about the work you're doing in the other bug, but IIUC you told me on IRC I should just go ahead and we'll drop this ugly Makefile hack when your fix makes it possible.
Attachment #629922 - Flags: review?(jmaher)
Not asking for review here, will attach it on a separate bug.
removed debug stuff
Attachment #629918 - Attachment is obsolete: true
Attachment #629918 - Flags: review?(jgilbert)
Attachment #629925 - Flags: review?(jgilbert)
Depends on: 761326
Comment on attachment 629925 [details] [diff] [review]
mark known failing WebGL conformance tests on Android

Review of attachment 629925 [details] [diff] [review]:
-----------------------------------------------------------------

Ouch, that's a lot of failing GLSL function tests.
Attachment #629925 - Flags: review?(jgilbert) → review+
Yeah, these Tegra 2 boards have 1+ year old drivers (or else they wouldn't fail to have contexts in same share group across 2 threads). That should get better when we get the pandaboards.
Comment on attachment 629922 [details] [diff] [review]
disable failing media tests on Android

Review of attachment 629922 [details] [diff] [review]:
-----------------------------------------------------------------

can we put a comment in here explaining why these are not run on android?  Also if you don't have objections, I would like to (in the future), move these into the android-failures.json from bug 761125 when that is ready to land.
Attachment #629922 - Flags: review?(jmaher) → review+
Comment on attachment 629921 [details] [diff] [review]
disable contentDuration tests on Android, they are timing out

Review of attachment 629921 [details] [diff] [review]:
-----------------------------------------------------------------

can we put a comment in here explaining why these are not run on android?  Also if you don't have objections, I would like to (in the future), move these into the android-failures.json from bug 761125 when that is ready to land.
Attachment #629921 - Flags: review?(jmaher) → review+
Attachment #629920 - Flags: review?(jmaher) → review+
Attachment #629917 - Flags: review?(jmaher) → review+
New try with more tests disabled according to last try log, and also, allowing more timeouts (and using a shorter timeout delay) so we can iterate faster.

https://tbpl.mozilla.org/?tree=Try&rev=5eead11120ca
New try, this time we should be at the end of media test failures, but there were some strange dom test failues in the previous push.
https://tbpl.mozilla.org/?tree=Try&rev=0750db6c1aaf
indeed, the other patches there and in Bug 761326 make them pass.
Attachment #631226 - Flags: review?(jmaher)
Attachment #629921 - Attachment is obsolete: true
Attachment #629922 - Attachment is obsolete: true
(In reply to Joel Maher (:jmaher) from comment #42)
> Comment on attachment 629922 [details] [diff] [review]
> disable failing media tests on Android
> 
> Review of attachment 629922 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> can we put a comment in here explaining why these are not run on android? 
> Also if you don't have objections, I would like to (in the future), move
> these into the android-failures.json from bug 761125 when that is ready to
> land.

This patch and the other one seem obsolete now that Bug 761125 landed. The above try push doesn't have them, I expect that android-failures.json is taking care of this now.
Comment on attachment 631220 [details] [diff] [review]
webgl mochitest needs a longer timeout on android

Review of attachment 631220 [details] [diff] [review]:
-----------------------------------------------------------------

+
Attachment #631220 - Flags: review?(jgilbert) → review+
Comment on attachment 631226 [details] [diff] [review]
remove canvas tests from android-failures

Review of attachment 631226 [details] [diff] [review]:
-----------------------------------------------------------------

great!  As a note, we are not using android-failures.json yet until I resolve some of the longer running tests.  If I can't get them resolved by fixing them this week, I will be turning off all the long running tests and getting this running.
Attachment #631226 - Flags: review?(jmaher) → review+
Ah ok! that explains the failures in the above try. New try, restoring the makefile hack to disable failing media tests:
https://tbpl.mozilla.org/?tree=Try&rev=2082090bebd6
Carrying forward r=jmaher.

This adds more failing tests per above try runs, and adds comments in the makefile where we do this, to point to this bug and mention that we want to get rid of that when the android-failures solution is ready.
Attachment #631358 - Flags: review+
Grreeeeeeeeeeeeeeeeeeeeeeeeeeeennnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn!!!!!!!!!!!!!!!1
Sorry, I backed this out on inbound because of failures in multiple media tests on desktop platforms:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3b2b9defe6f4
Target Milestone: Firefox 16 → ---
Try on all platforms with the manifest.js change reverted to see if that's what's causing it:
https://tbpl.mozilla.org/?tree=Try&rev=8b52381fccee
Cancelled. What's happening is that the manifest.js file isn't getting installed: the logs for my push don't mention

inflating: mochitest/tests/content/media/test/manifest.js  

contrary to other pushes.
Here was the bug:

_TEST_FILES = \
                ...
		manifest.js \
                ...
		$(NULL)

# Bug 759221
# These tests are not built on Android because they were failing when we
# enabled these tests on Android. The intent is to re-enable them as soon
# as possible, and/or switch to a cleaner way to manage known-to-fail tests.
ifneq ($(OS_TARGET),Android)
_TEST_FILES = \
		...
		$(NULL)
endif

The second _TEST_FILES = was overriding the first one, should have been +=.
New try:
https://tbpl.mozilla.org/?tree=Try&rev=09745adaa915

By the way: imperative build-systems suck.
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: