Closed Bug 1005907 Opened 10 years ago Closed 10 years ago

Run b2g desktop tests in docker image

Categories

(Taskcluster :: General, defect)

Product:
Taskcluster
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dminor, Assigned: gbrown)

References

Details

We currently run mochitests and reftests for B2G desktop builds. Getting these running (and passing) inside a docker image is the first step to being able to run these tests on taskcluster.

For mochitest, we currently have the following failures:

 1958 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_2d.fillText.gradient.html | pixel 15,49 is 63,191,0,255; expected 0,255,0,255 +/- 5
 9320 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_bug949946.html | Assertion count 15 is greater than expected range 0-0 assertions.
 9411 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_multi_sharedWorker.html | Assertion count 407 is greater than expected range 0-0 assertions.
 9417 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_multi_sharedWorker_lifetimes.html | Assertion count 301 is greater than expected range 0-0 assertions.
 9423 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_navigator.html | Assertion count 1 is greater than expected range 0-0 assertions.
 9464 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_rvals.html | Assertion count 3 is greater than expected range 0-0 assertions.
 9470 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_sharedWorker.html | Assertion count 65 is greater than expected range 0-0 assertions.
 9481 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_terminate.html | Assertion count 2 is greater than expected range 0-0 assertions.
 9492 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_threadTimeouts.html | Assertion count 1 is greater than expected range 0-0 assertions.
 10329 INFO TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_event_target_radius.html | checking 't' offset 10,135.40000915527344 [basic functionality]; got null
 TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_remote_passpointerevents.html | application terminated with exit code 11

Some of these failures and the process crash seem to be fallout from getting webgl running - I used to get a full test run and a handful of failures.. More investigation is required.

For reftests, there are around 350 failures which I have not begun investigating, although the number was around 500 prior to having webgl working.
Assignee: nobody → gbrown
What I did:

 - clone https://github.com/taskcluster/gecko-images
 - in gecko-images, update FROM path in tester/Dockerfile
 - make gecko-base && make gecko-tester
   - (tester/test_setup.sh fails on rm $0 -- but that doesn't break anything)
 - verify registry.taskcluster.net/root/gecko-tester image created
 - rather than make check-tester or running b2g_desktop_tests.sh, I ran a shell on gecko-tester and
    - xvfb-run -s "-screen 0 800x1000x24" python mozharness/scripts/b2g_desktop_unittest.py --no-read-buildbot-config --config-file /home/worker/b2g-desktop-config.py --download-symbols ondemand --test-suite mochitest --installer-url ftp://ftp.mozilla.org/pub/b2g/nightly/2014/07/2014-07-05-16-02-02-mozilla-central/b2g-33.0a1.multi.linux-x86_64.tar.bz2 --test-url ftp://ftp.mozilla.org/pub/b2g/nightly/2014/07/2014-07-05-16-02-02-mozilla-central/b2g-33.0a1.multi.linux-x86_64.tests.zip

What I get:

16:22:11     INFO -  2026 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_2d.fillText.gradient.html | pixel 15,49 is 63,191,0,255; expected 0,255,0,255 +/- 5
16:22:18     INFO -  2271 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl-mochitest/test_webgl_available.html | Expected WebGL creation to succeed.
16:22:18     INFO -  2285 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl-mochitest/test_webgl_request_context.html | Request for 'experimental-webgl' should succeed.
16:23:49     INFO -  5228 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_823965.html | Should have gotten a storage
16:23:49     INFO -  5229 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_823965.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_823965.html:96
16:23:49     INFO -  5235 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_addCorrectType.html | uncaught exception - TypeError: navigator.getDeviceStorage(...) is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_addCorrectType.html:31
16:23:49     INFO -  5241 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_available.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_available.html:39
16:23:49     INFO -  5250 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_basic.html | Should have gotten a storage
16:23:49     INFO -  5251 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_basic.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_basic.html:130
16:23:49     INFO -  5257 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_dotdot.html | uncaught exception - TypeError: testingStorage(...) is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_dotdot.html:35
16:23:49     INFO -  5266 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerate.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_enumerate.html:92
16:23:49     INFO -  5272 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerateMultipleContinue.html | uncaught exception - TypeError: navigator.getDeviceStorage(...) is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_enumerateMultipleContinue.html:33
16:23:49     INFO -  5278 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerateOptions.html | enumerate no parameter
16:23:49     INFO -  5279 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerateOptions.html | enumerate one string parameter
16:23:49     INFO -  5283 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerateOptions.html | enumerate a string and object parameter
16:23:49     INFO -  5284 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerateOptions.html | enumerate object parameter with path
16:23:49     INFO -  5288 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_enumerateOptions.html | enumerate object parameter with path
16:23:49     INFO -  5294 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_freeSpace.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_freeSpace.html:53
16:23:49     INFO -  5308 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_basic.html | Should have gotten a storage
16:23:49     INFO -  5309 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_basic.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_fs_basic.html:62
16:23:49     INFO -  5318 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_createDirectory.html | Should have gotten a storage.
16:23:49     INFO -  5319 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_createDirectory.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_fs_createDirectory.html:96
16:23:49     INFO -  5328 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_createFile.html | Should have gotten a storage.
16:23:49     INFO -  5329 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_createFile.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_fs_createFile.html:119
16:23:49     INFO -  5336 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_get.html | Should have gotten a storage.
16:23:49     INFO -  5337 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_get.html | uncaught exception - TypeError: gStorage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_fs_get.html:181
16:23:49     INFO -  5346 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_remove.html | Should have gotten a storage.
16:23:49     INFO -  5347 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_fs_remove.html | Failed to created test files.
16:23:49     INFO -  5353 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_lastModificationFilter.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_lastModificationFilter.html:73
16:23:49     INFO -  5367 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_overwrite.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_overwrite.html:79
16:23:49     INFO -  5378 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_sanity.html | pictures - Should have getDeviceStorage
16:23:49     INFO -  5379 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_sanity.html | music - Should have getDeviceStorage
16:23:49     INFO -  5380 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_sanity.html | videos - Should have getDeviceStorage
16:23:49     INFO -  5381 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_sanity.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_sanity.html:50
16:23:49     INFO -  5387 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_usedSpace.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_usedSpace.html:53
16:23:49     INFO -  5393 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_watch.html | Should have storage
16:23:49     INFO -  5394 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_watch.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_watch.html:64
16:23:49     INFO -  5400 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_watchOther.html | Should have storage
16:23:49     INFO -  5401 INFO TEST-UNEXPECTED-FAIL | /tests/dom/devicestorage/test/test_watchOther.html | uncaught exception - TypeError: storage is null at http://mochi.test:8888/tests/dom/devicestorage/test/test_watchOther.html:69
16:29:10     INFO -  10927 INFO TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_event_target_radius.html | checking 't' offset 10,135.40000915527344 [basic functionality]; got null

01:12:15     INFO -  12485 INFO Passed:  381790
01:12:15  WARNING -  12486 INFO Failed:  41
01:12:15  WARNING -  One or more unittests failed.
01:12:15     INFO -  12487 INFO Todo:    8729
01:12:15     INFO -  12488 INFO Slowest: 36071ms - /tests/image/test/mochitest/test_bug733553.html

...mostly devicestorage problems -- hopefully with a common cause?
Looks like the filesystem docker provides isn't compatible with our storage API.  Adding jonas and james to see if they have any ideas.
I'm also curious why these devicestorage issues weren't seen in dminor's original run; has the docker image changed since then in some relevant way?
(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I'm also curious why these devicestorage issues weren't seen in dminor's
> original run; has the docker image changed since then in some relevant way?

I wonder about that too. There have been no changes to gecko-images since April 17.
Wrt. to docker filesystem acting weird, then this could happen if you use the devicemapper storage driver. See "docker info", if you don't have AUFS installed it'll fallback to devicemapper which gave us some weird disk-space issues earlier. So make sure your kernel has the AUFS module, check `docker info`.

That said, I doubt it's a docker specific issue. The default ubuntu image is really slim, it doesn't even feature SSL certificates by default. So if the "devicestorage" specific code for b2g-desktop depends on XDG or something like that, then this could very well be missing or configured incorrectly on the image.

Note, if you want to test the file system theory, you can mount a folder from the host filesystem directly in a docker container and run the tests from that folder. Or try data folders with docker, it could be AUFS acting out, but I doubt it. Missing dependency sounds more likely.
Sorry, I was hoping to get things further along before committing the changes I had made. Since gbrown is looking at this now, I've pushed things to here: https://github.com/taskcluster/gecko-images/pull/4
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> Looks like the filesystem docker provides isn't compatible with our storage
> API.  Adding jonas and james to see if they have any ideas.

It is just missing the Documents, Pictures, etc. directories in the user home directory. I'm not going to admit how long it took me to realize that was the problem :)
Thank you Dan! With dminor's xdg .config directory and mesa installation, I rebuilt my gecko-tester image and re-ran the test from Comment 1. That brings our failures down to:

16:28:02     INFO -  2026 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/test_2d.fillText.gradient.html | pixel 15,49 is 63,191,0,255; expected 0,255,0,255 +/- 5
16:31:34     INFO -  9859 INFO TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_multi_sharedWorker_lifetimes.html | Still have data stored - got undefined, expected 0123456789abcdefghijklmnopqrstuvwxyz
16:35:11     INFO -  10915 INFO TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_event_target_radius.html | checking 't' offset 10,135.40000915527344 [basic functionality]; got null
16:35:44     INFO -  11525 INFO TEST-UNEXPECTED-FAIL | /tests/layout/style/test/test_animations_omta_start.html | uncaught exception - TypeError: a is null at http://mochi.test:8888/tests/layout/style/test/animation_utils.js:399
16:35:44     INFO -  11526 INFO TEST-UNEXPECTED-FAIL | /tests/layout/style/test/test_animations_omta_start.html | test left refresh driver under test control

16:36:22     INFO -  12418 INFO Passed:  382769
16:36:22  WARNING -  12419 INFO Failed:  5
16:36:22  WARNING -  One or more unittests failed.
16:36:22     INFO -  12420 INFO Todo:    8735
16:36:22     INFO -  12421 INFO Slowest: 36086ms - /tests/image/test/mochitest/test_bug733553.html
I merged dminors PR. In some ways this is similar to how I had to hardcode environment variables that is normally configured at login:
https://github.com/taskcluster/gecko-images/blob/master/base/Dockerfile#L10-L13

We could try to simulate the login process by using xdg-user-dirs-update, but hardcoding them in is also a sane solution.
(In reply to Geoff Brown [:gbrown] from comment #8)
> 16:31:34     INFO -  9859 INFO TEST-UNEXPECTED-FAIL |
> /tests/dom/workers/test/test_multi_sharedWorker_lifetimes.html | Still have
> data stored - got undefined, expected 0123456789abcdefghijklmnopqrstuvwxyz

This one is intermittent and matches bug 995688.
Depends on: 995688
Current b2g desktop builds on mozilla-central only run the sanity reftests:

/tools/buildbot/bin/python scripts/scripts/b2g_desktop_unittest.py --cfg b2g/desktop_automation_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest-sanity/reftest.list ...

The reftest-sanity tests also pass on docker -- no problems there.

It looks like we try to run all b2g desktop reftests on Cedar only, but there are many failures. eg. https://tbpl.mozilla.org/php/getParsedLog.php?id=43296570&tree=Cedar&full=1. Results on the docker image are approximately consistent -- very similar failures to those seen on Cedar. See bug 773482, bug 783632, etc.
That's good news.  I don't mind disabling 5 mochitests to get things going here, if needed.  James, Jonas, when do you need these running in production?  If the answer is 'now', let's disable these tests for b2gdesktop.

If it's not 'now', let's spend a little time investigating them; other than the workers test, most of them seem like rendering differences.
Depends on: 1040965
By the end of next week is ideally when we are ready for this to be turned on (sooner is great but having the those tests greened up would be great) if we can't achieve this by then we can disable tests.
sounds good to me
Depends on: 1040987
Depends on: 1041017
Taras wants to move quickly with TC test greening, so let's go ahead and disable these 5 tests so we can progress to working on emulator tests in bug 1027249.
James, Jonas, other than the greening, is there anything else you need from the A-team wrt b2g desktop tests?
Flags: needinfo?(jopsen)
Flags: needinfo?(jlal)
It may be worth noting that the Docker images are Ubuntu 13.10 (kernel 3.11.0-26-generic) while the aws slaves used to run tests in continuous integration (tbpl) are Ubuntu 12.04 LTS (kernel 3.2.0-23-generic). I don't know if that is causing any difference in test behavior -- just another difference between the environments to keep in mind.
It's fairly likely.  Changes such as these frequently cause new oranges appear when we make those changes deliberately.
(In reply to Jonathan Griffin (:jgriffin) from comment #15)
> Taras wants to move quickly with TC test greening, so let's go ahead and
> disable these 5 tests so we can progress to working on emulator tests in bug
> 1027249.

Those are disabled now on mozilla-central via bugs 1040965, 1040987, and 1041017. (There are really only 3 tests, since one of the 5 failures in Comment 8 was a low-frequency intermittent and 2 of the failures were caused by the same test.)
@gbrown,
Docker containers uses the host kernel, which on our slaves and vagrant box is Ubuntu 13.10.
Changing gecko-images to use "FROM ubuntu:12.04" would be possible, but it would still use the host kernel.

Anyways, while I have absolutely no experience porting tests to new kernel versions. I would be surprised if change the kernel broke tests. It seems much more likely (to me) that tests are broken because of different package/library versions on Ubuntu 12.04 and 13.10.
Also I might be wrong, but I might have heard something about the buildbot setup using X and not xvfb, but I might be wrong.

Note: It's probably better to push "ubuntu:12.04" to "registry.taskcluster.net/jonasfj/ubuntu:12.04" and then use "FROM registry.taskcluster.net/jonasfj/ubuntu:12.04" to ensure the image is loaded from our repository, in case someone wanted to try this.

@jgriffin,
Greening tests certainly goes a long way :)
I'm not sure what else we need. Better structure/docs/tests for gecko-images might be nice.
So we don't break tests that has been ported, but it's not critical at this stage.
Maybe jlal have other requests?
Flags: needinfo?(jopsen)
Greening the tests goes a huge way to making this stuff real++

The other thing in my mind is moving the code for the image generation into the tree... There is no technical reason that I can think of why we would need to move it there but it may be more obvious as to how this stuff is generated if it lives alongside gecko.

Feedback on painpoints / stuff to make this faster/more obvious is also desired (though we have plans / ideas floating around in progress which should help soon).
Flags: needinfo?(jlal)
Are these tests actually running in TC right now?  If not it would be good to add them, so we make sure they stay green.
(In reply to James Lal [:lightsofapollo] Unvailable until 6/30 from comment #21)
> Greening the tests goes a huge way to making this stuff real++
> 
> The other thing in my mind is moving the code for the image generation into
> the tree... There is no technical reason that I can think of why we would
> need to move it there but it may be more obvious as to how this stuff is
> generated if it lives alongside gecko.
> 
> Feedback on painpoints / stuff to make this faster/more obvious is also
> desired (though we have plans / ideas floating around in progress which
> should help soon).

+1 on making things more obvious.
I think we are done here: b2g-desktop mochitests and sanity reftests run green in the docker image.

We only disabled a few tests, and bugs for those tests have been filed and remain open.

Part of https://github.com/taskcluster/gecko-images/pull/8 captures the mozharness command lines in make targets.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.