946644 - Update the gaia-ui-tests for running on emulators

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

There are some changes needed to get the gaia-ui-tests running on emulators. This bug will track that work.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/14395

Pointer to Github pull-request

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8342989 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/14395

This initial patch should make sure the unit tests pass on emulator and the camera tests are marked as expected to fail due to bug 946375. I'm going to raise another bug to get the tests running against the emulator on the cedar branch, as running them locally is painfully slow.

Comment 3

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8342989 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/14395

nice, thanks for doing this!

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed unit/camera patch in:
https://github.com/mozilla-b2g/gaia/commit/fcc57fefa7fc78326925ce82c60839083b8b9fcf (master)
https://github.com/mozilla-b2g/gaia/commit/2f144b82da3680c512432712d92a0efa92c2a7d6 (v1.2)

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/14511

Pointer to Github pull-request

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8344749 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/14511

My last patch introduced a syntax issue, which caused the tests to be skipped on all devices.

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8344749 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/14511

ah yes, sorry for not catching that!

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed in:
https://github.com/mozilla-b2g/gaia/commit/6d8df043e1eafa576c16f22bcd2579012b959fd6 (master)
https://github.com/mozilla-b2g/gaia/commit/8ef8d95d1e91b94570ea5a92223df6470522053f (v1.2)

Comment 9

[Zac C (:zac)]

Looks like this was merged in above comment #8.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

No, this is blocked and there's much more work to do here.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Bug 946649 is resolved and we have many test failures on cedar. They look to be mostly related to timeouts, which is not surprising considering the emulator performance. I pick up my new work laptop next week so once I have that I'll set up a VM and look into this again.

Comment 12

[Florin Strugariu [:Bebe]]

Dave any updates on this? Can we help with anything?

Comment 13

[Jonathan Griffin (:jgriffin)]

Every test is currently timing out on the emulator.  You could try running them locally and debug this if you have time.

Comment 14

[Jonathan Griffin (:jgriffin)]

I tried this myself locally and found that the tests generally run fine; I think the problem is that the VM they're running on in TBPL is not powerful enough to run them in any reasonable amount of time.  I'm going to file a bug to try running them on real hardware and see if that allows them to pass.

Comment 15

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Recently, it's confirmed by QA that gaia-ui-tests run well on local emulator. 

Per comment 13, this seems be talking about gaia-ui-tests for running on emulators on Try. Make the bug description more descriptive.

Comment 16

[Hsin-Yi Tsai (she/her) [:hsinyi]]

So it seems a duplicate of bug 916368? Or, what are the scopes these two bugs focus on?

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Bug 916368 is for running the Gaia UI tests on buildbot and reporting to Treeherder. This bug is about getting the tests into shape for running on the emulator (and adjusting the manifests if any need to be skipped). For example, we can't run camera or wifi tests on the emulator.

Comment 18

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Dave Hunt (:davehunt) from comment #17)
> Bug 916368 is for running the Gaia UI tests on buildbot and reporting to
> Treeherder. This bug is about getting the tests into shape for running on
> the emulator (and adjusting the manifests if any need to be skipped). For
> example, we can't run camera or wifi tests on the emulator.

Thanks for the clarification.

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

These tests are being replaced with MarionetteJS tests, so this won't be needed.