1205118 - [gatt] GijTV is mostly perma-failing

Status: RESOLVED FIXED

(Testing Graveyard :: JSMarionette, defect)

Description

[Aus Lacroix [:aus]]

Right now, most of the GijTV chunks are failing. It looks like things aren't running like they should, it's completely failing to run tests on a lot of these runs. This indicates some more basic problem either launching Mulet with the TV profile or something like that.

Investigating...

CC'ing timdream for Taipei visibility.

Comment 1

[Fischer [:Fischer]]

From the log, it fails to run mobile app tests, however, GijTV should only run tests set with devices: ['tv'].

Comment 2

[Fischer [:Fischer]]

GijTV should be able to skip those tests not marked as 'tv'.
Is something wrong with the test framework about this part ?

Comment 3

[yifan [:yifan][:yliao]]

GijTV fails since bug 1199429 landed.

Comment 4

[yifan [:yifan][:yliao]]

Another speculated cause is bug 1146713. Will ask Ricky for a look.

Comment 5

[Aus Lacroix [:aus]]

I'll be looking at this throughout the week but I have other high priority items on my plate as well. Any help on your end is appreciated. :)

Comment 6

[Ricky Chien [:rickychien] (inactive)]

If we try to split marionette tests into chunks as a different tasks like GijTV, marionetteJS should be able to filter out tv tests by itself.

We need to patch marionette entry script (gaia/bin/gaia-marionette/gaia-marionette) and set an another env flag for marionette_js_tv_tests.yml

Comment 7

[Fischer [:Fischer]]

Have a deeper look into failed tests:

For instance, in the test:
https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=f7b389fda35e0aac1027b00aeb05fc882face23c

Although GijTV10 tests tv apps, the tv app tests still fail due to 'Error: Polling socket recv() timeout!'
(log: https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/Hy4YdhpvTYuOv8xoba84jw/4/public/logs/live_backing.log)

GijTV8 tests mobile apps, it looks like the tests skipped so passes
(Log: https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/CQc9bRzRQQOeJgI0FSzc4w/0/public/logs/live_backing.log)

However, other GijTV1, 2, ... fails.

Could the problem has to do with marionette-client ?

Comment 8

[Fischer [:Fischer]]

Attachment: PR #31959

Comment 9

[Fischer [:Fischer]]

Revert Bug 1199429 - Use Mulet for all Gaia tests and skip some failed GijTV tests due to unknown test environment issue first.

For now  let GijTV tests can be passed first but we have to investigate more about root cause.

See Bug 1207453

Comment 10

[yifan [:yifan][:yliao]]

Comment on attachment 8664683 [details] [review]
PR #31959

Seems the patch to use mulet for test is reverted in your PR.

Comment 11

[Fischer [:Fischer]]

Add Bug 1199429 - Use Mulet for all Gaia tests back.
Please see: https://github.com/mozilla-b2g/gaia/pull/31959

Comment 12

[yifan [:yifan][:yliao]]

Comment on attachment 8664683 [details] [review]
PR #31959

Thanks Fischer!

Comment 13

[yifan [:yifan][:yliao]]

Merged into master:

https://github.com/mozilla-b2g/gaia/pull/31959

Comment 14

[Aus Lacroix [:aus]]

Clearing my NI? since this issue was resolved.

Comment 15

[Fischer [:Fischer]]

The patch for the bug 1231680 has fixed the environment issue that mobile tests are not skipped when testing TV tests.