Closed Bug 1233565 Opened 9 years ago Closed 9 years ago

Intermittent TEST-UNEXPECTED-FAIL | apps/system/test/marionette/audio_channel_competing_test.js | Audio channel competing Notification audio channel competes with audio channels Notification

Categories

(Firefox OS Graveyard :: Gaia::System::Audio Mgmt, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::System::Audio Mgmt
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mikehenrty, Assigned: evanxd)

References

Details

(Keywords: intermittent-failure, Whiteboard: [MJS])

Attachments

(1 file)

[gaia] evanxd:bug-1233565 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
alwu
: review+
Details | Review 
TEST-UNEXPECTED-FAIL | apps/system/test/marionette/audio_channel_competing_test.js | Audio channel competing Notification audio channel competes with audio channels Notification channel competes with notification channel
Crash detected but error running stackwalk

This test fails a lot when we disable the retry logic in bug 1222215. I'm going to disable this test until we can get that landed.
Assignee: nobody → evan
Hi Michael, thanks for the help. And I think we can just disable the test cases failed a lot not all test cases in the test file 'apps/system/test/marionette/audio_channel_competing_test.js'.

Let me figure out this.
I re-triggered the audio channel competing test for 35 times[1]. Let's check which test will be failed a lot.

[1]: https://github.com/mozilla-b2g/gaia/pull/33635
In the 35 times of running test, we have 2 failures of `apps/system/test/marionette/audio_channel_competing_test.js`. But the error message only shows 
```
/home/worker/git_checkout/apps/system/test/marionette/audio_channel_competing_test.js failed. Will retry.               
Running tests in  /home/worker/git_checkout/apps/system/test/marionette/audio_channel_competing_test.js
```

But it said 
```
TEST-UNEXPECTED-FAIL | apps/system/test/marionette/audio_channel_competing_test.js | Audio channel competing Notification audio channel competes with audio channels Notification channel competes with notification channel
Crash detected but error running stackwalk
```
in Comment 0.

Let's try to disable the test(notification v.s. notification) then run the `apps/system/test/marionette/audio_channel_competing_test.js` test again. If there is no failure anymore, I think we can re-enable the `apps/system/test/marionette/audio_channel_competing_test.js` test and disable the "notification v.s. notification" test case at same time.
I disabled the notification channel competes notification channel test case and ran `apps/system/test/marionette/audio_channel_competing_test.js` test for 35 times. If the test are all passed, I think we can re-enable it.
Hi Michael,

The error message[1]
```
home/worker/git_checkout/apps/system/test/marionette/audio_channel_competing_test.js failed. Will retry.
Running tests in  /home/worker/git_checkout/apps/system/test/marionette/audio_channel_competing_test.js
```
looks weird. Do you know what is the meaning of it? Cannot debug without error message.

It didn't say which test case is failed or which assert is failed. It just said `home/worker/git_checkout/apps/system/test/marionette/audio_channel_competing_test.js failed`. Is this kind of failure more about JSMarionette framework, not about the test file?

If it is more about JSMarionette framework, I think we can re-enable the test file. How do you think?

And I also didn't see the error message like
```
EST-UNEXPECTED-FAIL | apps/system/test/marionette/audio_channel_competing_test.js | Audio channel competing Notification audio channel competes with audio channels Notification channel competes with notification channel
Crash detected but error running stackwalk
```
Could you show me the treeherder page?

Thanks.

[1]: https://tools.taskcluster.net/task-inspector/#FNSdPGE5QBWPHw1dFKygGA/0
Flags: needinfo?(mhenretty)
(In reply to Evan Tseng [:evanxd][:愛聞插低] from comment #10)
> Hi Michael,
> 
> The error message[1]
> ```
> home/worker/git_checkout/apps/system/test/marionette/
> audio_channel_competing_test.js failed. Will retry.
> Running tests in 
> /home/worker/git_checkout/apps/system/test/marionette/
> audio_channel_competing_test.js
> ```
> looks weird. Do you know what is the meaning of it? Cannot debug without
> error message.

This is one of the problems of the retry logic in bug 1222215, it hides the error messages when a test fails but then gets re-triggered and passes. In order to see the the error message when a test fails, first pull in this patch [1] which will disable the retry logic, and then re-trigger your test a bunch of times.

1.) https://tools.taskcluster.net/task-inspector/#FNSdPGE5QBWPHw1dFKygGA/0

> 
> It didn't say which test case is failed or which assert is failed. It just
> said
> `home/worker/git_checkout/apps/system/test/marionette/
> audio_channel_competing_test.js failed`. Is this kind of failure more about
> JSMarionette framework, not about the test file?
> 
> If it is more about JSMarionette framework, I think we can re-enable the
> test file. How do you think?

I'm not sure what is causing this error, but I don't think we can eliminate the test being the culprit just yet.

> 
> And I also didn't see the error message like
> ```
> EST-UNEXPECTED-FAIL |
> apps/system/test/marionette/audio_channel_competing_test.js | Audio channel
> competing Notification audio channel competes with audio channels
> Notification channel competes with notification channel
> Crash detected but error running stackwalk
> ```
> Could you show me the treeherder page?

If you look at Gij 18 here, you will see a couple
https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=ed9ebb758e19
Flags: needinfo?(mhenretty)
(In reply to Michael Henretty [:mhenretty] from comment #11)
> (In reply to Evan Tseng [:evanxd][:愛聞插低] from comment #10)
> > Hi Michael,
> > 
> > The error message[1]
> > ```
> > home/worker/git_checkout/apps/system/test/marionette/
> > audio_channel_competing_test.js failed. Will retry.
> > Running tests in 
> > /home/worker/git_checkout/apps/system/test/marionette/
> > audio_channel_competing_test.js
> > ```
> > looks weird. Do you know what is the meaning of it? Cannot debug without
> > error message.
> 
> This is one of the problems of the retry logic in bug 1222215, it hides the
> error messages when a test fails but then gets re-triggered and passes. In
> order to see the the error message when a test fails, first pull in this
> patch [1] which will disable the retry logic, and then re-trigger your test
> a bunch of times.
> 
> 1.) https://tools.taskcluster.net/task-inspector/#FNSdPGE5QBWPHw1dFKygGA/0
>
Let's remove retry logic and show error message in Bug 1222215. Then we can start to debug more easily.

> > 
> > It didn't say which test case is failed or which assert is failed. It just
> > said
> > `home/worker/git_checkout/apps/system/test/marionette/
> > audio_channel_competing_test.js failed`. Is this kind of failure more about
> > JSMarionette framework, not about the test file?
> > 
> > If it is more about JSMarionette framework, I think we can re-enable the
> > test file. How do you think?
> 
> I'm not sure what is causing this error, but I don't think we can eliminate
> the test being the culprit just yet.
> 
> > 
> > And I also didn't see the error message like
> > ```
> > EST-UNEXPECTED-FAIL |
> > apps/system/test/marionette/audio_channel_competing_test.js | Audio channel
> > competing Notification audio channel competes with audio channels
> > Notification channel competes with notification channel
> > Crash detected but error running stackwalk
> > ```
> > Could you show me the treeherder page?
> 
> If you look at Gij 18 here, you will see a couple
> https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=ed9ebb758e19

All failures in the treeherder page[1] are crash error, except [2][3]. In my experience, I don't think we can fix crash error with changing test code. Maybe this is a gecko issue? Need to find a way to figure out the crash problems.

Alastor, could you take a look of the crash issues?
Thanks.

[1]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=ed9ebb758e19
[2]: https://tools.taskcluster.net/task-inspector/#Anxgl-A_TE6kW-0ecGICxw/3
[3]: https://tools.taskcluster.net/task-inspector/#Anxgl-A_TE6kW-0ecGICxw/4
Flags: needinfo?(alwu)
Flags: needinfo?(alwu)
See Also: → 1175116
Depends on: 1218692
No longer depends on: 1218692
Blocks: 1096163
I used waitFor[1] to check audio channels' states rather than immediately check the states. I think it can fix the intermittent failures.

I run the `audio_channel_competing_test.js` for 20 times[2-21]. If all tests are passed, I think we can believe the test is already stable.

[1]: https://github.com/evanxd/gaia/commit/6e89d5e94bdadd9e76a41647cc0602dc5c8e08ef
[2]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=5df45e051c7df6971bc0dc2547cd28ec6cf13e18
[3]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=39d27f05f5d545734dad7b600ac11746c0a6a600
[4]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=f250daeec98003cfc5521f5421f60711f63377f2
[5]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=db51ba10e6cf6048d092b0b9d5755701fc32e47c
[6]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=6c4647a86f1bde491bd8c64d99f9a566ee664af4
[7]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=2b4c6751329ddc01a2dfb61fd27b90c5f05ce8bc
[8]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=a2a78e190ea841ac0302834f052cac3ee6783893
[9]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=dac159f75cc940f34e9217ef67a73116d4484c8e
[10]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=6cf59ebd89430d73e9befc3b3dfa22db7ca30637
[11]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=e66f0be377c3f115b7944e97aefd993e349846fb
[12]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=9dfc4b280917b33d0eefe92683fc1a5e939985d3
[13]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=8124f1c710be1895c7953cb316aa5c9e0a1244a7
[14]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=3287279588d3e75f9814bec9f222c625a8dec455
[15]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=15fa4d858775c4913279a2179c188f1437d4bef5
[16]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=dddd9adcf111e08f3df14d7967396550a1ad2efd
[17]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=cfb196c291c4fe8f1716963d34576394eba37c9e
[18]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=674024e869249194d04c4a8f961a6e9be23b9672
[19]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=b1f70461a991a05b7e65e2c254e4b2e9f8c21de9
[20]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=931f08372e7b9c648cc5a2433e31a8c1483eb40a
[21]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=1c91cd43dfa2a0e831f777d4514010535a928778
If the bug can be fixed by the patch[1], we also can fix Bug 1236307 with same way.

[1]: https://github.com/evanxd/gaia/commit/6e89d5e94bdadd9e76a41647cc0602dc5c8e08ef
Updated patch[21] to fix Bug 1236307 at same time. And I run the `audio_channel_competing_test.js` for 20 times[2-21]. If all tests are passed, I think we can believe the test is already stable.

[1]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=fa6928078037f39cdde731c4b9f4caece45af4fd
[2]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=7dbdbce3159697df3c41104c1c835d688572d260
[3]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=9c4d9a15c523ad8b305b0600c087164e81e0584d
[4]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=34ea212e0d85eef5d6d2fd2db4f608a0b6628c50
[5]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=263246a01072dc095baf3d78addeba2f7f7a9334
[6]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=9d7feef33d069ae02db83a179bda26dc6cbc2b98
[7]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=294b84c40763df218cfdd4cdd5be1af21bb311fa
[8]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=58a039682cd60d333dc92633527deabb0598b248
[9]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=70524bb470e5d59d45ed2a88cb57f1fcd95ebeb4
[10]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=1461bb12bea0ff9199d30db9e0ba02545f8ef1cf
[11]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=ca686b67fa91416b4cb28a508befb6d874c52ead
[12]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=166ba56a66913344a02b590fc93f10e59f33bc93
[13]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=b3ea6b2acb30b182915afdfd7c63371642ef0617
[14]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=2d0fdba7d63135be073b53890453ff33f8599a47
[15]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=b77f4eae2ff1d2fc58873fccb4ee83a0faae9760
[16]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=73685879480e2b3391fc958589139a64ce9019d3
[17]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=c9bef2338decd7362d6742c64f302177cdff3043
[18]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=8089fb6bd82c2c56a6e57fa168cea5fd05b02768
[19]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=b5100175278993b855dfd60adab2521d64ec2d97
[20]: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=fe52c2fe04a88c4d9c29efc97cad8446b8ae5bf5
[21]: https://github.com/evanxd/gaia/commit/2ad57c4c769c82a55097f2b17dc4bb6f2584f5b8
Comment on attachment 8700459 [details] [review]
[gaia] evanxd:bug-1233565 > mozilla-b2g:master

Hi Alastor,

Could you review the patch?

Thanks.
Attachment #8700459 - Flags: review?(alwu)
Comment on attachment 8700459 [details] [review]
[gaia] evanxd:bug-1233565 > mozilla-b2g:master

Only one nit, thanks!
Attachment #8700459 - Flags: review?(alwu) → review+
master: https://github.com/mozilla-b2g/gaia/commit/c5d58aa2d7474a0cec0fd9f48ec2967bb17e1155
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: