1179604 - Intermittent apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite Thread draft is correctly saved

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::SMS, defect)

Comment 1

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1735930
repository: mozilla-central
start_time: 2015-07-01T15:32:05
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: non-buildbot b2g-linux64 test [TC] - Gaia JS Integration Test
revision: 427c2e135702

TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite Thread draft is correctly saved
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 2

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=11360086
repository: mozilla-inbound
start_time: 2015-07-03T13:27:22
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: non-buildbot b2g-linux64 test [TC] - Gaia JS Integration Test
revision: 3d5f3ef83150

TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite Thread draft is correctly saved
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=11645136
repository: mozilla-inbound
start_time: 2015-07-13T21:13:47
who: rvandermeulen[at]mozilla[dot]com
machine: unknown
buildname: non-buildbot b2g-linux64 test [TC] - Gaia JS Integration Test
revision: e4e144e4590e

TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite Thread draft is correctly saved
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 4

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1801735
repository: mozilla-central
start_time: 2015-07-14T03:49:58
who: rvandermeulen[at]mozilla[dot]com
machine: unknown
buildname: non-buildbot b2g-linux64 test [TC] - Gaia JS Integration Test
revision: e786406bc683

TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite Thread draft is correctly saved
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 5

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=12586515
repository: mozilla-inbound
start_time: 2015-08-06T12:44:39
who: rvandermeulen[at]mozilla[dot]com
machine: unknown
buildname: non-buildbot b2g-linux64 test [TC] - Gaia JS Integration Test
revision: f4610f8fd0be

AssertionError: expected false to be truthy
TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite "before each" hook
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 6

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=2194673
repository: mozilla-central
start_time: 2015-09-14T05:36:13
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: non-buildbot b2g-linux64 test [TC] - Gaia JS Integration Test
revision: fba4b0cd3823

TypeError: Cannot call method 'send' of undefined
AssertionError: expected false to be truthy
AssertionError: Message input should be focused: expected false to be true
TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts Messages Drafts Test Suite Draft is correctly saved
TEST-UNEXPECTED-FAIL | apps/sms/test/marionette/drafts_test.js | Messages Drafts "after each" hook
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 7

[Oleg Zasypkin [:azasypkin]]

I believe drafts tests became more stable in the scope of bug 1176976, so closing this one as FIXED.

Comment 8

[Michael Henretty [:mikehenrty][:mhenretty]]

I'm seeing a lot of drafs_test.js errors after I disable the retry logic in bug 1222215. For instance:

https://treeherder.mozilla.org/logviewer.html#?job_id=3173866&repo=gaia

I'm going to disable this test for now so that we can land bug 1222215.

Comment 9

[Michael Henretty [:mikehenrty][:mhenretty]]

disabled in master: https://github.com/mozilla-b2g/gaia/commit/22355603dd1573c3014e7bcedfacef05c20ca0b2

Comment 10

[Oleg Zasypkin [:azasypkin]]

Thanks Michael, will take a look.

Comment 11

[Oleg Zasypkin [:azasypkin]]

Okay, looks like all our troubles come from the fact that sometimes IndexedDB/asyncStorage is too slow on Treeherder that leads to a lot of different races in tests that based on asyncStorage - drafts and activities (likely the same culprit for bug 1233552 as well).

We need a reliable way to determine that view is visually-loaded when draft/drafts is/are loaded as well. Looking into this.

Comment 12

[GitHub Autolander Bot]

Attachment: [gaia] azasypkin:bug-1179604-drafts-tests > mozilla-b2g:master

Comment 13

[Oleg Zasypkin [:azasypkin]]

Comment on attachment 8701510 [details] [review]
[gaia] azasypkin:bug-1179604-drafts-tests > mozilla-b2g:master

Hey Steve,

How does it look to you?

I was running Messages tests with this PR and disabled retry logic (see the link below, Gij1 includes all Messages integration tests) and it seems to be fine:

https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=029edcb905d7e476d6637840aec96ddf698a626d

Thanks!

Comment 14

[Steve Chung [:steveck]]

Comment on attachment 8701510 [details] [review]
[gaia] azasypkin:bug-1179604-drafts-tests > mozilla-b2g:master

This solution sounds fine, just wondering if there's other possible way to validate the accessibility of the send button(or the activity view).

Comment 15

[Oleg Zasypkin [:azasypkin]]

(In reply to Steve Chung [:steveck] from comment #14)
> Comment on attachment 8701510 [details] [review]
> [gaia] azasypkin:bug-1179604-drafts-tests > mozilla-b2g:master
> 
> This solution sounds fine, just wondering if there's other possible way to
> validate the accessibility of the send button(or the activity view).

Thanks for feedback! Replied at GitHub, in short don't have good alternative idea right now, would be glad to get any pointer :)

Comment 16

[Oleg Zasypkin [:azasypkin]]

Comment on attachment 8701510 [details] [review]
[gaia] azasypkin:bug-1179604-drafts-tests > mozilla-b2g:master

Hey Steve,

I've added unit tests as a separate commit, also please tell me if you have any good idea about "conversaion-view-is-totally-ready" flag! I can't really wrap up my mind to do anything easy and non-intrusive :/

But anyway maybe we can improve this later to get our tests back as soon as possible :)

Thanks!

Comment 17

[Steve Chung [:steveck]]

Comment on attachment 8701510 [details] [review]
[gaia] azasypkin:bug-1179604-drafts-tests > mozilla-b2g:master

Not sure whether we should consider compose input as a part of conversation view readiness. Current solution is fine for me and we can discuss more about the conversation view ready flag and the additional benefit for it. Thanks!

Comment 18

[Oleg Zasypkin [:azasypkin]]

Thanks for review!

I've rebased PR on the latest master since bug 1222215 has been landed - Gij 16 that includes drafts looks good for 6 consequent runs. Gij 17 failed once for new_activity_test - but I'll look into it in the scope of 1235842.

Master: https://github.com/mozilla-b2g/gaia/commit/a6a054cc21f029dc8a330bfbf7d5438a93a7501a