1127939 - Intermittent failing test, TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Search, defect)

Description

[Kevin Grandon :kgrandon]

Error: timeout exceeded!
02:50:10     INFO -      at Object.Client.waitForSync (/builds/slave/test/gaia/node_modules/marionette-client/lib/marionette/client.js:693:16)
02:50:10     INFO -      at Object.Client.waitFor (/builds/slave/test/gaia/node_modules/marionette-client/lib/marionette/client.js:661:60)
02:50:10     INFO -      at Context.<anonymous> (/builds/slave/test/gaia/apps/search/test/marionette/suggestions_test.js:51:12)
02:50:10     INFO -      at callFn (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:223:21)
02:50:10     INFO -      at Test.Runnable.run (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:216:7)
02:50:10     INFO -      at Runner.runTest (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:373:10)
02:50:10     INFO -      at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:451:12
02:50:10     INFO -      at next (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:298:14)
02:50:10     INFO -      at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:308:7
02:50:10     INFO -      at next (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:246:23)
02:50:10     INFO -      at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:270:7
02:50:10     INFO -      at done (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:185:5)
02:50:10     INFO -      at callFn (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:228:7)
02:50:10     INFO -      at Hook.Runnable.run (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:216:7)
02:50:10     INFO -      at next (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:258:10)
02:50:10     INFO -      at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:270:7
02:50:10     INFO -      at done (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:185:5)
02:50:10     INFO -      at /builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:199:9
02:50:10     INFO -      at Object.executeHook (/builds/slave/test/gaia/node_modules/marionette-client/lib/marionette/client.js:371:18)
02:50:10     INFO -      at process._tickCallback (node.js:415:13)

Comment 1

[Kevin Grandon :kgrandon]

Attachment: Screenshot at point of failure

Comment 2

[Kevin Grandon :kgrandon]

This is failing fairly frequently, so disabling for now: https://github.com/mozilla-b2g/gaia/commit/b33a1afb538e8c7e12f6fb792c454da34f9f5b76

Dale - can you take a look if you have a minute?

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=968321
repository: mozilla-central
start_time: 2015-01-30T10:33:36
who: gwagner[at]mozilla[dot]com
machine: tst-linux64-spot-722
buildname: b2g_ubuntu64_vm mozilla-central opt test gaia-js-integration-3
revision: d7e156a7a0a6

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 4

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=b2g-inbound&job_id=1262553
repository: b2g-inbound
start_time: 2015-01-30T10:44:34
who: gwagner[at]mozilla[dot]com
machine: tst-linux64-spot-883
buildname: b2g_ubuntu64_vm b2g-inbound opt test gaia-js-integration-3
revision: 9d4fa68b4cd9

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions
make: *** [test-integration-test] Error 1
Return code: 2
Tests exited with return code 2: harness failures
# TBPL FAILURE #

Comment 5

[Dale Harvey (:daleharvey)]

Doh sorry, taking a look

Comment 6

[Dale Harvey (:daleharvey)]

Attachment: https://github.com/mozilla-b2g/gaia/pull/27835

The screenshot and logs suggested that the search app couldnt load some of its own resources, havent been able to reproduce in a while and have 10 passing runs @ https://treeherder.mozilla.org/#/jobs?repo=gaia-try&revision=ba2019828880

Comment 7

[Kevin Grandon :kgrandon]

Comment on attachment 8559823 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/27835

Hmm, I really hope we're not masking over this with the new automatic retries, but sounds good. Let's try landing it for now.

Comment 8

[Kevin Grandon :kgrandon]

In master: https://github.com/mozilla-b2g/gaia/commit/e3a04fc63f25e8fb57a344c96f434c5cf173608a

Comment 9

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=1911878
repository: fx-team
start_time: 2015-02-05T18:33:13
who: philringnalda[at]gmail[dot]com
machine: tst-linux64-spot-324
buildname: b2g_ubuntu64_vm fx-team opt test gaia-js-integration-3
revision: 0870c45ca038

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 10

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=b2g-inbound&job_id=1299649
repository: b2g-inbound
start_time: 2015-02-05T18:32:28
who: philringnalda[at]gmail[dot]com
machine: tst-linux64-spot-838
buildname: b2g_ubuntu64_vm b2g-inbound opt test gaia-js-integration-3
revision: b9670101c765

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 11

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=6334226
repository: mozilla-inbound
start_time: 2015-02-05T18:39:02
who: philringnalda[at]gmail[dot]com
machine: tst-linux64-spot-789
buildname: b2g_ubuntu64_vm mozilla-inbound opt test gaia-js-integration-3
revision: 36e23204d256

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 12

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=b2g-inbound&job_id=1300373
repository: b2g-inbound
start_time: 2015-02-05T19:49:20
who: philringnalda[at]gmail[dot]com
machine: tst-linux64-spot-447
buildname: b2g_ubuntu64_vm b2g-inbound opt test gaia-js-integration-3
revision: 831271a6add5

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 13

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=6336140
repository: mozilla-inbound
start_time: 2015-02-05T19:49:57
who: philringnalda[at]gmail[dot]com
machine: tst-linux64-spot-490
buildname: b2g_ubuntu64_vm mozilla-inbound opt test gaia-js-integration-3
revision: 2be92aa69102

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 14

[Kevin Grandon :kgrandon]

Appears this is still happening. You can see in the log it retries the same test several times. This means likely there is some obscure platform issue here, or we might have a bug with our build system/profile/mozsettings =/

Log: http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound-linux64_gecko/1423191863/b2g-inbound_ubuntu64_vm-b2gdt_test-gaia-js-integration-3-bm54-tests1-linux64-build122.txt.gz

Backout: https://github.com/mozilla-b2g/gaia/commit/f6b79487eb188e7a818d578a23c54b10b684b6d3

Comment 15

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=6341623
repository: mozilla-inbound
start_time: 2015-02-06T01:00:02
who: tomcat[at]mozilla[dot]com
machine: tst-linux64-spot-829
buildname: b2g_ubuntu64_vm mozilla-inbound opt test gaia-js-integration-3
revision: 9c95e28087ca

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 16

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=6347127
repository: mozilla-inbound
start_time: 2015-02-06T06:38:46
who: rvandermeulen[at]mozilla[dot]com
machine: tst-linux64-spot-607
buildname: b2g_ubuntu64_vm mozilla-inbound opt test gaia-js-integration-3
revision: b4ea9b9f29e9

npm ERR! Error: ENOENT, open '/home/cltbld/.npm/c242ae5a-marionette-js-runner-0-6-0.lock'
make: *** [node_modules] Error 34
Return code: 2
TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 17

[Dale Harvey (:daleharvey)]

Ok figured this one out by sprinking random long timeouts, trying to come up with a fix

Comment 18

[Dale Harvey (:daleharvey)]

Kevin, cant quite figure out how to test this reliably, any idea?

    home.waitForLaunch();
    home.focusRocketBar();

    rocketbar.enterText('sometext');
    search.goToResults();
    // Ensure we get 2 results (hardcoded in the provider results)
    client.waitFor(function() {
      return client.findElements(search.Selectors.suggestions).length === 2;
    });

The issue is the SearchProvider constructor is async, it reads xhr / settings before being set, if the search is done before its set it returns empty results, I thought I could put the rocketbar.enterText('sometext'); inside the waitFor loop, but it exits on the first loop

Comment 19

[Kevin Grandon :kgrandon]

I think I would prefer to refactor the SearchProvider to ensure that we search after fetching the async settings - otherwise it may be possible to miss a search from a user? This is unlikely though..

Another solution would be to add some hook to the UI so we know we're ready.

I would prefer to refactor the provider a bit though so we store the searched terms, and search them after the constructor async init is done.

Comment 20

[GitHub Autolander Bot]

Attachment: [PullReq] daleharvey:1127939 to mozilla-b2g:master

Comment 21

[Dale Harvey (:daleharvey)]

Comment on attachment 8560529 [details] [review]
[PullReq] daleharvey:1127939 to mozilla-b2g:master

Well that was obvious, sorry should have figured that, will run a bunch on try and ensure there arent too many restarts before landing

Comment 22

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=6351744
repository: mozilla-inbound
start_time: 2015-02-06T08:54:26
who: rvandermeulen[at]mozilla[dot]com
machine: tst-linux64-spot-469
buildname: b2g_ubuntu64_vm mozilla-inbound opt test gaia-js-integration-3
revision: e23d1b5c00d6

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 23

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=6353845
repository: mozilla-inbound
start_time: 2015-02-06T09:12:59
who: rvandermeulen[at]mozilla[dot]com
machine: tst-linux64-spot-251
buildname: b2g_ubuntu64_vm mozilla-inbound opt test gaia-js-integration-3
revision: 271298feef37

TEST-UNEXPECTED-FAIL | apps/search/test/marionette/suggestions_test.js | Search - Suggestions Test Test suggestions

Comment 24

[Kevin Grandon :kgrandon]

Comment on attachment 8560529 [details] [review]
[PullReq] daleharvey:1127939 to mozilla-b2g:master

Looks good, I think we have an extraneous `resolve`. Fix that and I think we can land. Thanks!

Comment 25

[Dale Harvey (:daleharvey)]

Removed the extra resolve, landed - https://github.com/mozilla-b2g/gaia/commit/d93a7f38449c7849e7724fbcc6f2557813b11248, thanks

Comment 26

[Dale Harvey (:daleharvey)]

Wanna uplift to make sure 2.2 is green too

Comment 27

[Dale Harvey (:daleharvey)]

I cant set feature-b2b-2.2, can you, this isnt a blocking bug so much as part of feature dev

Comment 28

[Gregor Wagner [:gwagner]]

This is not blocking. Just request uplift approval.

Comment 29

[Dale Harvey (:daleharvey)]

Comment on attachment 8560529 [details] [review]
[PullReq] daleharvey:1127939 to mozilla-b2g:master

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): Regressed by https://bugzilla.mozilla.org/show_bug.cgi?id=1098494 feature dev
[User impact] if declined: Can impact ability to test (and not regress) the rest of feature dev on the branch
[Testing completed]: This patch increases the reliability of the tests 
[Risk to taking this patch] (and alternatives if risky): little risk
[String changes made]: None

Comment 30

[Dale Harvey (:daleharvey)]

Pushed to 2.2 in https://github.com/mozilla-b2g/gaia/commit/37521e91d4356acc18fca384db9b6c7e16fce318

Green try run @ https://treeherder.mozilla.org/#/jobs?repo=gaia-try&revision=4a6c9879b64e

Comment 31

[Brogan Zumwalt [Inactive]]

Due to nature of STR we are unable to verify on our end.

Comment 32

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.