807251 - Test failure "Suggestions are different - got 'false'" in testSearch/testSearchSuggestions.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P3)

Description

[Andreea Matei [:AndreeaMatei]]

Happens on Windows for now, with Aurora build.
This fails at the last check from the test.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Looks like this might have been a one-off. It occurred a few times in July/August on ESR. We could try to replicate it under heavy load, or add some debug statements for the next time it fails.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Didn't happen again.

Comment 3

[Andreea Matei [:AndreeaMatei]]

It happened last night:
http://mozmill-ci.blargon7.com/#/functional/report/9e41582ed5e806fa373351d9d32035f1

It's extremely rare, we weren't able to reproduce it. The cause must be that the suggestions are not being populated and Daniela found bug 542990 where this is a known issue, just it didn't got fixed.

Comment 4

[Daniela Petrovici]

Happened on Linux x86 with Nightly FR:
http://mozmill-ci.blargon7.com/#/functional/report/72e15dc943833b8fcba70aeb51e1c186
And Aurora FR:
http://mozmill-ci.blargon7.com/#/functional/report/72e15dc943833b8fcba70aeb51e6cb0a

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We should improve the checks so we know what we are comparing too. Lets take it for next week.

Comment 6

[Daniela Petrovici]

Attachment: patch v1.0 (improve checks)

This is only to improve the check for suggestions per previous comment

Comment 7

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 710159 [details] [diff] [review]
patch v1.0 (improve checks)

Review of attachment 710159 [details] [diff] [review]:
-----------------------------------------------------------------

This will do for debug, but I still feel is not enough.
We should also see the exact suggestion/s that caused the test to fail, maybe is always the same.
Instead of using dumps, could we make use of throws recently added?

Comment 8

[Daniela Petrovici]

Attachment: patch v1.1 (improve checks)

Changed the error message to show the suggestions for one search provider. In case assert.ok fails, then all suggestions for the two search providers are the same.

Comment 9

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 710704 [details] [diff] [review]
patch v1.1 (improve checks)

Review of attachment 710704 [details] [diff] [review]:
-----------------------------------------------------------------

Great, this allows us to get the suggestions as well as the providers next time we're failing.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 710704 [details] [diff] [review]
patch v1.1 (improve checks)

Review of attachment 710704 [details] [diff] [review]:
-----------------------------------------------------------------

http://hg.mozilla.org/qa/mozmill-tests/rev/945cbb978d10 (default)

Comment 11

[Daniela Petrovici]

This patch applies clearly on all branches

Comment 12

[Andreea Matei [:AndreeaMatei]]

And it's on all branches:
http://hg.mozilla.org/qa/mozmill-tests/rev/07c3b145def7 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/e9cbcfdbfc0d (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/7227a2a7f526 (release)
http://hg.mozilla.org/qa/mozmill-tests/rev/0448b139a767 (esr17)

Now lets hope it will fail soon on one of these branches so we can fix this.

Comment 13

[Daniela Petrovici]

The issue did not reproduce in a month (since 01/30).

Comment 14

[Andreea Matei [:AndreeaMatei]]

Lets reopen if needed.

Comment 15

[u279076]

A few new instances on some locales with Firefox 23.0.1:
> http://mozmill-release.blargon7.com/#/functional/failure?branch=All&platform=All&from=2013-08-14&to=2013-08-15&test=%2FtestSearch%2FtestSearchSuggestions.js&func=testSearchSuggestions.js%3A%3AtestMultipleEngines

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Anthony, this is an old fixed bug. Please create a new one with the specific locales in question in the summary. That way we can differentiate it. Thanks.