1051888 - Searching for Facebook several times returns different results

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::Search, defect)

Description

[Viorela Ioia [:viorela]]

Test test_launch_everything_me_link.py is failing intermittently when we check that the name of the first app returned after searching for 'Facebook' ('Welcome to Facebook - Log In, Sign Up or Learn More') is in the page title of that app - we can see app's title after tapping on it ('Welcome to Facebook'): https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py#L31

Stacktrace:
Traceback (most recent call last):
  File "/home/viorelaioia/.virtualenvs/gaia_moz/lib/python2.7/site-packages/marionette_client_mozilla_b2g32_v2_0-0.1-py2.7.egg/marionette/marionette_test.py", line 170, in run
    testMethod()
  File "/home/viorelaioia/WebQA/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py", line 31, in test_launch_everything_me_link
    self.assertIn(first_app_name, self.marionette.title)
TEST-UNEXPECTED-FAIL | test_everythingme_launch_link.py test_everythingme_launch_link.TestEverythingMeLaunchLink.test_launch_everything_me_link | AssertionError: u'Welcome to Facebook - Log In, Sign Up or Learn More' not found in u'Welcome to Facebook'

Gaia      19ed3c9e78eaf234cc08484bde6998ae21552ba5
Gecko     https://hg.mozilla.org/mozilla-central/rev/a9b43778f0c2
BuildID   20140811040202
Version   34.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014
B1TC00011220

Jenkins report: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame.mozilla-central.ui.functional.smoke/62/HTML_Report/

I notice that searching for the 'Facebook' several times returns different results. I'm not sure if that's expected. If it is, then we should update our test to use a more reliable assert. If not, then we should open a corresponding bug. 
#STR would be:
1. Tap 'Facebook' in the search bar
2. Wait for results to load
Repeat steps 1 and 2 several times. 

#Expected results:
The search results are always the same

#Actual results:
The search results are different

Comment 1

[Viorela Ioia [:viorela]]

Amir, can you please give us more info about the results returned by ev.me search. Is it expected that searching for the same word several times to return different results?

Comment 2

[Amir Nissim (:amirn)]

Results are the same as long as the query and the location do not change.

What results are you seeing? Are you sure the result is *different* rather than does not exist?

+Dale, might be it's a Rocketbar issue.

Comment 3

[Viorela Ioia [:viorela]]

Attachment: result1.png

Comment 4

[Viorela Ioia [:viorela]]

Attachment: result2.png

Comment 5

[Viorela Ioia [:viorela]]

Attachment: result3.png

Comment 6

[Viorela Ioia [:viorela]]

I attached the different results that I get by following STR from comment 0.

Comment 7

[Jason Smith [:jsmith]]

This reads off as an actual bug I think - that last screenshot shows that we're producing irrelevant search results in the rocketbar search query.

Dale - What do you think?

Comment 8

[Jason Smith [:jsmith]]

Kevin mentioned Dale is on PTO, so switching needinfo to him.

Comment 9

[Kevin Grandon :kgrandon]

If you can reproduce this on a device it sounds like it could be a problem in our gecko networking code, or potentially Everything.me.

I would recommend against using real HTTP requests to test this feature though. In marionette JS tests we've just landed a mock server to return stubbed E.me results. You are going to have intermittent tests whenever you reach out to a real server, so we should try to avoid doing that.

https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/eme_server/fixtures/apps_search.json
https://github.com/mozilla-b2g/gaia/blob/master/apps/system/test/marionette/rocketbar_bookmark_web_result_test.js

Comment 10

[Jason Smith [:jsmith]]

[Blocking Requested - why for this release]:

Regression that causes inconsistent search results in FB (including producing irrelevant search results in the last screenshot).

(In reply to Kevin Grandon :kgrandon from comment #9)
> If you can reproduce this on a device it sounds like it could be a problem
> in our gecko networking code, or potentially Everything.me.
> 
> I would recommend against using real HTTP requests to test this feature
> though. In marionette JS tests we've just landed a mock server to return
> stubbed E.me results. You are going to have intermittent tests whenever you
> reach out to a real server, so we should try to avoid doing that.

FWIW - I think this would go against the goal of what E2E smoketest automation is intending to solve. E2E smoketest automation is intending to find changes in behavior across the entire span of workflow of the feature, so if we pulled a piece of that out, then we wouldn't be modeling our smoketests correctly in automation.

Comment 11

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #10)
> [Blocking Requested - why for this release]:
> 
> Regression that causes inconsistent search results in FB (including
> producing irrelevant search results in the last screenshot).

Also causes one of our automated tests to go down, which makes this a qablocker.

Comment 12

[Kevin Grandon :kgrandon]

It looks like the first four results are actually history results and not E.me results? It seems like we don't have any E.me results until the third image. Does this device have any history results present, and do you visit any links from google/facebook when this test is run?

Anyway - if you want to assert on E.me results, which I still think is a terrible idea and will always cause you to have intermittent tests, perhaps try waiting until you have > 10 results?

Comment 13

[Amir Nissim (:amirn)]

Kevin is correct, "Welcome to Facebook" is not an E.me result.

Only result3.png actually shows E.me results, and from what I can tell, the Facebook E.me result is not shown (deduped) because there is already a Facebook bookmark result.

Perhaps the deduping process is failing your test. You should consider that after bookmarking or installing a website/app, the next time you search for the same term will show different (deduped) results.

The is however not an E.me issue, the correct component is Gaia::Search

Comment 14

[Viorela Ioia [:viorela]]

(In reply to Kevin Grandon :kgrandon from comment #12)
> It looks like the first four results are actually history results and not
> E.me results? It seems like we don't have any E.me results until the third
> image. Does this device have any history results present, and do you visit
> any links from google/facebook when this test is run?

I ran the test again after reflashing the phone and I still got history results present, even though I didn't visit any links from google/facebook during the test. I will attach a video of 2 different results that are returned when searching for Facebook

> Anyway - if you want to assert on E.me results, which I still think is a
> terrible idea and will always cause you to have intermittent tests, perhaps
> try waiting until you have > 10 results?

Checking that we have more than 10 search results still not reliable, as the search returns sometimes only 2-3 results. Is this a bug? I mean the fact that sometimes we are seeing a reduced number of results (~2-3)?

Comment 15

[Viorela Ioia [:viorela]]

Video of the issue: http://www.fileshare.ro/e30771909

Comment 16

[Kevin Grandon :kgrandon]

(In reply to Viorela Ioia [:viorela] from comment #14)
> Checking that we have more than 10 search results still not reliable, as the
> search returns sometimes only 2-3 results. Is this a bug? I mean the fact
> that sometimes we are seeing a reduced number of results (~2-3)?

I have not seen this. Can we verify that this bug also reproduces on a real device and not just b2g desktop?

Comment 17

[Viorela Ioia [:viorela]]

(In reply to Kevin Grandon :kgrandon from comment #16)
> (In reply to Viorela Ioia [:viorela] from comment #14)
> > Checking that we have more than 10 search results still not reliable, as the
> > search returns sometimes only 2-3 results. Is this a bug? I mean the fact
> > that sometimes we are seeing a reduced number of results (~2-3)?
> 
> I have not seen this. Can we verify that this bug also reproduces on a real
> device and not just b2g desktop?

But I didn't use b2g desktop. I used a Flame, with the build described in comment 0.

Comment 18

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Possibly broken by Bug 1049569

B2g-inbound Regression Window

Last working 
Environmental Variables:
Device: Flame Master
BuildID: 20140807194105
Gaia: 4c6506064270bc054f70b2923fe2b129227d479c
Gecko: c7f2e3564d3c
Version: 34.0a1 (Master) 
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

First Broken 
Environmental Variables:
Device: Flame Master
BuildID: 20140807195635
Gaia: bcad7934fb3cd2e282e87c2df1bc2ad558af41ae
Gecko: d7d7118a7ec0
Version: 34.0a1 (Master) 
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Last working gaia / First broken gecko - Issue does NOT occur 
Gaia: 4c6506064270bc054f70b2923fe2b129227d479c
Gecko: d7d7118a7ec0

First broken gaia / Last working gecko - Issue DOES occur
Gaia: bcad7934fb3cd2e282e87c2df1bc2ad558af41ae
Gecko: c7f2e3564d3c

Gaia Pushlog:  https://github.com/mozilla-b2g/gaia/compare/4c6506064270bc054f70b2923fe2b129227d479c...bcad7934fb3cd2e282e87c2df1bc2ad558af41ae

Comment 19

[Jason Smith [:jsmith]]

Based on the regressing cause identified here, the regressing cause is part of the e.me code, so moving back to the e.me component.

Comment 20

[Jason Smith [:jsmith]]

Adding needinfo to Chris & Kevin to figure out why bug 1049569 caused this.

Comment 21

[Kevin Grandon :kgrandon]

I've confirmed that you are only supposed to see 2 results for a "Facebook" search. We de-dupe the app result in favor of the marketplace result, so the expected result is that you only see a "Install Facebook" and "Google" result in search.

I'm currently unable to reproduce this on a device, and the build in the screenshots looks wrong because it does not have the proper fonts installed. I'm going to continue to try to reproduce this on a device.

Comment 22

[Joshua Mitchell (Inactive)]

Please disregard the previously posted window - we are currently re-working it and it seems the prior post was not accurate. Sorry for the noise.

Comment 23

[Joshua Mitchell (Inactive)]

sorry about the time this took - this regression-window was a nightmare involving intermittent repros, variable repro-rates, different looks and effects and took multiple testers in the end. Here are the results - 

B2G-inbound
Last workingBuildID: 20140604084216
Gaia: 2a4c7becdb141d2601e47a040a27eebe52a8db79
Gecko: fd5bb34861d6
Platform Version: 32.0a1
Firmware Version: V123
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

First broken
BuildID: 20140604105916
Gaia: 18e2e8dc2d9ff19cd1210026367c14956d04eb0d
Gecko: c36c5f011229
Version: 32.0a1 (2.0)
Firmware: V123
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Working Gaia / Broken Gecko = No repro
Gaia: 2a4c7becdb141d2601e47a040a27eebe52a8db79
Gecko: c36c5f011229
Broken Gaia / Working Gecko = Repro
Gaia: 18e2e8dc2d9ff19cd1210026367c14956d04eb0d
Gecko: fd5bb34861d6
Gaia pushlog: https://github.com/mozilla-b2g/gaia/compare/2a4c7becdb141d2601e47a040a27eebe52a8db79...18e2e8dc2d9ff19cd1210026367c14956d04eb0d


This window is the "First Vertical Homescreen" window - meaning that the First V. Homescreen is the 'first broken' build.  



*submitting this FOR Chris K (so not NI'ing another lead)

Comment 24

[Jason Smith [:jsmith]]

The above comment implies I think that this is a server-side regression on e.me, as we're seeing this on the earliest builds with e.me integration in the new vertical homescreen build-wise, recently starting seeing this regression on 8/11, and we aren't seeing this in in-tree automation hitting a mock server.

Ran - Can you get someone to investigate from a server-side perspective?

Comment 25

[Kevin Grandon :kgrandon]

Is it possible to get a video of this with qawanted?

I am unable to reproduce this on a device, and so far the server seems like it's doing what it intends to.

Comment 26

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Please see this video https://www.youtube.com/watch?edit=vd&v=oS61TI2xN0g

Comment 27

[Ran Ben Aharon [:ranbena] EverythingMe]

First of all, I can't reproduce. I always get the same 2 results.

BUT, I suspect that what we're seeing in the video, are stacked results from separate API responses.
Meaning, the search results for "faceboo" (about 10 results, it differs in countries) and then the 2 constant results for "facebook".
If this is the case, the search app isn't clearing or handling stale results.

Comment 28

[Jason Smith [:jsmith]]

Looking at the video - I don't think the STR that Jayme used here is modeling the bug Viorela filed. Viorela's bug deals with a clean search rocketbar for one term. It does not involve doing multiple searches within one rocketbar session, which is what the video shows.

First, can we get a separate bug on file for issue Jayme found with the inconsistent search results upon doing multiple searches while the rocketbar open?

Second, we need a clarification if the bug as filed here is still reproducing on the latest. Viorela - Are we still seeing intermittent failures with this test? If so, can we get a video of the bug you filed?

Comment 29

[Jason Smith [:jsmith]]

My understanding btw of the test (test_everythingme_launch_link.py) is that we do the following:

1. Tap the search bar
2. Type Facebook into the search bar
3. Wait for at least one result to be present
4. Confirm the search suggestion notice appears
5. Tap the first app search result
6. Confirm that the app name found in the search result matches the app name of the app that is launched

Comment 30

[Jason Smith [:jsmith]]

I can't reproduce this bug on a user build.

Comment 31

[Jason Smith [:jsmith]]

So I think I finally understand the confusion points on this bug. Let me explain this in detail.

1) The reason why the test is intermittently failing is unrelated to the bug filed here, as the test involves searching for only one term only, not doing multiple search & delete cycles. This bug as filed here is dealing with multiple searches & delete cycles with the same search term producing different search results. The bug is an issue that's been present since rocketbar was preffed on & is not a blocking issue.
2) The intermittent failure I think has to do with a problem in step [6] - that check might not be reliable if the e.me app does an immediate HTTP redirect upon launch. Robert is going to open a separate bug for this.

I'm leaving needinfo on Viorela to ensure that we get bug on file for [2] & on Josh to ensure that we rewrite the STR, expected results, actual results, etc of the bug that the window was done against.

Comment 32

[Joshua Mitchell (Inactive)]

just thought I should mention: this issue is no longer reproducing in the latest 2.1 build - 

Using both the steps as described by Jason for the bug written as comment 0 (1 search per delete cycle)
and the steps used for the regression window
STR
1) Connect to Wifi
2) Select E.me search bar
3) Type in 'Facebook'
4) Erase several of the letters and re-type them

Actual Results - only 1 result is ever seen - the "install Facebook" app.

Device: Flame Master
Build ID: 20140814005107
Gaia: 5e074831f9ddacdf6f622a6dffaecb626f740be8
Gecko: 5299864050ee
Version: 34.0a1 (Master)
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Comment 33

[Jason Smith [:jsmith]]

In that case, I'm going to close this out.

Comment 34

[Viorela Ioia [:viorela]]

I followed STR in comment 32 and I still got different search results.
If the search would return only 1 result - the "install Facebook" app - then our test would have failed. That's because in the test we tap on the first result that is a link, and not an app: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/apps/homescreen/regions/search_panel.py#L53

Gaia      b33b4d9558e0b9eabbfda7be23435e2b38fd40bf
Gecko     https://hg.mozilla.org/mozilla-central/rev/111a1da2a95d
BuildID   20140819040202
Version   34.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014
B1TC00011220