Searching for Facebook several times returns different results

RESOLVED WORKSFORME

Status

RESOLVED WORKSFORME
4 years ago
4 years ago

People

(Reporter: viorela, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Reporter)

Description

4 years ago
Test test_launch_everything_me_link.py is failing intermittently when we check that the name of the first app returned after searching for 'Facebook' ('Welcome to Facebook - Log In, Sign Up or Learn More') is in the page title of that app - we can see app's title after tapping on it ('Welcome to Facebook'): https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py#L31

Stacktrace:
Traceback (most recent call last):
  File "/home/viorelaioia/.virtualenvs/gaia_moz/lib/python2.7/site-packages/marionette_client_mozilla_b2g32_v2_0-0.1-py2.7.egg/marionette/marionette_test.py", line 170, in run
    testMethod()
  File "/home/viorelaioia/WebQA/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py", line 31, in test_launch_everything_me_link
    self.assertIn(first_app_name, self.marionette.title)
TEST-UNEXPECTED-FAIL | test_everythingme_launch_link.py test_everythingme_launch_link.TestEverythingMeLaunchLink.test_launch_everything_me_link | AssertionError: u'Welcome to Facebook - Log In, Sign Up or Learn More' not found in u'Welcome to Facebook'

Gaia      19ed3c9e78eaf234cc08484bde6998ae21552ba5
Gecko     https://hg.mozilla.org/mozilla-central/rev/a9b43778f0c2
BuildID   20140811040202
Version   34.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014
B1TC00011220

Jenkins report: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame.mozilla-central.ui.functional.smoke/62/HTML_Report/

I notice that searching for the 'Facebook' several times returns different results. I'm not sure if that's expected. If it is, then we should update our test to use a more reliable assert. If not, then we should open a corresponding bug. 
#STR would be:
1. Tap 'Facebook' in the search bar
2. Wait for results to load
Repeat steps 1 and 2 several times. 

#Expected results:
The search results are always the same

#Actual results:
The search results are different
(Reporter)

Comment 1

4 years ago
Amir, can you please give us more info about the results returned by ev.me search. Is it expected that searching for the same word several times to return different results?
Flags: needinfo?(amirn)
(Reporter)

Updated

4 years ago
Summary: Searching for Facebook returns different results → Searching for Facebook several times returns different results

Comment 2

4 years ago
Results are the same as long as the query and the location do not change.

What results are you seeing? Are you sure the result is *different* rather than does not exist?

+Dale, might be it's a Rocketbar issue.

Updated

4 years ago
Flags: needinfo?(amirn)
(Reporter)

Comment 3

4 years ago
Created attachment 8470893 [details]
result1.png
(Reporter)

Comment 4

4 years ago
Created attachment 8470894 [details]
result2.png
(Reporter)

Comment 5

4 years ago
Created attachment 8470895 [details]
result3.png
(Reporter)

Comment 6

4 years ago
I attached the different results that I get by following STR from comment 0.
This reads off as an actual bug I think - that last screenshot shows that we're producing irrelevant search results in the rocketbar search query.

Dale - What do you think?
Flags: needinfo?(dale)
Kevin mentioned Dale is on PTO, so switching needinfo to him.
Flags: needinfo?(dale) → needinfo?(kgrandon)
If you can reproduce this on a device it sounds like it could be a problem in our gecko networking code, or potentially Everything.me.

I would recommend against using real HTTP requests to test this feature though. In marionette JS tests we've just landed a mock server to return stubbed E.me results. You are going to have intermittent tests whenever you reach out to a real server, so we should try to avoid doing that.

https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/eme_server/fixtures/apps_search.json
https://github.com/mozilla-b2g/gaia/blob/master/apps/system/test/marionette/rocketbar_bookmark_web_result_test.js
Flags: needinfo?(kgrandon)
[Blocking Requested - why for this release]:

Regression that causes inconsistent search results in FB (including producing irrelevant search results in the last screenshot).

(In reply to Kevin Grandon :kgrandon from comment #9)
> If you can reproduce this on a device it sounds like it could be a problem
> in our gecko networking code, or potentially Everything.me.
> 
> I would recommend against using real HTTP requests to test this feature
> though. In marionette JS tests we've just landed a mock server to return
> stubbed E.me results. You are going to have intermittent tests whenever you
> reach out to a real server, so we should try to avoid doing that.

FWIW - I think this would go against the goal of what E2E smoketest automation is intending to solve. E2E smoketest automation is intending to find changes in behavior across the entire span of workflow of the feature, so if we pulled a piece of that out, then we wouldn't be modeling our smoketests correctly in automation.
blocking-b2g: --- → 2.1?
Component: Gaia::UI Tests → Gaia::Search
Keywords: qaurgent, regression, regressionwindow-wanted, smoketest

Updated

4 years ago
Component: Gaia::Search → Gaia::Everything.me

Updated

4 years ago
Keywords: qablocker
Whiteboard: [xfail]
(In reply to Jason Smith [:jsmith] from comment #10)
> [Blocking Requested - why for this release]:
> 
> Regression that causes inconsistent search results in FB (including
> producing irrelevant search results in the last screenshot).

Also causes one of our automated tests to go down, which makes this a qablocker.
It looks like the first four results are actually history results and not E.me results? It seems like we don't have any E.me results until the third image. Does this device have any history results present, and do you visit any links from google/facebook when this test is run?

Anyway - if you want to assert on E.me results, which I still think is a terrible idea and will always cause you to have intermittent tests, perhaps try waiting until you have > 10 results?
Flags: needinfo?(viorela.ioia)
QA Contact: jmercado
Kevin is correct, "Welcome to Facebook" is not an E.me result.

Only result3.png actually shows E.me results, and from what I can tell, the Facebook E.me result is not shown (deduped) because there is already a Facebook bookmark result.

Perhaps the deduping process is failing your test. You should consider that after bookmarking or installing a website/app, the next time you search for the same term will show different (deduped) results.

The is however not an E.me issue, the correct component is Gaia::Search
Component: Gaia::Everything.me → Gaia::Search
(Reporter)

Comment 14

4 years ago
(In reply to Kevin Grandon :kgrandon from comment #12)
> It looks like the first four results are actually history results and not
> E.me results? It seems like we don't have any E.me results until the third
> image. Does this device have any history results present, and do you visit
> any links from google/facebook when this test is run?

I ran the test again after reflashing the phone and I still got history results present, even though I didn't visit any links from google/facebook during the test. I will attach a video of 2 different results that are returned when searching for Facebook

> Anyway - if you want to assert on E.me results, which I still think is a
> terrible idea and will always cause you to have intermittent tests, perhaps
> try waiting until you have > 10 results?

Checking that we have more than 10 search results still not reliable, as the search returns sometimes only 2-3 results. Is this a bug? I mean the fact that sometimes we are seeing a reduced number of results (~2-3)?
Flags: needinfo?(viorela.ioia) → needinfo?(kgrandon)
(Reporter)

Comment 15

4 years ago
Video of the issue: http://www.fileshare.ro/e30771909
(In reply to Viorela Ioia [:viorela] from comment #14)
> Checking that we have more than 10 search results still not reliable, as the
> search returns sometimes only 2-3 results. Is this a bug? I mean the fact
> that sometimes we are seeing a reduced number of results (~2-3)?

I have not seen this. Can we verify that this bug also reproduces on a real device and not just b2g desktop?
Flags: needinfo?(kgrandon)
(Reporter)

Comment 17

4 years ago
(In reply to Kevin Grandon :kgrandon from comment #16)
> (In reply to Viorela Ioia [:viorela] from comment #14)
> > Checking that we have more than 10 search results still not reliable, as the
> > search returns sometimes only 2-3 results. Is this a bug? I mean the fact
> > that sometimes we are seeing a reduced number of results (~2-3)?
> 
> I have not seen this. Can we verify that this bug also reproduces on a real
> device and not just b2g desktop?

But I didn't use b2g desktop. I used a Flame, with the build described in comment 0.
Possibly broken by Bug 1049569

B2g-inbound Regression Window

Last working 
Environmental Variables:
Device: Flame Master
BuildID: 20140807194105
Gaia: 4c6506064270bc054f70b2923fe2b129227d479c
Gecko: c7f2e3564d3c
Version: 34.0a1 (Master) 
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

First Broken 
Environmental Variables:
Device: Flame Master
BuildID: 20140807195635
Gaia: bcad7934fb3cd2e282e87c2df1bc2ad558af41ae
Gecko: d7d7118a7ec0
Version: 34.0a1 (Master) 
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Last working gaia / First broken gecko - Issue does NOT occur 
Gaia: 4c6506064270bc054f70b2923fe2b129227d479c
Gecko: d7d7118a7ec0

First broken gaia / Last working gecko - Issue DOES occur
Gaia: bcad7934fb3cd2e282e87c2df1bc2ad558af41ae
Gecko: c7f2e3564d3c

Gaia Pushlog:  https://github.com/mozilla-b2g/gaia/compare/4c6506064270bc054f70b2923fe2b129227d479c...bcad7934fb3cd2e282e87c2df1bc2ad558af41ae
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(jmitchell)
Keywords: qaurgent, regressionwindow-wanted
Based on the regressing cause identified here, the regressing cause is part of the e.me code, so moving back to the e.me component.
Component: Gaia::Search → Gaia::Everything.me
Adding needinfo to Chris & Kevin to figure out why bug 1049569 caused this.
Blocks: 1049569
Flags: needinfo?(kgrandon)
Flags: needinfo?(chrislord.net)
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(jmitchell)
I've confirmed that you are only supposed to see 2 results for a "Facebook" search. We de-dupe the app result in favor of the marketplace result, so the expected result is that you only see a "Install Facebook" and "Google" result in search.

I'm currently unable to reproduce this on a device, and the build in the screenshots looks wrong because it does not have the proper fonts installed. I'm going to continue to try to reproduce this on a device.
Flags: needinfo?(kgrandon)
Please disregard the previously posted window - we are currently re-working it and it seems the prior post was not accurate. Sorry for the noise.
QA Whiteboard: [QAnalyst-Triage+]
Keywords: regressionwindow-wanted
QA Contact: jmercado → ckreinbring

Updated

4 years ago
No longer blocks: 1049569
Keywords: qaurgent

Updated

4 years ago
Flags: needinfo?(chrislord.net)
sorry about the time this took - this regression-window was a nightmare involving intermittent repros, variable repro-rates, different looks and effects and took multiple testers in the end. Here are the results - 

B2G-inbound
Last workingBuildID: 20140604084216
Gaia: 2a4c7becdb141d2601e47a040a27eebe52a8db79
Gecko: fd5bb34861d6
Platform Version: 32.0a1
Firmware Version: V123
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

First broken
BuildID: 20140604105916
Gaia: 18e2e8dc2d9ff19cd1210026367c14956d04eb0d
Gecko: c36c5f011229
Version: 32.0a1 (2.0)
Firmware: V123
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Working Gaia / Broken Gecko = No repro
Gaia: 2a4c7becdb141d2601e47a040a27eebe52a8db79
Gecko: c36c5f011229
Broken Gaia / Working Gecko = Repro
Gaia: 18e2e8dc2d9ff19cd1210026367c14956d04eb0d
Gecko: fd5bb34861d6
Gaia pushlog: https://github.com/mozilla-b2g/gaia/compare/2a4c7becdb141d2601e47a040a27eebe52a8db79...18e2e8dc2d9ff19cd1210026367c14956d04eb0d


This window is the "First Vertical Homescreen" window - meaning that the First V. Homescreen is the 'first broken' build.  



*submitting this FOR Chris K (so not NI'ing another lead)
QA Whiteboard: [QAnalyst-Triage+]
Keywords: qaurgent, regressionwindow-wanted
The above comment implies I think that this is a server-side regression on e.me, as we're seeing this on the earliest builds with e.me integration in the new vertical homescreen build-wise, recently starting seeing this regression on 8/11, and we aren't seeing this in in-tree automation hitting a mock server.

Ran - Can you get someone to investigate from a server-side perspective?
Flags: needinfo?(ran)
Is it possible to get a video of this with qawanted?

I am unable to reproduce this on a device, and so far the server seems like it's doing what it intends to.
Keywords: qawanted
Keywords: qaurgent
Please see this video https://www.youtube.com/watch?edit=vd&v=oS61TI2xN0g
QA Whiteboard: [QAnalyst-Triage+] → [QAnalyst-Triage?]
Flags: needinfo?(jmitchell)
Keywords: qaurgent, qawanted
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(jmitchell)
First of all, I can't reproduce. I always get the same 2 results.

BUT, I suspect that what we're seeing in the video, are stacked results from separate API responses.
Meaning, the search results for "faceboo" (about 10 results, it differs in countries) and then the 2 constant results for "facebook".
If this is the case, the search app isn't clearing or handling stale results.
Flags: needinfo?(ran)
Looking at the video - I don't think the STR that Jayme used here is modeling the bug Viorela filed. Viorela's bug deals with a clean search rocketbar for one term. It does not involve doing multiple searches within one rocketbar session, which is what the video shows.

First, can we get a separate bug on file for issue Jayme found with the inconsistent search results upon doing multiple searches while the rocketbar open?

Second, we need a clarification if the bug as filed here is still reproducing on the latest. Viorela - Are we still seeing intermittent failures with this test? If so, can we get a video of the bug you filed?
Flags: needinfo?(viorela.ioia)
Flags: needinfo?(jmitchell)
Keywords: steps-wanted

Updated

4 years ago
Keywords: steps-wanted
My understanding btw of the test (test_everythingme_launch_link.py) is that we do the following:

1. Tap the search bar
2. Type Facebook into the search bar
3. Wait for at least one result to be present
4. Confirm the search suggestion notice appears
5. Tap the first app search result
6. Confirm that the app name found in the search result matches the app name of the app that is launched
I can't reproduce this bug on a user build.
So I think I finally understand the confusion points on this bug. Let me explain this in detail.

1) The reason why the test is intermittently failing is unrelated to the bug filed here, as the test involves searching for only one term only, not doing multiple search & delete cycles. This bug as filed here is dealing with multiple searches & delete cycles with the same search term producing different search results. The bug is an issue that's been present since rocketbar was preffed on & is not a blocking issue.
2) The intermittent failure I think has to do with a problem in step [6] - that check might not be reliable if the e.me app does an immediate HTTP redirect upon launch. Robert is going to open a separate bug for this.

I'm leaving needinfo on Viorela to ensure that we get bug on file for [2] & on Josh to ensure that we rewrite the STR, expected results, actual results, etc of the bug that the window was done against.
blocking-b2g: 2.1? → ---
Component: Gaia::Everything.me → Gaia::Search
Keywords: qablocker, regression, smoketest
Whiteboard: [xfail]
just thought I should mention: this issue is no longer reproducing in the latest 2.1 build - 

Using both the steps as described by Jason for the bug written as comment 0 (1 search per delete cycle)
and the steps used for the regression window
STR
1) Connect to Wifi
2) Select E.me search bar
3) Type in 'Facebook'
4) Erase several of the letters and re-type them

Actual Results - only 1 result is ever seen - the "install Facebook" app.

Device: Flame Master
Build ID: 20140814005107
Gaia: 5e074831f9ddacdf6f622a6dffaecb626f740be8
Gecko: 5299864050ee
Version: 34.0a1 (Master)
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0
Flags: needinfo?(jmitchell)
In that case, I'm going to close this out.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Flags: needinfo?(viorela.ioia)
Resolution: --- → WORKSFORME
(Reporter)

Comment 34

4 years ago
I followed STR in comment 32 and I still got different search results.
If the search would return only 1 result - the "install Facebook" app - then our test would have failed. That's because in the test we tap on the first result that is a link, and not an app: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/apps/homescreen/regions/search_panel.py#L53

Gaia      b33b4d9558e0b9eabbfda7be23435e2b38fd40bf
Gecko     https://hg.mozilla.org/mozilla-central/rev/111a1da2a95d
BuildID   20140819040202
Version   34.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014
B1TC00011220
You need to log in before you can comment on or make changes to this bug.