Closed
Bug 1051888
Opened 10 years ago
Closed 10 years ago
Searching for Facebook several times returns different results
Categories
(Firefox OS Graveyard :: Gaia::Search, defect)
Tracking
(Not tracked)
RESOLVED
WORKSFORME
People
(Reporter: viorela, Unassigned)
Details
Attachments
(3 files)
Test test_launch_everything_me_link.py is failing intermittently when we check that the name of the first app returned after searching for 'Facebook' ('Welcome to Facebook - Log In, Sign Up or Learn More') is in the page title of that app - we can see app's title after tapping on it ('Welcome to Facebook'): https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py#L31 Stacktrace: Traceback (most recent call last): File "/home/viorelaioia/.virtualenvs/gaia_moz/lib/python2.7/site-packages/marionette_client_mozilla_b2g32_v2_0-0.1-py2.7.egg/marionette/marionette_test.py", line 170, in run testMethod() File "/home/viorelaioia/WebQA/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py", line 31, in test_launch_everything_me_link self.assertIn(first_app_name, self.marionette.title) TEST-UNEXPECTED-FAIL | test_everythingme_launch_link.py test_everythingme_launch_link.TestEverythingMeLaunchLink.test_launch_everything_me_link | AssertionError: u'Welcome to Facebook - Log In, Sign Up or Learn More' not found in u'Welcome to Facebook' Gaia 19ed3c9e78eaf234cc08484bde6998ae21552ba5 Gecko https://hg.mozilla.org/mozilla-central/rev/a9b43778f0c2 BuildID 20140811040202 Version 34.0a1 ro.build.version.incremental=109 ro.build.date=Mon Jun 16 16:51:29 CST 2014 B1TC00011220 Jenkins report: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame.mozilla-central.ui.functional.smoke/62/HTML_Report/ I notice that searching for the 'Facebook' several times returns different results. I'm not sure if that's expected. If it is, then we should update our test to use a more reliable assert. If not, then we should open a corresponding bug. #STR would be: 1. Tap 'Facebook' in the search bar 2. Wait for results to load Repeat steps 1 and 2 several times. #Expected results: The search results are always the same #Actual results: The search results are different
Reporter | ||
Comment 1•10 years ago
|
||
Amir, can you please give us more info about the results returned by ev.me search. Is it expected that searching for the same word several times to return different results?
Flags: needinfo?(amirn)
Reporter | ||
Updated•10 years ago
|
Summary: Searching for Facebook returns different results → Searching for Facebook several times returns different results
Comment 2•10 years ago
|
||
Results are the same as long as the query and the location do not change. What results are you seeing? Are you sure the result is *different* rather than does not exist? +Dale, might be it's a Rocketbar issue.
Updated•10 years ago
|
Flags: needinfo?(amirn)
Reporter | ||
Comment 3•10 years ago
|
||
Reporter | ||
Comment 4•10 years ago
|
||
Reporter | ||
Comment 5•10 years ago
|
||
Reporter | ||
Comment 6•10 years ago
|
||
I attached the different results that I get by following STR from comment 0.
Comment 7•10 years ago
|
||
This reads off as an actual bug I think - that last screenshot shows that we're producing irrelevant search results in the rocketbar search query. Dale - What do you think?
Flags: needinfo?(dale)
Comment 8•10 years ago
|
||
Kevin mentioned Dale is on PTO, so switching needinfo to him.
Flags: needinfo?(dale) → needinfo?(kgrandon)
Comment 9•10 years ago
|
||
If you can reproduce this on a device it sounds like it could be a problem in our gecko networking code, or potentially Everything.me. I would recommend against using real HTTP requests to test this feature though. In marionette JS tests we've just landed a mock server to return stubbed E.me results. You are going to have intermittent tests whenever you reach out to a real server, so we should try to avoid doing that. https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/eme_server/fixtures/apps_search.json https://github.com/mozilla-b2g/gaia/blob/master/apps/system/test/marionette/rocketbar_bookmark_web_result_test.js
Flags: needinfo?(kgrandon)
Comment 10•10 years ago
|
||
[Blocking Requested - why for this release]: Regression that causes inconsistent search results in FB (including producing irrelevant search results in the last screenshot). (In reply to Kevin Grandon :kgrandon from comment #9) > If you can reproduce this on a device it sounds like it could be a problem > in our gecko networking code, or potentially Everything.me. > > I would recommend against using real HTTP requests to test this feature > though. In marionette JS tests we've just landed a mock server to return > stubbed E.me results. You are going to have intermittent tests whenever you > reach out to a real server, so we should try to avoid doing that. FWIW - I think this would go against the goal of what E2E smoketest automation is intending to solve. E2E smoketest automation is intending to find changes in behavior across the entire span of workflow of the feature, so if we pulled a piece of that out, then we wouldn't be modeling our smoketests correctly in automation.
blocking-b2g: --- → 2.1?
Component: Gaia::UI Tests → Gaia::Search
Updated•10 years ago
|
Component: Gaia::Search → Gaia::Everything.me
Comment 11•10 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #10) > [Blocking Requested - why for this release]: > > Regression that causes inconsistent search results in FB (including > producing irrelevant search results in the last screenshot). Also causes one of our automated tests to go down, which makes this a qablocker.
Comment 12•10 years ago
|
||
It looks like the first four results are actually history results and not E.me results? It seems like we don't have any E.me results until the third image. Does this device have any history results present, and do you visit any links from google/facebook when this test is run? Anyway - if you want to assert on E.me results, which I still think is a terrible idea and will always cause you to have intermittent tests, perhaps try waiting until you have > 10 results?
Flags: needinfo?(viorela.ioia)
Updated•10 years ago
|
QA Contact: jmercado
Comment 13•10 years ago
|
||
Kevin is correct, "Welcome to Facebook" is not an E.me result. Only result3.png actually shows E.me results, and from what I can tell, the Facebook E.me result is not shown (deduped) because there is already a Facebook bookmark result. Perhaps the deduping process is failing your test. You should consider that after bookmarking or installing a website/app, the next time you search for the same term will show different (deduped) results. The is however not an E.me issue, the correct component is Gaia::Search
Component: Gaia::Everything.me → Gaia::Search
Reporter | ||
Comment 14•10 years ago
|
||
(In reply to Kevin Grandon :kgrandon from comment #12) > It looks like the first four results are actually history results and not > E.me results? It seems like we don't have any E.me results until the third > image. Does this device have any history results present, and do you visit > any links from google/facebook when this test is run? I ran the test again after reflashing the phone and I still got history results present, even though I didn't visit any links from google/facebook during the test. I will attach a video of 2 different results that are returned when searching for Facebook > Anyway - if you want to assert on E.me results, which I still think is a > terrible idea and will always cause you to have intermittent tests, perhaps > try waiting until you have > 10 results? Checking that we have more than 10 search results still not reliable, as the search returns sometimes only 2-3 results. Is this a bug? I mean the fact that sometimes we are seeing a reduced number of results (~2-3)?
Flags: needinfo?(viorela.ioia) → needinfo?(kgrandon)
Reporter | ||
Comment 15•10 years ago
|
||
Video of the issue: http://www.fileshare.ro/e30771909
Comment 16•10 years ago
|
||
(In reply to Viorela Ioia [:viorela] from comment #14) > Checking that we have more than 10 search results still not reliable, as the > search returns sometimes only 2-3 results. Is this a bug? I mean the fact > that sometimes we are seeing a reduced number of results (~2-3)? I have not seen this. Can we verify that this bug also reproduces on a real device and not just b2g desktop?
Flags: needinfo?(kgrandon)
Reporter | ||
Comment 17•10 years ago
|
||
(In reply to Kevin Grandon :kgrandon from comment #16) > (In reply to Viorela Ioia [:viorela] from comment #14) > > Checking that we have more than 10 search results still not reliable, as the > > search returns sometimes only 2-3 results. Is this a bug? I mean the fact > > that sometimes we are seeing a reduced number of results (~2-3)? > > I have not seen this. Can we verify that this bug also reproduces on a real > device and not just b2g desktop? But I didn't use b2g desktop. I used a Flame, with the build described in comment 0.
Comment 18•10 years ago
|
||
Possibly broken by Bug 1049569 B2g-inbound Regression Window Last working Environmental Variables: Device: Flame Master BuildID: 20140807194105 Gaia: 4c6506064270bc054f70b2923fe2b129227d479c Gecko: c7f2e3564d3c Version: 34.0a1 (Master) Firmware Version: v123 User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0 First Broken Environmental Variables: Device: Flame Master BuildID: 20140807195635 Gaia: bcad7934fb3cd2e282e87c2df1bc2ad558af41ae Gecko: d7d7118a7ec0 Version: 34.0a1 (Master) Firmware Version: v123 User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0 Last working gaia / First broken gecko - Issue does NOT occur Gaia: 4c6506064270bc054f70b2923fe2b129227d479c Gecko: d7d7118a7ec0 First broken gaia / Last working gecko - Issue DOES occur Gaia: bcad7934fb3cd2e282e87c2df1bc2ad558af41ae Gecko: c7f2e3564d3c Gaia Pushlog: https://github.com/mozilla-b2g/gaia/compare/4c6506064270bc054f70b2923fe2b129227d479c...bcad7934fb3cd2e282e87c2df1bc2ad558af41ae
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(jmitchell)
Keywords: qaurgent,
regressionwindow-wanted
Comment 19•10 years ago
|
||
Based on the regressing cause identified here, the regressing cause is part of the e.me code, so moving back to the e.me component.
Component: Gaia::Search → Gaia::Everything.me
Comment 20•10 years ago
|
||
Adding needinfo to Chris & Kevin to figure out why bug 1049569 caused this.
Updated•10 years ago
|
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(jmitchell)
Comment 21•10 years ago
|
||
I've confirmed that you are only supposed to see 2 results for a "Facebook" search. We de-dupe the app result in favor of the marketplace result, so the expected result is that you only see a "Install Facebook" and "Google" result in search. I'm currently unable to reproduce this on a device, and the build in the screenshots looks wrong because it does not have the proper fonts installed. I'm going to continue to try to reproduce this on a device.
Flags: needinfo?(kgrandon)
Comment 22•10 years ago
|
||
Please disregard the previously posted window - we are currently re-working it and it seems the prior post was not accurate. Sorry for the noise.
QA Whiteboard: [QAnalyst-Triage+]
Keywords: regressionwindow-wanted
QA Contact: jmercado → ckreinbring
Updated•10 years ago
|
Updated•10 years ago
|
Flags: needinfo?(chrislord.net)
Comment 23•10 years ago
|
||
sorry about the time this took - this regression-window was a nightmare involving intermittent repros, variable repro-rates, different looks and effects and took multiple testers in the end. Here are the results - B2G-inbound Last workingBuildID: 20140604084216 Gaia: 2a4c7becdb141d2601e47a040a27eebe52a8db79 Gecko: fd5bb34861d6 Platform Version: 32.0a1 Firmware Version: V123 User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0 First broken BuildID: 20140604105916 Gaia: 18e2e8dc2d9ff19cd1210026367c14956d04eb0d Gecko: c36c5f011229 Version: 32.0a1 (2.0) Firmware: V123 User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0 Working Gaia / Broken Gecko = No repro Gaia: 2a4c7becdb141d2601e47a040a27eebe52a8db79 Gecko: c36c5f011229 Broken Gaia / Working Gecko = Repro Gaia: 18e2e8dc2d9ff19cd1210026367c14956d04eb0d Gecko: fd5bb34861d6 Gaia pushlog: https://github.com/mozilla-b2g/gaia/compare/2a4c7becdb141d2601e47a040a27eebe52a8db79...18e2e8dc2d9ff19cd1210026367c14956d04eb0d This window is the "First Vertical Homescreen" window - meaning that the First V. Homescreen is the 'first broken' build. *submitting this FOR Chris K (so not NI'ing another lead)
QA Whiteboard: [QAnalyst-Triage+]
Keywords: qaurgent,
regressionwindow-wanted
Comment 24•10 years ago
|
||
The above comment implies I think that this is a server-side regression on e.me, as we're seeing this on the earliest builds with e.me integration in the new vertical homescreen build-wise, recently starting seeing this regression on 8/11, and we aren't seeing this in in-tree automation hitting a mock server. Ran - Can you get someone to investigate from a server-side perspective?
Flags: needinfo?(ran)
Comment 25•10 years ago
|
||
Is it possible to get a video of this with qawanted? I am unable to reproduce this on a device, and so far the server seems like it's doing what it intends to.
Keywords: qawanted
Comment 26•10 years ago
|
||
Please see this video https://www.youtube.com/watch?edit=vd&v=oS61TI2xN0g
Updated•10 years ago
|
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(jmitchell)
Comment 27•10 years ago
|
||
First of all, I can't reproduce. I always get the same 2 results. BUT, I suspect that what we're seeing in the video, are stacked results from separate API responses. Meaning, the search results for "faceboo" (about 10 results, it differs in countries) and then the 2 constant results for "facebook". If this is the case, the search app isn't clearing or handling stale results.
Flags: needinfo?(ran)
Comment 28•10 years ago
|
||
Looking at the video - I don't think the STR that Jayme used here is modeling the bug Viorela filed. Viorela's bug deals with a clean search rocketbar for one term. It does not involve doing multiple searches within one rocketbar session, which is what the video shows. First, can we get a separate bug on file for issue Jayme found with the inconsistent search results upon doing multiple searches while the rocketbar open? Second, we need a clarification if the bug as filed here is still reproducing on the latest. Viorela - Are we still seeing intermittent failures with this test? If so, can we get a video of the bug you filed?
Updated•10 years ago
|
Keywords: steps-wanted
Comment 29•10 years ago
|
||
My understanding btw of the test (test_everythingme_launch_link.py) is that we do the following: 1. Tap the search bar 2. Type Facebook into the search bar 3. Wait for at least one result to be present 4. Confirm the search suggestion notice appears 5. Tap the first app search result 6. Confirm that the app name found in the search result matches the app name of the app that is launched
Comment 30•10 years ago
|
||
I can't reproduce this bug on a user build.
Comment 31•10 years ago
|
||
So I think I finally understand the confusion points on this bug. Let me explain this in detail. 1) The reason why the test is intermittently failing is unrelated to the bug filed here, as the test involves searching for only one term only, not doing multiple search & delete cycles. This bug as filed here is dealing with multiple searches & delete cycles with the same search term producing different search results. The bug is an issue that's been present since rocketbar was preffed on & is not a blocking issue. 2) The intermittent failure I think has to do with a problem in step [6] - that check might not be reliable if the e.me app does an immediate HTTP redirect upon launch. Robert is going to open a separate bug for this. I'm leaving needinfo on Viorela to ensure that we get bug on file for [2] & on Josh to ensure that we rewrite the STR, expected results, actual results, etc of the bug that the window was done against.
blocking-b2g: 2.1? → ---
Component: Gaia::Everything.me → Gaia::Search
Keywords: qablocker,
regression,
smoketest
Whiteboard: [xfail]
Comment 32•10 years ago
|
||
just thought I should mention: this issue is no longer reproducing in the latest 2.1 build - Using both the steps as described by Jason for the bug written as comment 0 (1 search per delete cycle) and the steps used for the regression window STR 1) Connect to Wifi 2) Select E.me search bar 3) Type in 'Facebook' 4) Erase several of the letters and re-type them Actual Results - only 1 result is ever seen - the "install Facebook" app. Device: Flame Master Build ID: 20140814005107 Gaia: 5e074831f9ddacdf6f622a6dffaecb626f740be8 Gecko: 5299864050ee Version: 34.0a1 (Master) Firmware Version: v123 User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0
Flags: needinfo?(jmitchell)
Comment 33•10 years ago
|
||
In that case, I'm going to close this out.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(viorela.ioia)
Resolution: --- → WORKSFORME
Reporter | ||
Comment 34•10 years ago
|
||
I followed STR in comment 32 and I still got different search results. If the search would return only 1 result - the "install Facebook" app - then our test would have failed. That's because in the test we tap on the first result that is a link, and not an app: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/apps/homescreen/regions/search_panel.py#L53 Gaia b33b4d9558e0b9eabbfda7be23435e2b38fd40bf Gecko https://hg.mozilla.org/mozilla-central/rev/111a1da2a95d BuildID 20140819040202 Version 34.0a1 ro.build.version.incremental=109 ro.build.date=Mon Jun 16 16:51:29 CST 2014 B1TC00011220
You need to log in
before you can comment on or make changes to this bug.
Description
•