Closed Bug 1082787 Opened 5 years ago Closed 5 years ago

Search bar should not send URL to Everything.me / marketplace.

Categories

(Firefox OS Graveyard :: Gaia::Search, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

(blocking-b2g:-, feature-b2g:2.2+, tracking-b2g:+, b2g-v1.4 ?, b2g-v2.0 affected, b2g-v2.1 affected, b2g-v2.2 verified, b2g-master verified)

VERIFIED FIXED
2.2 S2 (19dec)
blocking-b2g -
feature-b2g 2.2+
tracking-b2g +
Tracking Status
b2g-v1.4 --- ?
b2g-v2.0 --- affected
b2g-v2.1 --- affected
b2g-v2.2 --- verified
b2g-master --- verified

People

(Reporter: nbp, Assigned: kgrandon)

Details

(Keywords: privacy, Whiteboard: [systemsfe])

Attachments

(2 files)

[Tracking Requested - why for this release]:

While running a marionette test which use "search.go_to_url(url)", I noticed in the logcat that the address being typed is fully transmitted to the Marketplace and to Everything.me.

W/Browser ( 1192): [JavaScript Warning: "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://marketplace.firefox.com/api/v2/apps/search/rocketbar/?q=http://people.mozilla.com/~npierron/sunspider/hoste&limit=4&lang=en-US&region=restofworld. This can be fixed by moving the resource to the same domain or enabling CORS." {file: "https://marketplace.firefox.com/api/v2/apps/search/rocketbar/?q=http://people.mozilla.com/~npierron/sunspider/hoste&limit=4&lang=en-US&region=restofworld" line: 0}]
I/Browser ( 1192): Content JS LOG: evme init: noop
I/Browser ( 1192):     at log (app://search.gaiamobile.org/shared/js/everythingme/eme.js:73:8)
I/Browser ( 1192): Content JS LOG: evme init: noop
I/Browser ( 1192):     at log (app://search.gaiamobile.org/shared/js/everythingme/eme.js:73:8)
I/Browser ( 1192): Content JS LOG: evme API request: https://api.everything.me/partners/1.0/Search/suggestions/?query=http://people.mozilla.com/~npierron/sunspider/hosted&apiKey=[…]&deviceId=fxos-app://search.gaiamobile.org/944b1bb8-5c4b-4ec5-91e4-b8301917ea99&ctx={"lc":"en-US","tz":"0","v":"2.2.0.0-prerelease","dn":"flame","sr":"480x853.5"}&
I/Browser ( 1192):     at log (app://search.gaiamobile.org/shared/js/everythingme/eme.js:73:8)
I/Browser ( 1192): Content JS LOG: evme API request: https://api.everything.me/partners/1.0/Apps/search/?query=http://people.mozilla.com/~npierron/sunspider/hosted&iconFormat=20&apiKey=[…]&deviceId=fxos-app://search.gaiamobile.org/944b1bb8-5c4b-4ec5-91e4-b8301917ea99&ctx={"lc":"en-US","tz":"0","v":"2.2.0.0-prerelease","dn":"flame","sr":"480x853.5"}&
I/Browser ( 1192):     at log (app://search.gaiamobile.org/shared/js/everythingme/eme.js:73:8)
I/Browser ( 1192): Content JS LOG: evme geolocation error 1 User denied geolocation prompt
I/Browser ( 1192):     at log (app://search.gaiamobile.org/shared/js/everythingme/eme.js:73:8)

The URL should not be transmitted to any service as soon as we can recognize it as being an URL in order to preserve the privacy of the user.
Currently when the user searches for a URL we show a list of related web results. E.g., if you type in facebook.com or twitter.com, we would show either web results from everything.me, or the app results from the marketplace.
(In reply to Kevin Grandon :kgrandon from comment #1)
> Currently when the user searches for a URL we show a list of related web
> results. E.g., if you type in facebook.com or twitter.com, we would show
> either web results from everything.me, or the app results from the
> marketplace.

This is another concern than this bug, but we should find a way to disable that if the user does not want it.
This bug is about sending the full content of the search bar constantly.

The search application does not need the ".com", "facebook" and "twitter" are enough for having good search results for applications.  Sending the full content of the search bar is really hurting privacy!

Currently we are sending:
  …?query=http://people.mozilla.com/~npierron/sunspider/hosted

We could have stopped sending anything after:
  …?query=http
  …?query=people
  …?query=people.mozilla.com

When this is clear that the user is not interested in any search result, but just in copying the link that he saw.  In addition to protecting his privacy, this would reduce the number of request made, which would them improve the responsiveness of the search completion, and save battery by avoiding Wifi / Data communications.
tracking-b2g: --- → ?
(In reply to Nicolas B. Pierron [:nbp] from comment #2)
> This is another concern than this bug, but we should find a way to disable
> that if the user does not want it.

We can do this in settings today, you can go to 'Search' and disable search suggestions. The user gets a notice for this the first time they run search, but we could do more to surface it.

I'm sure we could definitely have better heuristics of when to search remote providers or not.
(In reply to Kevin Grandon :kgrandon from comment #3)
> (In reply to Nicolas B. Pierron [:nbp] from comment #2)
> > This is another concern than this bug, but we should find a way to disable
> > that if the user does not want it.
> 
> I'm sure we could definitely have better heuristics of when to search remote
> providers or not.

Can we get this ASAP, and in all affected branches?
I agree that this is a pretty terrible behavior. Has the privacy team signed off on that 'functionality'?
[Blocking Requested - why for this release]:
This is critical privacy issue, I verified and I can reproduce this behaviour on 2.1.0.0-prerelease.
blocking-b2g: 2.2? → 2.1?
Johan, can you reproduce this issue on a device with 2.0?
I guess you can run one of the browser test of gaia-ui-test, while looking at the logcat.
Flags: needinfo?(jlorenzo)
Triage discussed and opted to block for privacy concerns and broader discussion. Alina, are you the right person to provide privacy team input on this issue?
Flags: needinfo?(ahua)
Peter, can you help find the right Product team member to help discuss this?
blocking-b2g: 2.1? → 2.1+
Flags: needinfo?(pdolanjski)
This also has a UX impacts as well. Some considerations:

- Search results are very different from entering "facebook.c" to "facebook.com". I'm not sure if this is a bug with the algorithm or not, but this impacts the results that users see.
- Clearing search results currently happens after every keystroke, our clearing strategy needs to change here.
- Do we also avoid sending the full URL to *all* remote providers including Marketplace?

We've already examined this issue when we implemented the ability and notice to disable suggestions. We may want to just leave this as-is for now, and make the search suggestions notice more prominent. Possibly consider popping it up on every new search unless the user checks some checkbox to never show it again.
Tested with :nbp on 2.0, when you type something in Rocketbar, you're able to see this log:
> E/GeckoConsole( 1480): Content JS LOG at app://search.gaiamobile.org/gaia_build_defer_index.js:349 in log: evme API request: https://api.everything.me/partners/1.0/Search/suggestions/?query=http://my-domain-name.tld/my/ressource.html?my-key=my-value&apiKey=79011a035b40ef3d7baeabc8f85b862f&deviceId=fxos-df9a9c60-94bd-4f5f-83f6-39f906ba9bc9&ctx={"lc":"en-US","tz":"2","v":"2.0.0.0-prerelease","dn":"Flame","cr":"Orange F","ct":"3G","mcc":"208","mnc":"01","sr":"480x853.5"}&
 
A request is sent to E.me even when you type the HTTP resource and some parameters (like /my/ressource.html?my-key=my-value just above)

Should we block on that on 2.0 too, Alina?
Flags: needinfo?(jlorenzo)
It certainly makes a lot of sense to improve the heuristics.  
I brought this up with privacy previously.  We felt that since you're usually sending everything up until the ":" or ".", not sending it at that point may not actually improve privacy that much in many cases.  I can also see that there are situations where it would, however.  

If we decide to make a change, it would probably be good to keep the results from e.me/marketplace that show prior to no longer sending the string out.  For example, if I type "flipkart", I get the icon to install flipkart, but as soon as I type the ".", it would negatively impact the user to remove that icon result.

I'm not sure we should block on this as Mozilla has reviewed e.me's privacy policy (and ours covers marketplace, obviously) and our privacy team felt comfortable with strings being typed in shared with these trusted partners.  I'll let Alina chime in as well.
Flags: needinfo?(pdolanjski)
Instead of just basing the discussions on privacy issues, there might be a better way to think about it:

Let's say it is a new feature.

1. Change of expectations 
   Users who were used that the system was local only for Firefox browser have a change of behavior, but are not informed about this new behavior before using it. We might want to explain the feature.

2. Deactivating the feature
   Currently in Settings -> Search -> Search Suggestions.
   It says that it is shared with search providers, but doesn't have a link to a more detailed explanations of who and what. 

3. Type of things 
   You might be comfortable sending away a keyword, but maybe not a URL as you type it.
   You might want to blacklist some URLs from search.

> I'm not sure we should block on this as Mozilla has 
> reviewed e.me's privacy policy (and ours covers 
> marketplace, obviously) and our privacy team felt 
> comfortable with strings being typed in shared with 
> these trusted partners.

This is all cool, but how the user can access for themselves if **they** are comfortable with the Privacy policy of everything.me. How do they discover it.
s/access/assess/ rha.
I don't like this behaviour very much, but as I understand it it has already been reviewed by the privacy team who were comfortable with this design for specified trusted partners only. The kinds of changes in behaviour being suggested above sound quite high risk to make at this stage of 2.1.

Note that all keypresses have always been sent to e.me on homescreen search, the difference now is that this search bar also serves as the only Firefox URL bar. The upside is that in 2.1 users are actually notified of this behaviour when they first use the search bar and are now given the ability to turn it off, which they weren't able to do in 2.0.

I recommend that unless the privacy team have new objections, we instead re-visit this behaviour in 2.2. We are considering adding multiple search suggestion providers and re-designing the search interface in 2.2 and there are other good reasons for doing this such as data consumption issues.
If there was a way to have it disabled by default and let the user enable it in FTU, it would be good enough for my concern. Even enabled by default and letting the user disable the feature in FTU would work for me if the FTU page is clear and prominent enough.
[Blocking Requested - why for this release]:

(In reply to Julien Wajsberg [:julienw] from comment #17)
> If there was a way to have it disabled by default and let the user enable it
> in FTU, it would be good enough for my concern. Even enabled by default and
> letting the user disable the feature in FTU would work for me if the FTU
> page is clear and prominent enough.

In implementing this capability we reviewed this as an option.  We felt that given that the target users have low levels of digital literacy (by this I mean they don't have experiences with the online world) and that showing app recommendations as you type was highly valued in our testing that the pros of defaulting to on exceeded the cons.  We do tell them how to change the settings after the user types three characters.  
The thing that isn't as clear is the implication of leaving it enabled.  The challenge with this is that explaining that characters are sent to a third party and that there are implications is inherently difficult for users with low digital literacy.  In particular, because e.me is integrated in the OS, the user is completely unaware of the existence of a third party, so referencing it in detail is very confusing.

Fennec does what you describe by asking the user "Would you like to turn on Google search suggestions?".  We cannot do the same (Would you like to turn on Everything.me search suggestions?) as the user would have no clue what they are being asked.

This is why we ended up enabling by default (to ensure that the value provided by the suggestions is there for most users) and not cramming in a novel-worth of information into the FTU prompt.  Can it be improved?  Absolutely.  Should we block the release on it?  I don't think we should, but I'd like Alina's opinion.
blocking-b2g: 2.1+ → 2.1?
(In reply to Peter Dolanjski [:pdol] from comment #18)
> Fennec does what you describe by asking the user "Would you like to turn on
> Google search suggestions?".  We cannot do the same (Would you like to turn
> on Everything.me search suggestions?) as the user would have no clue what
> they are being asked.

This has the form of a dark pattern. 

> Would you like to turn
> on Everything.me search suggestions?

Indeed this is not the right question and maybe you could test: 

1. the type of sentences that would be working
2. the feeling of users once you have explained, educated 
   about them about the non choice they are put in.

For example, as an "educated user", the good question for me would be:

> All searches will be sent to a 3rd party partner 
> Everything.me under this [policy](link). Would you like 
> to turn it on?
Ah yes forgotten:

As an "educated user", I had no idea I could deactivate it. 
If I had known it would have been one of the first things I would have done.
I think there is a huge scope for improvement here, but I think the ship has sailed for 2.1, its a similiar experience as to what has been shipped previously but with added notifications (there is notification given to the user the first time they search to tell them how to turn them off)

It sounds like we will be implementing the ability to configure your own search providers in 2.2, at that point an ftu screen with "do you want search results from X, Y or just local" would provide a massive improvement, as well as the ability to turn off suggestions (that line of text at the top, which I find pointless)
Francis, can you consider an improvement to the messaging for 2.2 UX?
It will be much easier to message with more traditional and known search engines being used for search suggestions in 2.2.
Flags: needinfo?(fdjabri)
Sure, I'll include this as an item for the Search app 2.2 specs.
Whiteboard: [systemsfe]
Not blocking for 2.1 will be fixed in the redesign slotted for 2.2
blocking-b2g: 2.1? → -
feature-b2g: --- → 2.2+
Assignee: nobody → kgrandon
Status: NEW → ASSIGNED
Comment on attachment 8536909 [details] [review]
[PullReq] KevinGrandon:bug_1082787_search_provider_url to mozilla-b2g:master

Dale - could you review when you get a chance? Thanks!
Attachment #8536909 - Flags: review?(dale)
Comment on attachment 8536909 [details] [review]
[PullReq] KevinGrandon:bug_1082787_search_provider_url to mozilla-b2g:master

Redtriggered the OOP mochitests although undoubtedly unrelated

This looks great, thanks
Attachment #8536909 - Flags: review?(dale) → review+
In master: https://github.com/mozilla-b2g/gaia/commit/1b23a7a03da9f3a55ff7439b1bb62f7f8b1da31b

This patch should stop sending urls as soon as you type in a dot (.) as that triggers our isURL() check.

I don't think we really need discussion here so going to clear the needinfos on Alina/Francis. Feel free to chime in or re-flag them if there's more to discuss here. Thanks for the feedback on this everyone.
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(fdjabri)
Flags: needinfo?(ahua)
Resolution: --- → FIXED
Target Milestone: --- → 2.2 S2 (19dec)
This issue is verified fixed on Flame 3.0

If the user is conducting a search in the search bar, they will receive relevant E.Me searches until they put a "." in their search, in which case the search behaves like a URL search

Environmental Variables:
Device: Flame 3.0 (319mb)(Kitkat)(Full Flash)
Build ID: 20150210010523
Gaia: 0cf517083f7eb5fc269e1236edba50534f65e3cd
Gecko: 2cb22c058add
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 38.0a1 (3.0)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0
Status: RESOLVED → VERIFIED
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(pbylenga)
adding the verifyme keyword for this bug to be verified on 2.2
Keywords: verifyme
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(pbylenga)
This bug has been successfully verified on latest Flame v2.2.
See attachment: verified_v2.2.mp4.
Reproduce rate: 0/5.

STR:
1.Tap Rocket bar at the top
2.Input a dot(.)
**The search behaves like a URL search.
3.Input some words and then input a dot(.)
**They will receive relevant E.Me/marketplace until they put a "." in their search,in which case the search behaves like a URL search.
4.Go to Settings and disable "Search Suggestions".
5.Back to Rocket bar and input some words or a dot(.).
**User will not receive relevant Everything.me/marketplace searches and the search behaves like a URL search.

Flame 2.2 build:
Build ID               20150211002505
Gaia Revision          943be6fd146017dcd9d4c9d1027be1e43bad13eb
Gaia Date              2015-02-11 08:01:09
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/e614443583e7
Gecko Version          37.0a2
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150211.040242
Firmware Date          Wed Feb 11 04:02:53 EST 2015
Bootloader             L1TC000118D0
Leaving "verifyme" for v2.0&2.1.
You need to log in before you can comment on or make changes to this bug.