596439 - give searches from url bar a unique parameter to distinguish it from search bar

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[Alex Limi (:limi) — Firefox UX Team]

Filing this bug to ensure we have the means to track the searches from the following locations with separate codes:* Searches from the search bar* Searches from Firefox Start* Searches from the URL barI assume the search bar + FF Start already have separate codes, we need to make sure we can identify the searches from the URL bar with the recent changes in FF4.There might be a Google component here too (e.g. do we need to tell them about the new code?), but I'll leave that to the CCed parties to clarify.This probably also needs to block the final release of Firefox 4.

Comment 1

[Alex Limi (:limi) — Firefox UX Team]

Requesting blocking for Firefox 4 final release.

Comment 2

[Alex Limi (:limi) — Firefox UX Team]

For context on the URL bar searches, see bug 586821.

Comment 3

[Kev Needham [:kev]]

Just to clarify, we're talking about Google channels only, correct?

Comment 4

[Alex Limi (:limi) — Firefox UX Team]

Yes, I assume our Yandex volume isn't substantial enough yet to need this (and would likely result in similar distribution anyway)

Comment 5

[:Gavin Sharp [email: [e-mail]]]

(In reply to comment #0)
> Filing this bug to ensure we have the means to track the searches from the
> following locations with separate codes

You mean "so that Google has the means to track them", right? We don't proxy searches through URLs that we control - they go directly to Google.

We use the same URL for keyword.URL and the search bar. We also use that URL for the search bar on about:home, but add the "&source=hp&channel=np" parameters to it (per bug 594675).

Comment 6

[:Gavin Sharp [email: [e-mail]]]

Given keyword.URL a unique parameter would be doable, if desired (though we'd probably need to use a hacky approach similar to the one used for about:home...). Do you want to morph this bug to cover that?

Comment 7

[Alex Limi (:limi) — Firefox UX Team]

(In reply to comment #5)
> You mean "so that Google has the means to track them", right? We don't proxy
> searches through URLs that we control - they go directly to Google.

Yes, whatever we add to the query string to separate Start page searches from search bar searches should also be added to the URL bar search. I have no idea how this is implemented, just wanted a bug on it, so it is tracked somewhere. :)

Comment 8

[Kev Needham [:kev]]

Most likely the parameter will be "channel", but I'll need to coordinate with some folks here and at Google first. I think the same approach that we've looked at for the local home page applies, and adding the location bar searches should be the only other search surface that'd be affected.

Specifics to follow, but typically look to separate out queries form the home page, queries from the search bar, and queries from the location bar. The parameter will vary by provider, so it should be flexible. I'll update this bug with specifics when I have it.

Comment 9

[:Gavin Sharp [email: [e-mail]]]

We already separate out about:home from search bar (see comment 5 / bug bug 594675).

Current state of affairs is this:
1) URL bar keyword search
  - Normal Google URL
2) Search bar search
  - Normal Google URL
3) about:home search field
  - Normal Google URL, plus "source=hp&channel=np"
4) Selected text context menu search
  - Normal Google URL

Adding a parameter for 1) is easy, just need to know what parameter to add (channel=kw?).

Adding a parameter for 4) would involve more work, so I'd like to avoid it. (It's likely a tiny number compared to the others...)

Comment 10

[Alex Limi (:limi) — Firefox UX Team]

(In reply to comment #9)
> We already separate out about:home from search bar (see comment 5 / bug bug
> 594675).
> 
> Current state of affairs is this:
> 1) URL bar keyword search
>   - Normal Google URL
> 2) Search bar search
>   - Normal Google URL
> 3) about:home search field
>   - Normal Google URL, plus "source=hp&channel=np"
> 4) Selected text context menu search
>   - Normal Google URL
> 
> Adding a parameter for 1) is easy, just need to know what parameter to add
> (channel=kw?).

Works for me.

> Adding a parameter for 4) would involve more work, so I'd like to avoid it.
> (It's likely a tiny number compared to the others...)

Yeah, I wouldn't worry about that one for now.

Comment 11

[Axel Hecht [:Pike]]

Right now, the comments here lead me to believe that we're just talking about making google know where a search came from. I don't see us having anything that tells us, aka a feedback experiment, a search came from.

Comment 12

[Mike Beltzner [:beltzner, not reading bugmail]]

Kev: you want to take this to Steve? I suspect they'll want to modify the channel= variable, as you say. They're using "np" for "new page"; perhaps "lb" for "location bar"? :)

Comment 13

[Alex Limi (:limi) — Firefox UX Team]

Not sure I understand why this is not blocking; having the ability to track how many of our users prefer the URL bar searches over the search box seems to be very important data to me, and is also important when considering the evolution and possible merge of search field & URL input fields in the future.

Re-requesting blocking.

Comment 14

[Axel Hecht [:Pike]]

Limi, what you're asking for doesn't do what you expect it to, AFAICT.

With what folks are talking about in this bug, google will know how our folks use url bar searches, we will not. Is that your intention?

Also, http://mxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js#219 sets keyword.URL to blank now.

Comment 15

[:Gavin Sharp [email: [e-mail]]]

keyword.URL isn't relevant, we can make location bar searches differentiable (to Google, not to us, as you point out) using the same technique we used for Yandex (x-moz-keywordsearch).

Comment 16

[Dietrich Ayala (:dietrich)]

Blocking- for comment #14 and #15. IIUC, this change doesn't do what Alex thinks it does, and outside of that, there's a better way to do it.

Comment 18

[Frank Yan (:fryn)]

I explained to Limi how the code works, so he now has an understanding for why keyword.URL is irrelevant. Changed the bug title to reflect this.

Comment 19

[Alex Limi (:limi) — Firefox UX Team]

(In reply to comment #16)
> Blocking- for comment #14 and #15. IIUC, this change doesn't do what Alex
> thinks it does, and outside of that, there's a better way to do it.

I was asked by Jim to make sure we track this on the Google side. He can get the data on how search is initiated, and will — just because we currently don't do it, doesn't mean we don't want to. Happy to discuss it if you need more data, or more clarification.

Re-requesting blocking.

Comment 20

[Alex Limi (:limi) — Firefox UX Team]

And, carrying over information from bug 624054:

Ideally, searches from the Location Bar should have a source attribute, I
propose we add "&source=lb" for searches done here.

Comment 21

[Kev Needham [:kev]]

We'll need to confirm with Google on what an acceptable parameter would be, as they'll be pulling it. I believe they have a field for channels, but can't remember off-hand what it is. I'll talk to them tomorrow and put a patch in when I have it, but generally it can't be an arbitrary attribute.

While the current implementation of application/x-moz-keywordsearch is irrelevant in this use-case, I would like to see if we can't make it relevant for all other search surfaces in the future; it would help simplify search plugin integration/admin.

Comment 22

[Frank Yan (:fryn)]

(In reply to comment #20)
> And, carrying over information from bug 624054:
> 
> Ideally, searches from the Location Bar should have a source attribute, I
> propose we add "&source=lb" for searches done here.

(In reply to comment #21)
> We'll need to confirm with Google on what an acceptable parameter would be, as
> they'll be pulling it. I believe they have a field for channels, but can't
> remember off-hand what it is. I'll talk to them tomorrow and put a patch in
> when I have it, but generally it can't be an arbitrary attribute.

There is a parameter called "channel" that is used in some cases, but I don't know what it means. Setting the "source" parameter seems to make sense, because it's currently set to "hp" (short for home page) when searching from about:home.

Comment 23

[Kev Needham [:kev]]

Per above, it's something we shouldn't assume. 

Beltzner: could you confirm Google asked us to use "source" for the local start page? I'll use that as the lead-in with Google if so, and advise them of plans if we're all in agreement. 

> There is a parameter called "channel" that is used in some cases, but I don't
> know what it means. Setting the "source" parameter seems to make sense, because
> it's currently set to "hp" (short for home page) when searching from
> about:home.

Comment 24

[Jim Cook (NEED INFO ONLY - Please flag)]

Hey all...jumping in here late.  Sorry for the confusion by not being on here earlier.

Overarching Goal of this request
1)  Ensure we start getting reports from Google that separates Location Bar Searches (lb) from Startpage Searches vs Chrome Searches.  We get the latter 2 already on a monthly basis from Google in a critical report we analyze deeply.

2)  Without this separation of Location Bar vs Chrome, we will lose some critical data on our User Behavior for which we can build future great UI/product offerings

In reply to:
(comments 11 and 14:  Axel) - Yes, Google will know but they already know whether or not we do anything (they have enough statisical data on their own Chrome product feature).  By adding the attribute to FF4, we can force Google to report it back to us at Mozilla

(comment 13:  Limi) - I agree with this comment

(comments 15 & 16 - Gavin & Dietrich) - sounds like there might be a better way of doing it?  That's great.  Don't really care about the way as long as the requirement is met that Google starts adding this Location Bar Field to our Monthly Report they deliver to us.

(comment 21 - Kev) - Yes, we definitely need to communicate with Google to ensure they are aware of any change and can and will start reporting it back to us in a separate field

Comment 25

[:Gavin Sharp [email: [e-mail]]]

Bug 594675 is where we added the about:home parameter.

Not sure I understand the previous comments about x-moz-keywordsearch - we do want to use it for this, just like we did with Yandex:
http://hg.mozilla.org/l10n-central/ru/rev/07ae2c762256

Comment 26

[Kev Needham [:kev]]

(In reply to comment #25)
> Bug 594675 is where we added the about:home parameter.
> 
> Not sure I understand the previous comments about x-moz-keywordsearch - we do
> want to use it for this, just like we did with Yandex:
> http://hg.mozilla.org/l10n-central/ru/rev/07ae2c762256

my age-addled brain. I thought we had removed that bit. my bad.

Comment 27

[Justin Dolske [:Dolske]]

Unowned blocker, over to Frank.

Comment 28

[Mike Beltzner [:beltzner, not reading bugmail]]

I'm taking this off the blocker list. There's nothing stopping us from doing this after we ship Firefox 4, and a precursor step of working out the appropriate codes with Google still hasn't happened.

Comment 29

[Frank Yan (:fryn)]

Unassigning self, unless it becomes a blocker again or until we work out the appropriate codes with Google.
I'm happy to pick it up later though, if no one else is.

Comment 30

[Alex Limi (:limi) — Firefox UX Team]

OK, finally got confirmation from Google on what to do here:

The query string for the location bar searches should be:

http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&client=google&channel=fflb&q=

Google representative said: "The only 2 new parameters being added to the existing setting are client=google and channel=fflb."

Comment 31

[:Gavin Sharp [email: [e-mail]]]

Attachment: patch

This adds the two extra parameters for URL-bar triggered searches.

With this patch, the URLs used for a query of "test 123" are:

Search bar (unchanged):
http://www.google.com/search?q=test+123&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a
URL bar (two added parameters):
http://www.google.com/search?q=test+123&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a&client=google&channel=fflb

Comment 32

[:Gavin Sharp [email: [e-mail]]]

There's a problem with comment 30 that I failed to notice earlier - we already send a "client" parameter, whose value is typically "firefox-a" (or "firefox" for builds where Google is not the default engine). I imagine we'll just want to continue sending that, and not add the additional client=google.

Comment 33

[:Gavin Sharp [email: [e-mail]]]

Attachment: updated patch

Comment 34

[Kev Needham [:kev]]

Gavin: correct. client codes need to match what we're sending in the search bar.

Comment 35

[Alex Limi (:limi) — Firefox UX Team]

I asked Kathy from Google, and she said:

> For tracking of the awesome bar, you will need to set the client=google (not firefox-a or firefox in this use case).  Please note that tracking will not work otherwise and all our previous testing was using this new keyword.URL setting, with client=google and channel=fflb.  This setting is only for searches coming from the awesome bar.

Kev, can you make sure we're doing the right thing here with Google?

Comment 36

[Alex Limi (:limi) — Firefox UX Team]

Bleh, damn Bugzilla. Let's make that readable:

“For tracking of the awesome bar, you will need to set the client=google (not firefox-a or firefox in this use case).  Please note that tracking will not work otherwise and all our previous testing was using this new keyword.URL setting, with client=google and channel=fflb.  This setting is only for searches coming from the awesome bar.”

Comment 37

[:Gavin Sharp [email: [e-mail]]]

That doesn't sound right - we still want these searches to be identified as coming from Firefox. Changing the value of the "client" parameter doesn't really make sense, assuming that's what it's used for.

Comment 38

[:Gavin Sharp [email: [e-mail]]]

note to self: need to also update browser_keywordSearch.js

Comment 39

[:Gavin Sharp [email: [e-mail]]]

Attachment: patch with test fix

We still need to get confirmation about the parameters, but no matter what the details end up being, this is the approach we'll need. I still think a simple additional parameter is correct.

Comment 40

[:Gavin Sharp [email: [e-mail]]]

Attachment: real patch with test fix

Comment 41

[Justin Dolske [:Dolske]]

Comment on attachment 527409 [details] [diff] [review]
real patch with test fix

r+ assuming you check with Google to confirm the arg is what they'll want.

Comment 42

[Kev Needham [:kev]]

We're still in a holding pattern with Google, so please hold off landing this. Hope to have red light/green lighr next week, per Google reps.

Comment 43

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 527409 [details] [diff] [review]
real patch with test fix

This works for the moment, but I think we'd rather wait to fix bug 587780 and use that mechanism.

Comment 44

[:Gavin Sharp [email: [e-mail]]]

This was fixed by bug 724116. Bug 587780 will eventually make it a little more maintainable in general.