Closed Bug 1029189 Opened 11 years ago Closed 11 years ago

[UX] Design experience for new default search test

Categories

(Firefox :: Search, defect)

Product:
Firefox
Component:
Search
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal
Points:
13

Tracking

()

Status:
RESOLVED FIXED
Iteration:
33.2

People

(Reporter: madhava, Assigned: phlsa)

References

Details

(Whiteboard: [ux])

Attachments

(3 files, 4 obsolete files)

Example of something we could do
1.23 MB, image/jpeg
Details
Mockup v2.1
377.59 KB, image/png
Details
Mockup for switching back to the old provider
155.11 KB, image/png
Details
In support of the test of switching users to a new default search engine, we need to define the experience through which a user discovers the change. Is it: (a) On first run? On first search? (b) Where does it occur? (b) Is it transient or does it persist? (c) Where do users go to learn more? Attached is an concept sketch.
QA Whiteboard: [qa-]
Whiteboard: [ux]
Assignee: nobody → philipp
Status: NEW → ASSIGNED
Iteration: --- → 33.2
Points: --- → 13
Attached image WIP Mockup (obsolete) —
This would appear when a user starts typing in the search box. It is still lacking final copy and the reversal of search engines at the end of the experiment isn't dealt with yet, but it should give us a good starting point. Some additional notes: - This should be shown only in the first session after the switch, but for a maximum of five searches - The »Learn more« link leads to: https://support.mozilla.org/en-US/kb/search-bar-easily-choose-your-search-engine?esab=a&s=search&r=2&as=s - Other search entry points aren't affected right now. If we actually switch default search providers in the future, we should already have suggestions in our other search properties, so this approach would work there as well
If we're concerned about the field on about:home, could we usurp the snippet until a search is performed? Too complicated to be worth it?
And we could use the same mechanism to tell people when we've switched back, post-test.
(In reply to Philipp Sackl [:phlsa] from comment #1) > Created attachment 8445372 [details] > WIP Mockup How about an [x] to close it, as in http://cl.ly/image/423g1x1n0a0p
Would this also appear when the user starts typing in the about:home search box?
Flags: firefox-backlog+
(In reply to Philipp Sackl [:phlsa] from comment #1) Overall looks good to me. I like offering the new and avoiding spelling out the [x] -> [y] transition. My only question goes back to Madhava's comment 0, is this the right moment to inform. I'm wondering if we can quickly test when its best to inform people of the change. I lean toward a notifier after the first search since the change; when viewing the results page. I'm not sure what the right answer is or if we should just go ahead with one of them. Also it'd be good to get gavin or another dev to quickly comment on ease of adding an item like that to the auto-complete list; there might be a similar opportunity available if this is cumbersome (as I naively assume it is) to implement. > It is still lacking final copy and the reversal of search engines at the end > of the experiment isn't dealt with yet, but it should give us a good > starting point. Bill would be really helpful crafting the copy since he's been running isolated search experiments that have uncovered some user mental models of our search interfaces. I might try to present this as a new opportunity: "Firefox supports Yahoo! integrated search! We thought you'd like to try it out, you can change this setting at any time." but I bet Bill will have better suggestions.
Based on the SAP study that I conducted and the current study that I am conducting specifically on Search Diverted users, I think this is a fantastic idea. While I have just started analysis on the Search Diversion tests, it is very clear already that the idea of of "default search" in Firefox is not clear to some portion of Firefox users. Through several pilots, I had to hone down the question of how to ask participants what their current default search engine is in Firefox. For some participants "default search engine" always means "navigating to google or bing," not using Firefox's built-in SAPs. So, we could posit that there is a kind of SAP blindness for some of our users. There are a few observed reasons for these users not using SAPs: 1. Some hijacked users are navigating directly to the search engine URL as a bookmark or the homepage as a work-around to reach their desired search engine. These users frequently also do not understand how to set the Firefox default search engine to their desired search engine. We know this happens from Gregg's study and this is a partial explanation for this behavior. 2. Some users (hijacked or not) are completely unaware of the SAPs' functionality and are blind to it. These users navigate directly to the search url. 3. Some users prefer to search in google because they prefer the display of search suggestions. While we offer search suggestions in Firefox in the search box, its narrow width prevents the full view of suggestions. Users can of course customize this width, however, in the ~100+ search related user tests I have run in the last 2 months, I have not seen a single instance of a participant's search box being larger than its default setting. This indicates to me that customization here is either undiscoverable or undesirable. Regardless we should reevaluate the display of search suggestions or the search box default width. Regardless, drawing more attention to SAPs and their functionality will be valuable to (re)demonstrating this feature to some of our users. I would be happy to explore proposals and validate them in the wild.
(In reply to Bryan Clark (Firefox PM) [:clarkbw] from comment #6) > My only question goes back to Madhava's comment 0, is this the right moment > to inform. I'm wondering if we can quickly test when its best to inform > people of the change. I lean toward a notifier after the first search since > the change; when viewing the results page. I'm not sure what the right > answer is or if we should just go ahead with one of them. An alternative design I considered was an info bar when the user reaches the results page. But I think it makes sense to surface this message at a moment when the next action of the user would be as simple as possible. In the case of the current design, that would be just pressing enter (or selecting an autocomplete result). On the search results page, the user is back in scanning mode, which could make him more easily distracted by our message. That means that the users who care about what is their default search engine get a heads up and don't feel cheated, while the (presumably larger) portion that doesn't care has as little interference and distraction as possible.
Pasting a concern that Chad raised via Email (mostly for Matejs attention): > based on conversation b/w eric, denelle, susan, me, our > recommendation is this direction to satisfy partner, user > and experiment purity concerns: > In this version of Firefox beta, we are using Yahoo! as the > default search engine. You can change this setting at any time.
Re #9, the "you can change this" is a bold claim. If we offer a link for how to do that, that's one experiment. If we don't, it's another. I would summon John on that. (my vote, offer the link to change back, so we can assess worst case scenarios, if we would ever do this in the real deploy.)
Why is it a bold claim? You just pick a different engine from the dropdown. The UI is all right there.
#11: Many, many users don't know 1. That you can change the engine. 2. How to do it. Remember that what seems obvious, often isn't. Bill literally just watched someone who didn't know you can change the engine.
Picking a different engine from the dropdown is not discoverable to some portion of our users. In fact, I just finished watching a test for the hijacking study where a participant said, "Oh, I didn't know that I could change the search engine [in the search box]." This indicates that some participants would not be able to return to their desired search engine. It is not our policy to publicly share user test and interview data. If others would like to see evidence of this phenomenon, ping me on internal email.
I agree that it's not obvious that you can switch engines. To clarify something that may have become less clear -- even with the revised copy in Comment 9, we should retain the "Learn More" link as in Philipp's mockup. It will point to the SUMO article on how to switch engines. This isn't as ironclad a route back to the previous default engine as a control/link to actually do make the switch, but it seems appropriate for this test. If we think that there should be a more direct "revert" path then we should decide that.
#14 Madhava, that "learn more" link (and any messaging here) is the 'revert path' I meant. Nothing more magical. Balancing UX, Mozilla Values, Goals, and Design is hard here :) As we all know, there are many outcomes for users: notice change in engine, are annoyed, can revert -> will revert = empowered notice change in engine, are annoyed, can't revert -> stop using = helpless notice change in engine, like it -> keep using, improved don't notice change in engine, dislike results, subtle feelings of malaise -> stop using = helpless don't notice change in engine, like -> keep using = happy Both NOTIFYING and EMPOWERING here affect the final totals for the different outcome buckets of the experiment. NOTIFY + EMPOWER -> more switchbacks, reduced "quit", fewer try new engine. NOTIFY - EMPOWER -> more 'quit' (because disempowered people have no recourse) - NO NOTIFY -> more 'quit', but maybe more people try new engine (helpless!)
Blocks: 1029265
Yes, I agree with Madhava's point about Philip's mock up. However, we need to be more explicit about what "default search engine" in Firefox means because it is not immediately obvious to all users. Perhaps illustrate how to use it in one test cell? One of the goals of the search hijacking study I am conducting is to determine if participants intend to be hijacked. It has been challenging to explain "default search engine in Firefox" to participants. Even with a very detailed explanation in the test protocol of how to identify it, there are still some users who are confused. As an example, one user had the interesting mental model that because she was signed into her Bing account that it should change her default search engine to Bing in Firefox.
No longer blocks: 1029265
Blocks: 1029265
Further copy ideas from Matej: > You're now searching with Yahoo. You can change your default search engine at any time. > > Yahoo is now your default search engine. Find out how you can change this setting. > > Yahoo is now the default search engine in Firefox.
Attached image Mockup v2 (obsolete) —
Updated mockup with different copy. I did a very slight adjustment to the wording (dropping the »we«). Let me know what you think.
Attachment #8445372 - Attachment is obsolete: true
Attached image Search Test Restore Mockup (obsolete) —
This is how we can handle restoring the search engine after the test. Copy tbd.
Attached image Mockup v2.1 —
There's really no reason people shouldn't be able to dismiss this message, so this mockup takes that into account.
Attachment #8446566 - Attachment is obsolete: true
Attached image Search Test Restore Mockup v1.1 (obsolete) —
Also adding the close button. For both instances: when the user clicks the »Learn more« link, we should stop showing the notification.
Attachment #8446638 - Attachment is obsolete: true
Note that the style of popup shown in the mockup is not the actual style of the searchbar popup. Are we supposed to just add the footer to the popup as presented, or change the whole popup styling? The latter is a much wider change so we'd need to know soon if that also should be implemented. Also it would be nice to think how the styling fits on Windows.
Philipp, I added a work-in-progress implementation over in bug 1029792 comment 9. Could you please run it and tweak the CSS that I have there to be closer to the specs you want?
Since this is now in implementation in bug 1029792, I'm closing the design bug.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
How do we handle this corner case: - Experiment switches from default provider A, to provider B. (And the test group is meant to get a notification) - User immediately notices change (or any time before the "display 5 times" is reached) and changes back to provider A (or a different provider, C). Do we: - display the original intended notification, which is now inaccurate, saying "This version of Firefox now uses provider B" - display the notification with the updated provider, e.g. "This version of Firefox now uses provider A" (which is redundant information given that the user has just set it) - stop displaying notification I guess my description is a bit biased towards the obvious answer :), but I just wanted to state this corner case and confirm it (and maybe there are other options that I'm not thinking about)..
Flags: needinfo?(philipp)
In this case we should stop displaying the notification – I guess that's the option you've been biasing towards ;)
Flags: needinfo?(philipp)
Posting the mockup with the updated copy here for completeness.
Attachment #8446642 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: