649457 - Data on what people do after they open a new tab

Status: RESOLVED FIXED

(Mozilla Labs Graveyard :: Test Pilot Data Requests, defect, P1)

Description

[Jinghua zhang]

Faaborg and the UI team are interested to have this data to help design the content on new tab page:

After people open a new tab (via mouse or keyboard), what do they normally do:
- use search box
- use bookmark
- use history
- use the URL bar
- use the home tab
- others?

It would be great to have both data on percent cross all samples, and data per user.

We could use "Firefox Beta UI - v2" data for this request.

Comment 1

[Alex Faaborg [:faaborg] (Firefox UX)]

Primarily what I want us to find here is how many different URLs do users travel to after opening a new tab over the course of a fixed amount of time.

The hypothesis is that when using a search/keyboard interface they will travel to a long tail-ish set of many different pages, (based on the intention that is currently in their head when they decided to open a new tab).

Then I want to contrast this data with potentially introducing a grid of 9 pages, with the theory being that user's behavior may shift to a fat-head-ish pattern of visiting 9 sites, especially since those site have been selected to be compelling to each individual user.

If we see this occur with the same set of users, I think we can mostly determine that they likely had the same initial intentions in the second test, and the navigation events were the result of the interface compelling them to visit one of their 9 favorite sites.

By getting data on the current usage, we will be able to determine if the interface is changing behavior, and the browser is starting to drive the user instead of the user driving the browser.

Comment 2

[Jono Xia]

The dataset from "Firefox Beta UI - v2" lets us answer the "what ui action did the user do after opening a new tab" (url bar vs search box vs whatever) but it won't tell us how many different URLs are visited since there was no URL collection in the beta UI study.

To answer the URLs question we need a new study with some kind of URL-hashing data structure in it (since we want to upload only the count of uniques, not the URLs themselves.)

Comment 3

[Jinghua zhang]

@faaborg:  I'd like to step back a bit and ask some background questions, 

1. why we want to redesign the new tab page?  What benefit you want to provide to users by having a new design? 

2. Does this benefit align with other FX browser features, or conflict/overlap with them?

3. Is the grid of 9 favorite pages the only design proposal we have now? If not, what other proposals do we have?  And why they are important or not important?

4. Will the grid of 9 favorite pages be conflict/overlap with other FX features? How is the overall FX experience planned by considering this design?

We can get the URL data for you in the next study if needed. But before that, let us know if we can help you with other information.

Comment 4

[Jinghua zhang]

(In reply to comment #0)
> Faaborg and the UI team are interested to have this data to help design the
> content on new tab page:
> 
> After people open a new tab (via mouse or keyboard), what do they normally do:
> - use search box
> - use bookmark
> - use history
> - use the URL bar
> - use the home tab
> - others?
> 
> It would be great to have both data on percent cross all samples, and data per
> user.
> 
> We could use "Firefox Beta UI - v2" data for this request.

// Limi is also interested to know these numbers by mouse, and by keyboard.

Comment 5

[Alex Faaborg [:faaborg] (Firefox UX)]

>@faaborg:  I'd like to step back a bit and ask some background questions, 

Sure thing, always useful to review the existing set of assumptions :)

>1. why we want to redesign the new tab page?  What benefit you want to provide
>to users by having a new design? 

Reasons why a redesign of the new tab page may overall be a good idea include:
1) feature parity with competing browsers.  This is something that users mention we are missing in market research studies when asked what they prefer about other browsers
2) the heuristic of recognition over recall.  A blank page is potentially the least useful thing we can show the user if they aren't really sure where they want to go.
3) new opportunities to streamline certain interactions, for instance we could run data detection on the contents of the clipboard and provide intelligent actions, like mapping an address or looking up a movie title on imdb (full disclosure: my masters thesis was on data detection, so I have an very biased view on how incredibly magical and wonderful it is :)

Reasons why a new design may not benefit users in general (or a subset of users):
1) users find the page is visually distracting, which results in an increased time on task for some users as they consider multiple targets (this effect should be detectable)
2) the page actually changes user behavior, and they get mentally redirected from the navigational target they were planning to go to, to one of the more appealing procrastination targets that we are promoting.  (This effect will be present, but harder to detect, we need to get a clear picture of current navigational behavior before we can see if the interface changes it, and people start to visit fewer and fewer sites as they are tempted by the ones that the interface is promoting).
3)  Some users may feel that a more complex interface makes Firefox bloated, slow (purely a feeling), and in general less elegant.  This is purely a product perception issue as we introduce any level of complexity into the UI.  This can also be mitigated by the visual design, but it's generally speaking a risk.

>2. Does this benefit align with other FX browser features, or conflict/overlap
>with them?

The existing feature in Firefox that is the closest to a visual grid of top sites is the drop marker in the location bar, that provides a list of the top sites that users go to.  Usage of this control is surprisingly high given its placement in secondary UI and overall visual treatment (I think we've made the arrow smaller and lighter in each release of Firefox).

In terms of alignment with other Firefox features, it's my personal belief at the moment that we may be able to get the optimal solution by introducing a home tab for the more visual and distracting aspects of the design, and try to keep New tab based on the user's existing intent (new item on the clipboard, command words, etc.)  More details here:

http://blog.mozilla.com/faaborg/2011/04/13/the-firefox-home-tab/
http://people.mozilla.com/~faaborg/files/20110413-homeTab/comparison.png

However, the team is split on this.  Other people believe that it introduces needless redundancy and we could achieve everything with a single UI change on new tab.

>3. Is the grid of 9 favorite pages the only design proposal we have now? If
>not, what other proposals do we have?  And why they are important or not
>important?

Design work is in the very early stage.  Most of the proposals specifically for new tab (as opposed to home tab) revolve around:
-Top sites
-Available command words (search engines and ubiquity-ish)
-Clipboard data detection

>4. Will the grid of 9 favorite pages be conflict/overlap with other FX
>features? How is the overall FX experience planned by considering this design?

If one of the pages is already open, we may want to use switch to tab, to reduce the number of redundant tabs open.  Also, if that tab is in another group, we may want to show the panorama transition to help the user build a mental model of switching groups.

Otherwise, there is also some indirect overlap with the bookmarks toolbar, however this is now not displayed with new profiles, or if the user did not modify it.

>We can get the URL data for you in the next study if needed. But before that,
>let us know if we can help you with other information.

Next study is fine, the most important thing is that we start to understand existing behavior before we start to change the interface, which will subsequently change user behavior.  The only way for us to really understand the effect we had on user behavior is to get the most accurate data we can on time on task for new tab, and the breadth of different page that users are visiting.  I'm personally concerned that we may regress both of these things if we aren't careful.

Comment 6

[Alex Faaborg [:faaborg] (Firefox UX)]

>Limi is also interested to know these numbers by mouse, and by keyboard.

the reason for this is that (hypothetically), the breadth of the Web that user's visit may be smaller for the people who use the mouse to click on new tab, than for users who use the keyboard to enter accel-T.

If we see enough of a difference, that might be a reason for us to lean towards displaying different interfaces on the page depending on how the page was invoked.  Or displaying the same set if information differently (placement, thumbnails versus icons, etc.)

Comment 7

[Jinghua zhang]

(In reply to comment #5)

> Next study is fine, the most important thing is that we start to understand
> existing behavior before we start to change the interface, which will
> subsequently change user behavior.  The only way for us to really understand
> the effect we had on user behavior is to get the most accurate data we can on
> time on task for new tab, and the breadth of different page that users are
> visiting.  I'm personally concerned that we may regress both of these things if
> we aren't careful.

Got it, thanks. I am with you here. The advantage and disadvantage of having a not-empty new tab page is not necessarily conflicted with each other if we carefully design it.  User behaviors will be changed, but not necessarily have to be a negative way.  

We will help you with the data on tasks for new tab from the current data, and the breadth of different page that users are visiting in the next study. 

About the concerns on "visually distracting" or "perception of slowing down", we can help you test them when the mock is ready, since these concerns could be minimized by visual design, after the core function and the info architecture are decided.

Comment 8

[Jinghua zhang]

This bug will be used to get data on tasks for new tab from the Beta UI Study- V2.  

I opened a new bug to request a new study for the breadth of different page that users are visiting after opening a new tab in the next study.

Comment 9

[Jono Xia]

OK, here's what I was able to find out from the Beta Interface 2 study so far.

Using the small random subsample (1800 users) and counting events that were done immediately (less than 15 seconds) after a new tab was opened (by button, menu item, or keyboard shortcut), I get the following percentages:

go to url by enter key : 43.43 %
search by enter key : 25.52 %
home button: 9.21 %
open most frequently used drop down menu : 7.32 %
new tab button : 4.93 %
bookmarks button : 2.44 %
drop down menu : 1.68 %
firefox button : 1.23 %
slider track and drag : 1.04 %
search by go button : 0.89 %
back : 0.55 %
new tab : 0.53 %
go to url by go button : 0.26 %
reload : 0.22 %
same search same engine : 0.20 %
panorama button : 0.10 %
feedback button : 0.08 %
find : 0.07 %
exit : 0.05 %

All others:  Less than 0.05%

Caveat: this is counting events, not users, so a single user very active user could have more effect on these numbers than several less active users.  I can do another one where I count each user only once and see if that makes a significant difference to the results.

Of course the really interesting part will be the breakdown of "go to url by enter key: 43%" since that lumps a lot of different behaviors together (e.g. we can't tell if it was a brand-new, typed-in URL or a frequently visited URL or what).  The beta interface study just didn't provide the data we need to break that down any further.

It's weird that "new tab button" would be as high as 5% since this represents doing the same thing twice in a row.  Could it be a misclick?  Or an extra click the user did because the UI didn't respond fast enough?

Comment 10

[Alex Faaborg [:faaborg] (Firefox UX)]

>after a new tab was opened (by button,
>menu item, or keyboard shortcut), I get the following percentages:

Is there any way to break these percentages apart based on the way that the new tab was invoked?

Comment 11

[Hamilton]

Here is a breakdown of the various actions taken after opening a new tab, broken down by how the new tab was invoked.  I took the same criteria that Jono used, but also subset on Windows users only.


total 31515
0.363 (u'urlbar', u'url', u'enter key')
0.119 (u'searchbar', u'', u'enter key')
0.096 (u'bookmark toolbar', u'personal bookmark', u'click')
0.078 (u'urlbar', u'most frequently used menu', u'open')
0.077 (u'urlbar', u'search term', u'enter key')
0.073 (u'home-button', u'', u'click')
0.041 (u'tabbar', u'new tab button', u'click')
0.028 (u'bookmarksMenuPopup', u'unknown', u'mouse')
0.014 (u'bookmarks-menu-button', u'bookmarks-menu-button', u'click')
0.013 (u'search engine dropdown', u'menu item', u'click')
0.009 (u'window', u'1', u'window closed')
0.008 (u'searchbar', u'go button', u'click')
0.007 (u'appmenu-button', u'appmenu-button', u'click')
0.007 (u'tabbar', u'drop down menu', u'click')
0.006 (u'', u'unknown', u'mouse')

tab bar (button) 26618
0.328 (u'urlbar', u'url', u'enter key')
0.121 (u'searchbar', u'', u'enter key')
0.106 (u'bookmark toolbar', u'personal bookmark', u'click')
0.091 (u'urlbar', u'most frequently used menu', u'open')
0.084 (u'home-button', u'', u'click')
0.074 (u'urlbar', u'search term', u'enter key')
0.048 (u'tabbar', u'new tab button', u'click')
0.033 (u'bookmarksMenuPopup', u'unknown', u'mouse')
0.016 (u'bookmarks-menu-button', u'bookmarks-menu-button', u'click')
0.014 (u'search engine dropdown', u'menu item', u'click')
0.01 (u'window', u'1', u'window closed')
0.009 (u'searchbar', u'go button', u'click')
0.007 (u'appmenu-button', u'appmenu-button', u'click')
0.007 (u'', u'unknown', u'mouse')
0.005 (u'back-button', u'', u'click')

shortcut 4313
0.629 (u'urlbar', u'url', u'enter key')
0.124 (u'searchbar', u'', u'enter key')
0.106 (u'urlbar', u'search term', u'enter key')
0.041 (u'bookmark toolbar', u'personal bookmark', u'click')
0.022 (u'menus', u'key_newNavigatorTab', u'key shortcut')
0.014 (u'home-button', u'', u'click')
0.013 (u'menus', u'key_close', u'key shortcut')
0.013 (u'search engine dropdown', u'menu item', u'click')
0.007 (u'menus', u'key_search', u'key shortcut')
0.003 (u'menus', u'goHome', u'key shortcut')
0.003 (u'urlbar', u'text selection', u'select')
0.003 (u'urlbar', u'most frequently used menu', u'open')
0.002 (u'menus', u'', u'key shortcut')
0.002 (u'bookmarks-menu-button', u'bookmarks-menu-button', u'click')
0.002 (u'window', u'1', u'window closed')

app menu button 47
0.255 (u'urlbar', u'url', u'enter key')
0.234 (u'appmenu-button', u'appmenu-button', u'click')
0.17 (u'searchbar', u'', u'enter key')
0.085 (u'bookmark toolbar', u'personal bookmark', u'click')
0.064 (u'home-button', u'', u'click')
0.043 (u'searchbar', u'go button', u'click')
0.021 (u'vertical scrollbar', u'slider', u'drag')
0.021 (u'window', u'1', u'window closed')
0.021 (u'contentAreaContextMenu', u'context-paste', u'mouse')
0.021 (u'urlbar', u'most frequently used menu', u'open')
0.021 (u'Panorama', u'Tab View Interface', u'Closed')
0.021 (u'reload-button', u'', u'click')
0.021 (u'bookmarks-menu-button', u'bookmarks-menu-button', u'click')

Comment 12

[dtang]

Study to be run that will also address this question: https://wiki.mozilla.org/Labs/Test_Pilot/New_Tab_Study