597560 - Firefox 4 Beta Study v2: Tracking Bug

Status: NEW

(Mozilla Labs Graveyard :: Test Pilot Studies, defect)

Description

[Jono Xia]

We are releasing a new version of the Firefox 4 beta study.  The study code is in the hg.mozilla.org/labs/testpilotweb repo, in the file testcases/beta/combined.js

On the web: http://hg.mozilla.org/labs/testpilotweb/raw-file/066aeec111f5/testcases/beta/combined.js

Highest priority features to be added to this version, over the previous version, are:

 * Sync instrumentation: How many people have configured sync, and how recently did they sync up?
 * Panorama instrumentation: Observing when people click the sync button or hit the keyboard shortcut; recording how many tab groups they have and how many tabs per group.
 * App tab instrumentatin: Observing when someone makes a tab into an app tab or back, and counting how many app tabs people have.

Secondary priorities are the items in red on the tracking spreadsheet at https://spreadsheets.google.com/ccc?key=0AnJReXqM7IBPdGNnM0xyOGVJXzFydS1ScERGNUJvU1E&hl=en#gid=0

Things for testers and code reviewers to watch out for:

- Are all events recorded correctly on every operating system?

- Are we throwing any exceptions for conditions that should be caught and handled more gracefully - for example, attempting to register listeners on interface elements that don't happen to exist on the current OS or build

- Have we kept up with all the changes to the Firefox 4 interface in the last couple of betas - do all our observers that used to work, still work?

- Do we record UI customizations correctly, and do the observers still work right no matter how someone has customized their UI?

- Are we correctly recording the difference between using something by clicking the toolbar, clicking the menu, hitting the keyboard shortcut, and using a context menu?

- Are there any ways we haven't considered yet to get the "missing" UI interactions - for example, the keyboard shortcuts for cut/copy/paste, which do not appear to generate any command or key events on the main command set or main menu bar.

Comment 1

[Jono Xia]

More things for testers and code reviewers to look at:

All users with any of the 4.0 betas should be getting and running this test, whether or not they ran the Beta Interface Study round 1.
(Round 1 had id = 100; round 2 has id = 101.)

If a user submit both round 1 and round 2, they should have the same GUID attached to both submissions, for longitudinal comparison purposes.  (This works by having the round 2 study look at the GUID stored in prefs for round 1, and copying this over its own GUID).

Comment 2

[Jono Xia]

Attachment: The code for the study itself.

Comment 3

[Jono Xia]

Attachment: The code for the beta interface study

Updated version of the study code itself to match up with latest revision in Hg.

Comment 4

[Atul Varma [:atul]]

Comment on attachment 479101 [details]
The code for the beta interface study

I assume I am reviewing this, so setting r? to me.

Comment 5

[Atul Varma [:atul]]

Comment on attachment 479101 [details]
The code for the beta interface study

My review comments are here:

http://github.com/toolness/my-code-reviews/commit/83f34cf9b5c3a0aaa61f5840a4e11f29c7e4fcc2

Feel free to reply to the comments on that page.

Comment 6

[Jono Xia]

Some changes to the code made in http://hg.mozilla.org/labs/testpilotweb/rev/58f97868d104  and http://hg.mozilla.org/labs/testpilotweb/rev/b70add898bda

1. now recording tab groupings when user closes panorama interface
2. now correctly recording sub-menu items of windows appmenu button

Comment 7

[Jono Xia]

Attachment: The code for the study itself.

Attached the latest version of the study code with the latest bug fixes.

Comment 8

[Atul Varma [:atul]]

Comment on attachment 481533 [details]
The code for the study itself.

All my review comments so far are here:

http://github.com/toolness/my-code-reviews/pull/1

Comment 9

[Jinghua zhang]

https://bugzilla.mozilla.org/show_bug.cgi?id=577386