576482 - If a survey changes, TP attempts to apply your old survey answers to the new survey questions

Status: RESOLVED FIXED

(Mozilla Labs Graveyard :: Test Pilot, defect, P1)

Description

[Ed Morley [:emorley]]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6

After updating from Test Pilot 1.0b4 to 1.0rc1, some of the answers to the previously completed "User Background Survey" had been altered to incorrect values and no notification given to retake the new version of the survey (which has now been renamed "Firefox 4 Beta User Background Survey" and appears to have more questions now). 

Reproducible: Didn't try

Steps to Reproduce:
1. Complete the user background survey using an older version of Test Pilot.
2. Update from Test Pilot 1.0b4 to 1.0rc1.
3. Open the Test Pilot add-on and view your responses to the newly renamed "Firefox 4 Beta User Background Survey", comparing them with the answers you gave previously.
Actual Results:  
No prompt given to retake the survey or even any indication that answers had been changed/new questions available.

The following existing answers were changed:
- "How long have you used firefox" - used to be at the maximum setting, it's now at "1-2 years"
- "What are the main reasons you use the web?" - all my answers have disappeared.

The following new questions have already had boxes ticked even though I didn't set them:
- "What smartphone device are you currently using?" - I believe this is a new question - and for some reason "apple iphone", "Microsoft Windows Mobile", "Palm" and "RIM" are ticked already, even though I don't use those devices and so wouldn't have ticked them myself.

Expected Results:  
- Existing answers remain on previous correct values or else are cleared completely, so the user knows to answer them again.
- User informed of new survey questions and prompted to take the survey again.

Comment 1

[Ed Morley [:emorley]]

Seeing as all Fx4 beta 1 users will have this extension installed and that beta 1 has just been released, surely this is going to start affecting quite a few users?

Comment 2

[Jono Xia]

The way to fix this is:
Thanks for reporting this, Ed.  This happened because we updated the survey to change the questions.  Test Pilot had stored the numbers corresponding to your stored answers, and when it loaded the updated survey it filled in the answers with the same numbers... but those were no longer the same answers.

Here's what we need to do to fix this:

1. When you hit the button to save your survey answers, it should record the *current* survey version number along with those answers.

2. When submitting, it should submit the survey version that you answered.  So if there are multiple versions of a survey that have been released, the analyzers of the data will know which version each set of answers came from.

3. If you open a survey, and it's a newer version than the one you have saved answers for (that's what happened to Ed) then it should not fill in any of your old answers -- it should treat this like a survey you haven't taken yet.

Comment 3

[Jono Xia]

Fixed in http://hg.mozilla.org/labs/testpilot/rev/41f59cc938da

Comment 4

[Tracy Walker [:tracy]]

verified with recent build TP using updated Week in the life of a browser (v2) study

Comment 5

[Jono Xia]

Attachment: Patch fixing the bug

Dave, could you please review this patch and merge it into Feedback?  Thanks!

Comment 6

[Dave Townsend [:mossop]]

Landed in mozilla-central: http://hg.mozilla.org/mozilla-central/rev/339dae0c8e3b