If a survey changes, TP attempts to apply your old survey answers to the new survey questions

RESOLVED FIXED in 1.1

Status

Mozilla Labs Graveyard
Test Pilot
P1
normal
RESOLVED FIXED
8 years ago
2 years ago

People

(Reporter: emorley, Assigned: Christopher Jung)

Tracking

unspecified
x86_64
Windows 7
Bug Flags:
in-testsuite ?
in-litmus ?

Details

(Whiteboard: data)

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6

After updating from Test Pilot 1.0b4 to 1.0rc1, some of the answers to the previously completed "User Background Survey" had been altered to incorrect values and no notification given to retake the new version of the survey (which has now been renamed "Firefox 4 Beta User Background Survey" and appears to have more questions now). 

Reproducible: Didn't try

Steps to Reproduce:
1. Complete the user background survey using an older version of Test Pilot.
2. Update from Test Pilot 1.0b4 to 1.0rc1.
3. Open the Test Pilot add-on and view your responses to the newly renamed "Firefox 4 Beta User Background Survey", comparing them with the answers you gave previously.
Actual Results:  
No prompt given to retake the survey or even any indication that answers had been changed/new questions available.

The following existing answers were changed:
- "How long have you used firefox" - used to be at the maximum setting, it's now at "1-2 years"
- "What are the main reasons you use the web?" - all my answers have disappeared.

The following new questions have already had boxes ticked even though I didn't set them:
- "What smartphone device are you currently using?" - I believe this is a new question - and for some reason "apple iphone", "Microsoft Windows Mobile", "Palm" and "RIM" are ticked already, even though I don't use those devices and so wouldn't have ticked them myself.

Expected Results:  
- Existing answers remain on previous correct values or else are cleared completely, so the user knows to answer them again.
- User informed of new survey questions and prompted to take the survey again.
(Reporter)

Comment 1

8 years ago
Seeing as all Fx4 beta 1 users will have this extension installed and that beta 1 has just been released, surely this is going to start affecting quite a few users?
(Reporter)

Updated

8 years ago
Severity: normal → critical

Updated

8 years ago
Assignee: nobody → jdicarlo
Priority: -- → P1
Target Milestone: -- → 1.1

Updated

8 years ago
Severity: critical → blocker
Status: UNCONFIRMED → NEW
Ever confirmed: true

Updated

8 years ago
Status: NEW → ASSIGNED

Updated

8 years ago
Assignee: jdicarlo → cjung

Updated

8 years ago
Severity: blocker → normal
Flags: in-testsuite?
Flags: in-litmus?
Target Milestone: 1.1 → 1.3

Comment 2

7 years ago
The way to fix this is:
Thanks for reporting this, Ed.  This happened because we updated the survey to change the questions.  Test Pilot had stored the numbers corresponding to your stored answers, and when it loaded the updated survey it filled in the answers with the same numbers... but those were no longer the same answers.

Here's what we need to do to fix this:

1. When you hit the button to save your survey answers, it should record the *current* survey version number along with those answers.

2. When submitting, it should submit the survey version that you answered.  So if there are multiple versions of a survey that have been released, the analyzers of the data will know which version each set of answers came from.

3. If you open a survey, and it's a newer version than the one you have saved answers for (that's what happened to Ed) then it should not fill in any of your old answers -- it should treat this like a survey you haven't taken yet.
Summary: Test pilot silently changed existing answers in User Background Survey when upgrading to 1.0rc1 → If a survey changes, TP attempts to apply your old survey answers to the new survey questions

Updated

7 years ago
Whiteboard: data
Target Milestone: 1.3 → 1.1

Comment 3

7 years ago
Fixed in http://hg.mozilla.org/labs/testpilot/rev/41f59cc938da
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
verified with recent build TP using updated Week in the life of a browser (v2) study
Status: RESOLVED → VERIFIED

Comment 5

7 years ago
Created attachment 500433 [details] [diff] [review]
Patch fixing the bug

Dave, could you please review this patch and merge it into Feedback?  Thanks!
Attachment #500433 - Flags: review?(dtownsend)

Updated

7 years ago
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Attachment #500433 - Flags: review?(dtownsend) → review+
Landed in mozilla-central: http://hg.mozilla.org/mozilla-central/rev/339dae0c8e3b
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Product: Mozilla Labs → Mozilla Labs Graveyard
You need to log in before you can comment on or make changes to this bug.