520206 - Already submitted studies generate a new submit notification after restart

Status: RESOLVED WONTFIX

(Mozilla Labs Graveyard :: Test Pilot, defect, P1)

Description

[Dietrich Ayala (:dietrich)]

each time i restart the browser (trunk nightly) i get reprompted for the tab study being complete.

Comment 1

[Jono Xia]

Did you submit the tab study results?  Or did you click the link that says "quit the study"?

If you didn't do either, than it's working as designed.  If you did submit or quit the study, can you tell me what value you have for the preference extensions.testpilot.taskstatus.1 ?

Comment 3

[Jono Xia]

Boriss reported the same thing:

"I submitted my data weeks ago, but still see the speech-bubble-shaped
notification pointing to the status bar about once a day."

Comment 4

[Jono Xia]

Has also been observed to happen with the Week-in-the-life study, so it's not specific to the Tabs study.

Comment 5

[Jono Xia]

I can't manage to reproduce this right now.  I had a profile that seemed to be doing it, but when I went back to it and clicked submit everything worked correctly.

Comment 6

[Jono Xia]

Neither I nor the contractors can reproduce this on the latest code, so I'm closing this bug as 'worksforme' until such time as another user reports it.

Comment 7

[Jono Xia]

I just talked to somebody who still had this problem (with week-in-the-life and menu-item-usage studies, after starting up with a profile that hadn't been used in a while).  I managed to snag a zipped copy of her profile directory, so I am re-opening this bug and I will attempt to replicate it using that profile.

Comment 8

[Jono Xia]

The profile shows that the studies have no data present, and that they are at status 4 (finished).  I can submit them and everything works fine, so whatever happened to cause this, the profile doesn't help in reproducing it.  It also provides no clue as to whether the studies were previously at status 6 (submitted) and got reset back to status 4, or if so, how.

What would be really useful is writing errors and debugging info to a logfile which we could then get users to send to us.  (See bug 515670 ).

A possible explanation:  If the user clicked "submit" but then upload failed, then a timer would have been set up to repeatedly try to upload the data.  If the user restarted Firefox before the upload succeeded, then after the restart the task will still be in status 4, the timer will not be running, and the user will again be prompted to submit.

If this is what happened, then that would actually be working as designed; possibly a poor design choice but not an unexplained bug.

However, this still wouldn't explain why the data for the offending studies in the user's profile is all blank.  That would seem to imply that dataStore.wipeAllData() was called at some point, which is only supposed to happen *after* a successful submit or an opt-out, both of which would have changed the status code.

Comment 9

[Jono Xia]

We've had a couple more people report this problem now, and this time they have error logs to send us!  See attachments.

Comment 10

[Jono Xia]

Attachment: Test Pilot error log file from a user who had this bug.

This user had redundant notifications to submit data for a finished study which he had already submitted.

Comment 11

[Jono Xia]

Attachment: Test Pilot error log file from a user who had this bug.

This user had multiple redundant "there is a new study" notifications for the same study.

Comment 12

[Raymond Lee [:raymondlee]]

A possible reason why user had multiple redundant "there is a new study" notifications for the same study is that we don't keep a list of displayed notifications.

_notifyUserOfTasks in the setup.js is called by a timer every X seconds and is also called when a task state is changed.  Since we don't keep a list of displayed notifications, the first (same) notification on the taskList always gets displayed.  This can be solved by storing a list of displayed notifications and check against it before showing a notification.

However, this still doesn't explain why the user had redundant notifications to submit data for a finished study which he had already submitted. Do you know which study did the notification come from?

Comment 13

[Raymond Lee [:raymondlee]]

Attachment: Patch for this bug.

The patch keeps a record of displayed notifications to prevent showing redundant notifications for the same study.

Comment 14

[Jono Xia]

I don't want to add a list of displayed notifications because we already have the study STATUS codes which are supposed to handle all that kind of stuff.  Rather than adding an additional subsystem, if we want to prevent it displaying again we should separate STATUS_FINISHED into STATUS_FINISHED and STATUS_FINISHED_AND_NOTIFIED.  Only notify if in the former state, and advance to the latter state after notifying.

However, I don't think that's the right thing either.  What I think is going on with this bug is that the user submits, the submission fails for whatever reason, so the timer keeps retrying it, but Firefox gets shut down without the retry timer ever succeeding, so the next time Firefox starts up the study is still at STATUS_FINISHED and the user gets notified again.  In other words, it's working as (badly) designed.

What I propose doing is changing the semantics of STATUS_SUBMITTED a little bit.  Instead of going to SUBMITTED only after the upload is a confirmed success, go to SUBMITTED as soon as the user has clicked Submit.  We keep retrying on failure, but if Firefox shuts down without the submission having succeeded, then we either abandon the data or we set a flag to try uploading it again next time Firefox starts up.  Either way, the status is no longer FINISHED so the user no longer will get notified.

If we take that approach and people still get redundant notifications for the same study, then we know there's another bug somewhere.

If we take that approach, we need to think about what happens if you click Submit when you don't have a network connection, because it's pretty obvious that everything is going to fail then.  If we can detect whether testpilot.mozillalabs.com is reachable, then we can simply delay showing the "submit your results" notification at times when it is not.

Comment 15

[Jinghua zhang]

(In reply to comment #14)
  Instead of going to SUBMITTED only after the upload is a confirmed
> success, go to SUBMITTED as soon as the user has clicked Submit.  We keep
> retrying on failure, but if Firefox shuts down without the submission having
> succeeded, then we either abandon the data or we set a flag to try uploading it
> again next time Firefox starts up.  Either way, the status is no longer
> FINISHED so the user no longer will get notified.

// i think it is a right approach. Users should only get feedback based on their intended action, and we can take care of the rest. 
> 
> If we take that approach, we need to think about what happens if you click
> Submit when you don't have a network connection, because it's pretty obvious
> that everything is going to fail then.  If we can detect whether
> testpilot.mozillalabs.com is reachable, then we can simply delay showing the
> "submit your results" notification at times when it is not.

// I like this design too. This may related to https://bugzilla.mozilla.org/show_bug.cgi?id=557220 and https://bugzilla.mozilla.org/show_bug.cgi?id=570581

Comment 16

[Jinghua zhang]

the same thing happens with Survey too:

"Test Pilot Survey for Firefox 4 Beta annoying and won't go away. I took the survey 2 times, and it's still there??"

"I took the pilot test survey and then it popped up again the next time I opened firefox."

"I filled out a test pilot survey for FF, saved it; then a little while later, FF hung, restarted it, and then it asked the same survey again"

Comment 17

[Adam Muntner [:adamm] (use NEEDINFO)]

decommissioned