536491 - All Studies: Record current study version number into event log every time study starts up

Status: RESOLVED FIXED

(Mozilla Labs Graveyard :: Test Pilot, defect, P1)

Description

[Jono Xia]

Right now, we have experiment fixes being applied partway through the run of an experiment.  That means when we get the upload, the first half of the data in the log is pre-bugfix, the second half of the log is post-bugfix, but the overall experiment version number is the post-bugfix number.  This makes it hard to identify data that came in after the bug fix.  It would be better to make a clean break, by simply having the extension core not apply updates to already-running experiments.

Comment 1

[Jono Xia]

This is closely related to bug 533858.

Comment 2

[Jono Xia]

I've got a better solution to this, actually.  Every experiment needs to record its own version number each time it starts up (therefore its data storage schema needs to include an event code for 'STUDY_STATUS' or something like that, but this is a small price to pay.)  Then when we are analyzing data we can look not only at the *current* version number in the metadata, but also look at the STUDY_STATUS events in the file to see the most recent version number.  If, for instance, we fixed a bug in version 4 of the study, then we can do our analysis using ONLY the parts of each file that come AFTER the first "STUDY_STATUS version=4" record.

Changing name of the bug to reflect this.

Has to be implemented in each study, NOT in the core.

Comment 3

[Jono Xia]

Done for week in the life study in http://hg.mozilla.org/labs/testpilotweb/rev/55ef20e98a29

Comment 4

[Jono Xia]

Done for tab study in http://hg.mozilla.org/labs/testpilotweb/rev/14337561b066