565470 - Implement new submission specification for Test Pilot 1.0 add-on

Status: VERIFIED FIXED

(Mozilla Labs Graveyard :: Test Pilot, defect, P1)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

Requirements:

The URL should be different from older versions of Test Pilot so there won't be any potential mixing of old data.

The URL should start with an alphanumeric experiment id.
The HTTP request should be a POST
The Content-Type header should be set to application/json

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

The data should be a single JSON blob with two top level attributes, "metadata" and "events".
Metadata should be a dictionary of attributes.
Events should be a two dimensional array of integers. i.e.
  events: [ [0,0,0,0,1235677544], [1,0,0,1,1235677600] ]

We need to determine the best possible location for the mapping of event column names to inner array index.  Should it be in the metadata or maybe not even part of the submission but stored relative to the overall experiment?
Advantage of submission metadata storage is that it makes it easier for the user to understand the data in the client.
Advantage of storage in overall experiment metadata is reduced submission payload size.

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

In the interest of self-contained data, let's say that the headers will be an attribute in the metadata section as follows:

{
  metadata: {
    event_headers: { foo: 0, bar: 1, baz: 2, quz: 3, timestamp: 4 },
    ...(other metadata attributes)
  },
  events: [
    [0,0,0,0,1235677544],
    [1,0,0,1,1235677600]
  ]
}

Comment 3

[Jono Xia]

So the url will be
https://testpilot.mozillalabs.com/submit/<experiment_id>

I will include a preference so that either the old submission spec or the new one can be used during the transitional period.

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

What exactly do you mean by a transitional period?
I was expecting that when 1.0 was released, it would have this URL and format enabled.  It could definitely be useful to change the base URL if we wanted to be able to point it somewhere else, but it would complicate things if we were to try to accept submissions to the old URL in Riak or if we were to try to accept the old format of submissions.

Comment 5

[Jono Xia]

Sorry, by "transitional period" I just meant that while we're still testing the new code and doing the changeover, I would prefer to keep the code for the old format in just in case we need it for whatever; I'm just trying to proceed carefully about this since we're breaking backwards compatibility.  I certainly don't anticipate asking Riak to accept submissions either in the old format or to the old URL.

Comment 6

[Jono Xia]

Partially done in http://hg.mozilla.org/labs/testpilot/rev/ea24f526ea70

Comment 7

[Jono Xia]

Attachment: Example of data upload format as a JSON text file

This is the upload contents produced by Test Pilot, from the Toolbar study, after my latest changes.  If this format looks OK then I'll close the bug.

Comment 8

[Jono Xia]

Latest changes are in http://hg.mozilla.org/labs/testpilot/rev/b874944f1b48 .  I can't test the full upload process until we have the server set up to receive the data submissions, but I believe everything is working to spec now.

Comment 9

[Jono Xia]

Made survey answer uploads use the new upload spec in http://hg.mozilla.org/labs/testpilot/rev/5d68780f8913

Made opt-out messages use the new upload spec in http://hg.mozilla.org/labs/testpilot/rev/e7fd5cf67cc0

Removed code for old-style CSV uploads in http://hg.mozilla.org/labs/testpilot/rev/b493a76d7234

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

Can you point me at docs regarding the opt out thing? Not familiar with it.

Comment 11

[Jono Xia]

There aren't really any docs for it, but a user can opt out of a study at any time; when they do so, they can optionally give us a reason for opting out.  The vast majority of opt-outs won't generate any input so we'll never see them, but when somebody does provide a reason it gets sent to the url:

/submit/opt-out

and contains the following json:

{id: experiment_id,
 reason: "whatever"}

Comment 12

[Jono Xia]

We have now run a study that submit data using the new submission specification and sent it to the new backend, and today I verified that the data was present in the database on the backend server, so I declare this complete.

Comment 13

[Jono Xia]

Also needs to accept standalone survey uploads, which contain json like this:

{"metadata": {},
  "survey_data": {"answers": [], "version_number": "" } }

These survey uploads are currently being rejected.

Comment 14

[Daniel Einspanjer [:dre] [:deinspanjer]]

Jono, can you give me some details of what bucket they are trying to submit to and the response returned by the server?

Ideally, the example POST complete with URL, headers, body, and response would be best.

Comment 15

[Jinghua zhang]

This is my error log when trying to submit a survey (twice):

2010-07-16 18:28:50	TestPilot.Task_beta_perception_survey_1 INFO	Trying to json.parse this: {"answers":[["1"],["0"],["0"],["1"],["7"],["8"],["0"],["nothing"],["nothing"],["0"],[]],"version_number":1}
2010-07-16 18:28:53	TestPilot.Survey     INFO	Storing survey answers for survey id beta_perception_survey_1
2010-07-16 18:28:53	TestPilot.Survey     INFO	Storing survey answers {"answers":[["1"],["0"],["0"],["1"],["7"],["8"],["0"],["nothing"],["nothing"],["0"],[]]}
2010-07-16 18:28:53	TestPilot.Task_beta_perception_survey_1 WARN	ERROR POSTING DATA: 
2010-07-16 18:28:54	TestPilot.Survey     INFO	Storing survey answers for survey id beta_perception_survey_1
2010-07-16 18:28:54	TestPilot.Survey     INFO	Storing survey answers {"answers":[["1"],["0"],["0"],["1"],["7"],["8"],["0"],["nothing"],["nothing"],["0"],[]]}
2010-07-16 18:28:54	TestPilot.Task_beta_perception_survey_1 WARN	ERROR POSTING DATA:

Comment 16

[Daniel Einspanjer [:dre] [:deinspanjer]]

'k,  I'll need some details from Jono still though, the actual URL it is trying to post to, and the HTTP response the client receives.

Comment 17

[Jono Xia]

POST url = https://testpilot.mozillalabs.com/submit/beta_perception_survey_1
Headers: "Content-type" = "application/json"
"Content-length" = "1207"
"Connection" = "close"

POST body is 
{"metadata":{"extensions":[snip],"accessibilities":[snip],"location":"en-US","fxVersion":"4.0b1","operatingSystem":"Darwin Intel Mac OS X 10.5","tpVersion":"1.0rc1","surveyAnswers":{snip},"task_guid":""},"survey_data":{"answers":[snip],"version_number":1}}

Response is code 201
No body text

Comment 18

[Jono Xia]

Oh.  201 means "Created", doesn't it.  Maybe we are actually seeing success here.  I think the problem is on my end.

Comment 19

[Jono Xia]

Attachment: Patch to make 200, 201, and 202 all accepted success codes for upload

This patch fixes the problem with uploading surveys by accepting 201 as a success code (previously it was only accepting 200, but the new backend sends back 201.)  I would like this to be reviewed for inclusion in Fx4beta2.

Comment 20

[Dave Townsend [:mossop]]

Landed in Firefox trunk: http://hg.mozilla.org/mozilla-central/rev/d595418f0a77

Comment 21

[Tony Chung [:tchung]]

Verifiex fix on Mozilla/5.0 (Windows; Windows NT 6.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2.   Survey pages submit successfully.