1019788 - Server-side ping collection for update hotfix logging

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Benjamin Smedberg]

Want to stand up a dirt-simple collection system for pings from the addon hotfix. The payload will be JSON, and given the fairly low expected volume I think we should just partition by ping date.

mreid or jonasfj can you take this? We're hoping to have this working and deployed by the end of next week. This is intended to be a temporary service for now, running less than 8 weeks.

Comment 1

[(dormant account)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> Want to stand up a dirt-simple collection system for pings from the addon
> hotfix. The payload will be JSON, and given the fairly low expected volume I
> think we should just partition by ping date.
> 
> mreid or jonasfj can you take this? We're hoping to have this working and
> deployed by the end of next week. This is intended to be a temporary service
> for now, running less than 8 weeks.

Mark can do this in a few hours. Try to have the clientside ready asap so we can do some interop testing and move on.

Comment 2

[Gregory Szorc [:gps]]

Tell me what the HTTP API will be and I will have the client side ready.

Comment 3

[Mark Reid [:mreid]]

The API will be a POST to 
https://<hostname tbd>.telemetry.mozilla.org/submit/hotfix/<docid>

Where the body contains your json payload (either gzipped or plain-text are acceptable, just be sure to include a content-length header)

Server response code will be 200 with a message of "OK" if all went well.
Other possible responses:
405 if you use an incorrect request type
404 if you hit an unexpected endpoint
411 if you don't include the content-length header
202 if your request is too large (we don't want to trigger client-side retries for oversized requests. Currently I've set the limit for this endpoint to 400k)
202 if your request path is too long (currently set to 10k)
500 if you hit an internal error

Submissions flow through to an Amazon S3 bucket (a different bucket than "normal" telemetry data per bsmedberg on IRC), where files will be partitioned only by submission date (not by channel, version, buildid, etc as telemetry submissions are). Each file will be lzma-compressed, and each line in the file will be of the form
<docid><tab><payload>

So be sure not to put any tab characters in your docids. I recommend using UUIDs for docids.

You will be able to consume the data using the mapreduce framework used for the telemetry data, though you will need AWS credentials to access the new data bucket in S3.

What sort of volumes are you expecting? Are the above limits acceptable for payload size and path length?

Comment 4

[Mark Reid [:mreid]]

Per conversation on IRC there is a test server up and running in AWS EC2, and :gps has the details for sending requests.

Comment 5

[Gregory Szorc [:gps]]

I'm able to send data to the test server. At this point, the biggest concern from my perspective is having a final URL to plug in.

Comment 6

[Mark Reid [:mreid]]

(In reply to Gregory Szorc [:gps] from comment #5)
> I'm able to send data to the test server. At this point, the biggest concern
> from my perspective is having a final URL to plug in.

How does "hotfix.telemetry.mozilla.org" sound as a final name?

Comment 7

[Mark Reid [:mreid]]

Ok, "hotfix.telemetry.mozilla.org" is live - please test that everything is working as expected.

Comment 8

[Gregory Szorc [:gps]]

I have the code pointing to the new URL. https://hotfix.telemetry.mozilla.org/submit/hotfix/ (port 443). Upload appears to work just fine.

Comment 9

[Mark Reid [:mreid]]

The server side of this is ready to receive submissions. Data will end up in an S3 bucket called 'hotfix-published-v1'.