1242017 - Set up endpoint for receiving mach build metrics

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P2)

Description

[Rob Miller [:rmiller]]

The build team wants to gather and process data that is generated by the Mach tool, so they can measure and track build times and other relevant developer metrics / pain points. We can start with the minimum viable set up, providing them an HTTP endpoint to which they can POST their JSON data, which we will route to an S3 bucket from which they can retrieve said data. First iteration doesn't require us to parse or crack open the JSON at all. It's okay for the data to go into S3 as a Heka message stream, I (i.e. rmiller) will work with them to make sure they have the tooling they need to convert the Heka message stream back into the original JSON for their own processing.

Comment 1

[Katie Parlante]

Scheduled for the next sprint so it can make Q1

Comment 2

[Thomas Huelbert]

Due to e10s and ops related fires, we are unable to address this ticket during this sprint.

Comment 3

[Gregory Szorc [:gps]]

Per IRL chat today, we'd like to revive this request.

Initially, we're looking to do some rapid prototyping on the type of data we send. So we're probably looking for a single endpoint that accepts N different message types with initially no schema validation. The data will almost certainly be JSON.

Once we have the client-side bits implemented and have confidence in the data we're sending, we can formalize a schema "productionize" the ingestion. How that switchover works, I'm not sure. It probably involves 2 endpoints (e.g. "stage" vs "production") or some kind of routing key in the HTTP request. Not sure what options are available.

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #3)
 > Initially, we're looking to do some rapid prototyping on the type of data we
> send. So we're probably looking for a single endpoint that accepts N
> different message types with initially no schema validation. The data will
> almost certainly be JSON.
> 
> Once we have the client-side bits implemented and have confidence in the
> data we're sending, we can formalize a schema "productionize" the ingestion.
> How that switchover works, I'm not sure. It probably involves 2 endpoints
> (e.g. "stage" vs "production") or some kind of routing key in the HTTP
> request. Not sure what options are available.

Greg: do we have any further information about the format/content of data to be submitted via |mach telemetry| after 9 months?

Comment 5

[Gregory Szorc [:gps]]

(In reply to Chris Cooper [:coop] from comment #4)
> Greg: do we have any further information about the format/content of data to
> be submitted via |mach telemetry| after 9 months?

No. The best we have is the mach resource-usage.json file that is produced during builds. But that lacks metadata like what mach command was used, CPU count, etc. These are things we'd almost certainly want in mach telemetry.

At some point, someone just needs to hack up a PoC for what to capture. Or we can set up the endpoint to allow ingestion of anything until we get our act together.

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #5)
> (In reply to Chris Cooper [:coop] from comment #4)
> > Greg: do we have any further information about the format/content of data to
> > be submitted via |mach telemetry| after 9 months?
> 
> No. The best we have is the mach resource-usage.json file that is produced
> during builds. But that lacks metadata like what mach command was used, CPU
> count, etc. These are things we'd almost certainly want in mach telemetry.
> 
> At some point, someone just needs to hack up a PoC for what to capture. Or
> we can set up the endpoint to allow ingestion of anything until we get our
> act together.

Ted has signed up to do the PoC as a deliverable this quarter. Laura was quite interested in this when I spoke to her yesterday.

Comment 7

[Chris Cooper [:coop] (he/him)]

Wes: any update on this, or the generic ingestion service in general? Do you have a new ETA?

Comment 8

[Wesley Dawson [:whd]]

I spec'd out the service requirements with :mreid and :jason, which resulted in the creation of bug #1368197 and bug #1368196, which I'm adding as blockers. Unfortunately I've not had time to do the development work on this as I've been focused on Mission Control. I would expect to next be able to work on this, priority permitting, in mid to late July, and depending on whether the development work has been done by then, that should take a week or a few weeks.

Comment 9

[Chris Cooper [:coop] (he/him)]

I'm not seeing any updates in the new dependencies...has this been de-prioritized?

Comment 10

[Wesley Dawson [:whd]]

I expect to start working on this within the next two weeks.

Comment 11

[Chris Cooper [:coop] (he/him)]

Any update here? We've punted on this for 2 quarters now.

Comment 12

[Wesley Dawson [:whd]]

This infrastructure is (finally) available. The process for adding a namespace isn't documented and will probably change to be more self-service, but the current instructions are to file a PR against https://github.com/mozilla-services/mozilla-pipeline-schemas similar to https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/104/files with JSON and parquet schemas for your data. This PR unfortunately also has some extra telemetry-specific diffs; you should only need to add schema files to e.g. schemas/mach and templates/mach for your ping types.

Once that's merged and deployed, you should be able to POST your JSON blobs to e.g. https://incoming.telemetry.mozilla.org/submit/mach/<doctype>/<docversion>/<docid> and have the result accessible in stmo and atmo.

Comment 13

[Mark Reid [:mreid]]

Greg, are you able to start working out a schema for the Mach submissions? I'm happy to help you with that, just let me know.

Comment 14

[Kim Moir [:kmoir] ET]

I talked to mcote about this and we decided to de-prioritize this work for a few weeks until we have a better handle on the hypothesis we want to test before collecting this data.

Comment 15

[Gregory Szorc [:gps]]

Per kmoir, I guess there's nothing to do right this moment. But thanks for the offer to help with a schema!

Comment 17

[Wesley Dawson [:whd]]

I'm going to call this fixed, in that the infrastructure is available and using it is now documented at https://docs.telemetry.mozilla.org/cookbooks/new_ping.html. The meta bugs this bug is associated with are tracking the actual data generation.