1237610 - Gather and report local build metrics

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Dan Minor [:dminor]]

We would like to gather and report build metrics from local developer's machines to help identify problems and quantify progress in making the build system faster. Because of privacy implications, this system will be opt-in and will not collect PII (at least initially.)

Data we're considering collecting:
    * mach command (or at least build target)
    * machine characteristics - processor, ram, drive type, os
    * build type - full, incremental, or files affected (or a count of files affected)
    * elapsed time
    * CPU usage (samples, by second?)
    * Disk usage (samples)
    * Memory usage (samples)
    * Swap usage (samples)

We already collect resource usage data and store it in resource_usage.json during a build, so we might be able to reuse that.

Our initial plan is to report this data to Telemetry.

Comment 1

[Gregory Szorc [:gps]]

This is on me now.

Comment 2

[(not currently active) Ted Mielczarek]

Attachment: bug 1237610 - make mach telemetry opt-out rather than opt-in.

This also changes the submission endpoint to a server that exists. The
previous dev server no longer exists. This is expected to be temporary
until the data platform team's generic ingestion service is available.

Review commit: https://reviewboard.mozilla.org/r/149928/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/149928/

Comment 3

[(not currently active) Ted Mielczarek]

Until the generic ingestion service is available we're going to continue using ekyle's server. It stores data in a private s3 bucket, which should be good enough for now. This patch updates the address of the server and also flips submission from opt-in to opt-out. I'll post about this on dev-platform before landing.

Once this lands we should be able to start looking at some of the data and we can figure out if we need more information to answer the questions we'd like to ask.

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8878612 [details]
bug 1237610 - make mach telemetry opt-out rather than opt-in.

https://reviewboard.mozilla.org/r/149928/#review155250

The code seems correct. But this gets a r- for policy.

I'm reasonably confident that we can't collect data from client installs without at least prompting the end user that this occurs. This would likely violate Mozilla data collection policies and probably some laws. We need sign-off from someone authorized to make such a sign-off. And I reckon that sign-off won't occur until we have some kind of notice that data collection occurs.

Also, submitting to an IP address via http:// is sketchy. Can we not get https:// or a hostname (I have access to some Mozilla-related DNS zones hosted on AWS that we could use)? Of course, https:// presents its own challenges, such as validating the certificate - which isn't guaranteed to work on all Python installs :/

Comment 5

[(not currently active) Ted Mielczarek]

Ah, I thought we had sorted out the privacy issues of submitting as opt-out, thanks for the info! We'll have to figure out what the right way to handle that is, then.

RE: SSL, good call. If you have a hostname to use I'm sure we can point it at ekyle's server and get a Let's Encrypt cert configured. I'll file a bug for that.

Comment 6

[(not currently active) Ted Mielczarek]

As I was writing a comment about the privacy issue, I realized the simplest way to do this might just be to put a prompt in `mach bootstrap`.

Comment 7

[Gregory Szorc [:gps]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> Ah, I thought we had sorted out the privacy issues of submitting as opt-out,
> thanks for the info! We'll have to figure out what the right way to handle
> that is, then.

I think I had a verbal agreement that a proposal I came up sounded reasonable. Not sure if AutomatedTester or Coop ever got something more formal.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #7) 
> I think I had a verbal agreement that a proposal I came up sounded
> reasonable. Not sure if AutomatedTester or Coop ever got something more
> formal.

If we want to wait for the generic ingestion service, we're covered, but if we want to collect the data before that service is ready on our own custom server, we'll need to address the concerns around PII and retention.

I'll grab bug 1291053 and start driving it.

Comment 9

[Chris Cooper [:coop] (he/him)]

We're still waiting on bug 1242017, so gps has setup a new, temporary ingestion point. See https://bugzilla.mozilla.org/show_bug.cgi?id=1374380#c5 for host details.

Comment 10

[Kim Moir [:kmoir] ET]

In our workweek last week, we decided that these are the initial characteristics we are looking to capture from the build telemetry work

What mach commands are they running? Mach build is most important
Basic timing information - time to run mach build
Basic hardware info - cpu brand string, memory, type of disk
Files were changed through invocation - mach watchman
Configure flags: Artifact build or not
Debug vs opt
Exception code if applicable 
Sequence of commands
Persistent client id
Sccache and icecream usage

Comment 11

[Connor Sheehan [:sheehan]]

A quick update from the data ingestion side: I have submitted a pull request of the updated JSON schema to the mozilla-pipeline-schemas repository at https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/191. I had an issue getting tests to pass against the Parquet schema format and after speaking with :frank in #datapipeline, I was told to submit the pull request with a comment asking for help finding the error. I am waiting for a review on that PR.

Comment 12

[(not currently active) Ted Mielczarek]

Attachment: bug 1237610 - slight cleanup in should_skip_dispatch. r?build

MozReview-Commit-ID: HJLO82QZQVO

Comment 13

[(not currently active) Ted Mielczarek]

Attachment: bug 1237610 - don't call post_dispatch_handler when using debug-command. r?build

MozReview-Commit-ID: 4UG6RH4b6tZ

Comment 14

[(not currently active) Ted Mielczarek]

Attachment: bug 1237610 - use a mach setting to control telemetry submission. r?build

Comment 15

[(not currently active) Ted Mielczarek]

Attachment: bug 1237610 - Collect telemetry data matching the new schema. r?build

This patch rewrites `gather_telemetry` to collect data matching the new schema.
This includes all required fields and most of the optional fields. Some fields
are not currently recorded and followup bugs have been filed to track their
implementation.

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 9005006 [details]
bug 1237610 - slight cleanup in should_skip_dispatch. r?build

Gregory Szorc [:gps] has approved the revision.

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 9005007 [details]
bug 1237610 - don't call post_dispatch_handler when using debug-command. r?build

Gregory Szorc [:gps] has approved the revision.

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 9005008 [details]
bug 1237610 - use a mach setting to control telemetry submission. r?build

Gregory Szorc [:gps] has approved the revision.

Comment 19

[Connor Sheehan [:sheehan]]

Attachment: Bug 1237610: update `submit_telemetry_data.py` r?ted

This commit updates submit_telemetry_data.py to send data
to the Telemetry pipeline. The script assumes the presence
of a "telemetry" directory within the statedir, and an
"outgoing" directory within the "telemetry" directory (otherwise
there is no data to submit). The script will create a
"submitted" directory and "telemetry.log" file if absent,
making the assumption that this is the first build telemetry
submission for that user. UUID values for submitted data points
are seeded from the filename, without the ".json" suffix.

Comment 20

[(not currently active) Ted Mielczarek]

Comment on attachment 9008480 [details]
Bug 1237610: update `submit_telemetry_data.py` r?ted

Ted Mielczarek [:ted] [:ted.mielczarek] has approved the revision.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ce05cf6d5e19
update `submit_telemetry_data.py` r=ted

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 9005009 [details]
bug 1237610 - Collect telemetry data matching the new schema. r?build

Gregory Szorc [:gps] has approved the revision.

Comment 23

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/ce05cf6d5e19

Comment 24

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a413b8fa5c7f3b12b7161782d2b774e96120a78e
bug 1237610 - move build system telemetry collection code to a common place. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/61de9d8b0bb22a0e95dac87b297f727f15e4f346
bug 1237610 - slight cleanup in should_skip_dispatch. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/328a9f04072daa04708d44ee729f54291e983691
bug 1237610 - don't call post_dispatch_handler when using debug-command. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/9eed32a2c7f7965aa11e25da68d8eb714982b05b
bug 1237610 - use a mach setting to control telemetry submission. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/1b85b45460a36022d99cba049a92635fa8ffd1da
bug 1237610 - Collect telemetry data matching the new schema. r=gps

Comment 25

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/a413b8fa5c7f
https://hg.mozilla.org/mozilla-central/rev/61de9d8b0bb2
https://hg.mozilla.org/mozilla-central/rev/328a9f04072d
https://hg.mozilla.org/mozilla-central/rev/9eed32a2c7f7
https://hg.mozilla.org/mozilla-central/rev/1b85b45460a3