1076875 - local build time metrics

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

We'd like to collect local build time metrics from developers, and aggregate them, and report them as one of our KPI metrics.

We'll most likely need to collect some local system information at the same time, so I expect this should be an opt-in option in mach or something.

Greg has already thought a lot about this, so I'll let him fill in some more details, or point to other bugs that may already exist.

Comment 1

[Gregory Szorc [:gps]]

Companies like Facebook and Twitter monitor and record pretty much every interaction people have with their internal tools. They are recording things like execution time of mach commands, build times, command line arguments to mach, etc. This information is invaluable to identify performance issues, developer pain points, and to target areas for improvement.

Here are snippets of email correspondence I had in March this year:

Me to a few folks on the Privacy front:
---

We talked at Summit (and coincidentally earlier today) about having the
Firefox build system collect and report usage data back to Mozilla so we
may use this data to see what people are doing so we can target
improvement areas. I wanted to follow up with you about that.

Can you provide me some rough guidance on what getting this past privacy
review would look like? Here's my rough proposal:

* On first use of the build system, the user sees a notice that usage
info may be anonymously reported to Mozilla. They have the opportunity
to opt out by typing "no" or some such.

* The user has the opportunity to provide his or her name and contact
info so Mozilla may follow up with them regarding oddities, suggested
tips, etc.

* Initially, we'll collect basic system stats (essentially what FHR and
Telemetry do), some metrics gathered from the build system (times, CPU,
disk, and memory use).

* I'd also like to collect source tree stats. Build revision (it's just
a hash), Mercurial vs Git, that sort of thing.

* Over time, I'd also like to count invocations of certain behavior.
e.g. "performed a full build with mach" or "performed a partial build
with mach." We may even expand this to capture even more details. e.g.
"mach build browser". The goal here is to learn how people are using the
tools we have so we can identify best/worst practices, establish clearer
messaging, identify popular areas for improvement, make data-driven
decisions about deprecations, notify people about changes relevant to
them, etc. I assume new collections would go through the lightweight
privacy review process similar to Telemetry.

* The collection framework will collect to-be-submitted data and will
automatically bulk submit this data (if allowed) to a Mozilla-operated
server somewhere (likely in EC2 for ease-of-deployment) over SSL.

Please let me know how we can move this forward.

------------

There was some discussion about the default setting of the submission part. I want it enabled and anonymous by default. The natural position of privacy-minded people is to have it opt-in. But I think enabled by default is necessary to have a reflective sample set.

There was also talk of capturing IP addresses (seeing how office connectivity would impact things like Mercurial clone times). We also talked about making the system enabled by default for Mozilla employees (identified by network signature, email address in hgrc, etc).

But these are mostly implementation details that need to be worked out with Privacy. If we reduce scope to collecting anonymous information from users that have opted in, I was told that would likely get little pushback.

The following high-level tasks need to happen for this:

1) Privacy review
2) Someone to stand up a server that ingests data
3) Someone to code the mach bits that collect and submit data

All of those need owners. I don't think we should write a line of code until #1 is done.