We'd like to collect local build time metrics from developers, and aggregate them, and report them as one of our KPI metrics. We'll most likely need to collect some local system information at the same time, so I expect this should be an opt-in option in mach or something. Greg has already thought a lot about this, so I'll let him fill in some more details, or point to other bugs that may already exist.
Companies like Facebook and Twitter monitor and record pretty much every interaction people have with their internal tools. They are recording things like execution time of mach commands, build times, command line arguments to mach, etc. This information is invaluable to identify performance issues, developer pain points, and to target areas for improvement. Here are snippets of email correspondence I had in March this year: Me to a few folks on the Privacy front: --- We talked at Summit (and coincidentally earlier today) about having the Firefox build system collect and report usage data back to Mozilla so we may use this data to see what people are doing so we can target improvement areas. I wanted to follow up with you about that. Can you provide me some rough guidance on what getting this past privacy review would look like? Here's my rough proposal: * On first use of the build system, the user sees a notice that usage info may be anonymously reported to Mozilla. They have the opportunity to opt out by typing "no" or some such. * The user has the opportunity to provide his or her name and contact info so Mozilla may follow up with them regarding oddities, suggested tips, etc. * Initially, we'll collect basic system stats (essentially what FHR and Telemetry do), some metrics gathered from the build system (times, CPU, disk, and memory use). * I'd also like to collect source tree stats. Build revision (it's just a hash), Mercurial vs Git, that sort of thing. * Over time, I'd also like to count invocations of certain behavior. e.g. "performed a full build with mach" or "performed a partial build with mach." We may even expand this to capture even more details. e.g. "mach build browser". The goal here is to learn how people are using the tools we have so we can identify best/worst practices, establish clearer messaging, identify popular areas for improvement, make data-driven decisions about deprecations, notify people about changes relevant to them, etc. I assume new collections would go through the lightweight privacy review process similar to Telemetry. * The collection framework will collect to-be-submitted data and will automatically bulk submit this data (if allowed) to a Mozilla-operated server somewhere (likely in EC2 for ease-of-deployment) over SSL. Please let me know how we can move this forward. ------------ There was some discussion about the default setting of the submission part. I want it enabled and anonymous by default. The natural position of privacy-minded people is to have it opt-in. But I think enabled by default is necessary to have a reflective sample set. There was also talk of capturing IP addresses (seeing how office connectivity would impact things like Mercurial clone times). We also talked about making the system enabled by default for Mozilla employees (identified by network signature, email address in hgrc, etc). But these are mostly implementation details that need to be worked out with Privacy. If we reduce scope to collecting anonymous information from users that have opted in, I was told that would likely get little pushback. The following high-level tasks need to happen for this: 1) Privacy review 2) Someone to stand up a server that ingests data 3) Someone to code the mach bits that collect and submit data All of those need owners. I don't think we should write a line of code until #1 is done.