1481774 - Collect file_types_changed data in build telemetry

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[(not currently active) Ted Mielczarek]

One of the things we decided would be useful to collect is a list of file extensions in the source directory that were changed since the previous build. The telemetry key I used for this is `file_types_changed`: it's a list of `{ext, count}` like `[{"ext": "cpp", "count": 1}, {"ext": "js", "count": 1}]`.

To collect this data effectively we'll probably need to use watchman and have it running in the background.

Comment 1

[Nick Alexander :nalexander [he/him]]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #0)
> One of the things we decided would be useful to collect is a list of file
> extensions in the source directory that were changed since the previous
> build. The telemetry key I used for this is `file_types_changed`: it's a
> list of `{ext, count}` like `[{"ext": "cpp", "count": 1}, {"ext": "js",
> "count": 1}]`.
> 
> To collect this data effectively we'll probably need to use watchman and
> have it running in the background.

Is there a reason not to keep more information, say... the whole set of modified files?  Are we concerned about people working on files in security-sensitive tickets, or naming files in a way that might leak features?

Comment 2

[(not currently active) Ted Mielczarek]

It felt a little too invasive, honestly. Plus it makes it much more likely that someone could de-anonymize the data by cross-referencing against lists of files from hg. Our biggest use for this data is to answer questions like "are there people doing non-artifact builds that are only touching content files?".

Comment 3

[(not currently active) Ted Mielczarek]

Attachment: add a .watchmanconfig to instruct watchman to ignore VCS directories

watchman's docs say that this should be the default setting, but while testing I did see files under .hg show up in watchman's output, so I tried adding this config file to see if it made a difference. (I'm not sure it did, TBH.)

Comment 4

[(not currently active) Ted Mielczarek]

Attachment: collect file_types_changed in telemetry for `mach build` using watchman

This is very much a WIP, but in local testing it did produce correct output in the simple case.