1419703 - generate an artifact from the bugzilla source job to indicate which files are test files

Status: ASSIGNED

(Testing :: General, enhancement, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I would love to know which files are test cases (to answer how many test cases do I have in my component).  In addition knowing which files are test support files are good as for code coverage we care about source file coverage and knowing that we have 231000 total files in tree, but <60000 have any coverage it is concerning- knowning that 114606 files are test related, makes the total covered files less concerning.

We have data in tree right now to build a list of all test related filed- we should generate that as an artifact so we can start being smarter.  To split out the test cases, possibly we can do that with a regex, or maybe we need to parse manifests or maybe add more meta data into moz.build files.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: generate testfiles.json artifact from bugzilla job

first attempt at this- not very optimal and just quickly hacked up.  While this appears to work just fine, I am not sure if I am using the data properly- if this is r+ territory, please switch from feedback to review, I suspect there are other great ideas here or something obvious I am overlooking.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

here is a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7174c3c9affe073c4f02efcd46f809212fda2bc2

Comment 3

[Gregory Szorc [:gps]]

Comment on attachment 8930849 [details] [diff] [review]
generate testfiles.json artifact from bugzilla job

Review of attachment 8930849 [details] [diff] [review]:
-----------------------------------------------------------------

This seems pretty reasonable.

FWIW, that "test_files" attribute only captures the results of IMPACTED_TESTS: it isn't reading test manifests. I'm not sure if you want to call this out somewhere.

::: python/mozbuild/mozbuild/frontend/mach_commands.py
@@ +188,5 @@
>          missing_component = set()
>          seen_components = set()
>          component_by_path = {}
> +        test_paths = []
> +        test_files = []

I think these should be sets. Then you can just insert items without testing for membership first.

@@ +211,5 @@
> +        for p in component_by_path:
> +            for tp in test_paths:
> +                path = tp
> +                if tp.endswith('**'):
> +                    path = tp.strip('**')

This can just be `path = tp.rstrip('**')`

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: update components.json to be extensible and contain a field describing the type of file

now we will have a field for 'source' and 'test', we could eventually add 'testcase', 'third_party', 'data/binary', 'precompiled', etc.

This does change the format of components.json- I do know that we consume components.json in activedata and can change our ingestion to be the new format.

If desired I could create a new format and we can build on that.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

for reference, here is a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cd5ee8bbb7f311699cbffc60b4d59996dd2106c

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8933854 [details] [diff] [review]
update components.json to be extensible and contain a field describing the type of file

Review of attachment 8933854 [details] [diff] [review]:
-----------------------------------------------------------------

Please write a new output file. We don't know who is consuming these files and need to maintain backwards compatibility.

At the point you do that, we now have 3 output formats. It is probably best to move the logic for producing the data structure to its own function so the @Command method doesn't get any more polluted.