Closed Bug 1264697 Opened 3 years ago Closed 3 years ago

Investigate a more compact format for all-tests.json

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox48 --- fixed

People

(Reporter: cmanchester, Assigned: cmanchester)

References

Details

Attachments

(1 file)

It is extremely large. Writing it accounts for a significant portion of build-backend, and loading/parsing it accounts for a significant overhead when running tests.
Ok, the simple optimization I had in mind saves about 22mb on disk, so that seems promising.
Unfortunately, while the speed improvements that result (measuring the operations in comment 0) are measurable, they are nowhere near proportional to the space improvement.
The format provided to the build system by the manifest parser is highly redundant:
every test lists all variables for that test, and many tests use a large
support-files entry in DEFAULT that ends up in individual test objects. This
patch stores these DEFAULTS per-manifest rather than per-test to save disk
space, resulting in about a ~22mb smaller all-tests.json file. The
in-memory representation of tests is not changed by this patch, as the defaults
are again propagated to individual tests as all-tests.json is read by the test
resolver.

Review commit: https://reviewboard.mozilla.org/r/46559/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/46559/
Attachment #8741545 - Flags: review?(gps)
each test also has a path relative to topsrcdir as well as a full absolute path.
Comment on attachment 8741545 [details]
MozReview Request: Bug 1264697 - Change the format of all-tests.json to reduce redundant data. r=gps

https://reviewboard.mozilla.org/r/46559/#review43453

::: python/mozbuild/mozbuild/frontend/emitter.py:1083
(Diff revision 1)
> +            default_support_files = defaults.get('support-files')
> +            if default_support_files:
> +                obj.default_support_files = default_support_files

You can just assign directly since .get() will default to returning None, which is the default value in the data class.

::: python/mozbuild/mozbuild/testing.py:66
(Diff revision 1)
> +                        manifest = metadata['manifest']
> +                        support_files = manifest_support_files.get(manifest)
> +                        if support_files and 'support-files' not in metadata:
> +                            metadata['support-files'] = support_files

Please add a short inline comment about the new file format.
Attachment #8741545 - Flags: review?(gps) → review+
https://hg.mozilla.org/mozilla-central/rev/70bfda59e76f
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.