Closed
Bug 1264697
Opened 8 years ago
Closed 8 years ago
Investigate a more compact format for all-tests.json
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(firefox48 fixed)
RESOLVED
FIXED
mozilla48
Tracking | Status | |
---|---|---|
firefox48 | --- | fixed |
People
(Reporter: chmanchester, Assigned: chmanchester)
References
Details
Attachments
(1 file)
MozReview Request: Bug 1264697 - Change the format of all-tests.json to reduce redundant data. r=gps
58 bytes,
text/x-review-board-request
|
gps
:
review+
|
Details |
It is extremely large. Writing it accounts for a significant portion of build-backend, and loading/parsing it accounts for a significant overhead when running tests.
Assignee | ||
Comment 1•8 years ago
|
||
Ok, the simple optimization I had in mind saves about 22mb on disk, so that seems promising.
Assignee | ||
Comment 2•8 years ago
|
||
Unfortunately, while the speed improvements that result (measuring the operations in comment 0) are measurable, they are nowhere near proportional to the space improvement.
Assignee | ||
Comment 3•8 years ago
|
||
The format provided to the build system by the manifest parser is highly redundant: every test lists all variables for that test, and many tests use a large support-files entry in DEFAULT that ends up in individual test objects. This patch stores these DEFAULTS per-manifest rather than per-test to save disk space, resulting in about a ~22mb smaller all-tests.json file. The in-memory representation of tests is not changed by this patch, as the defaults are again propagated to individual tests as all-tests.json is read by the test resolver. Review commit: https://reviewboard.mozilla.org/r/46559/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/46559/
Attachment #8741545 -
Flags: review?(gps)
Comment 4•8 years ago
|
||
each test also has a path relative to topsrcdir as well as a full absolute path.
Comment 5•8 years ago
|
||
Comment on attachment 8741545 [details] MozReview Request: Bug 1264697 - Change the format of all-tests.json to reduce redundant data. r=gps https://reviewboard.mozilla.org/r/46559/#review43453 ::: python/mozbuild/mozbuild/frontend/emitter.py:1083 (Diff revision 1) > + default_support_files = defaults.get('support-files') > + if default_support_files: > + obj.default_support_files = default_support_files You can just assign directly since .get() will default to returning None, which is the default value in the data class. ::: python/mozbuild/mozbuild/testing.py:66 (Diff revision 1) > + manifest = metadata['manifest'] > + support_files = manifest_support_files.get(manifest) > + if support_files and 'support-files' not in metadata: > + metadata['support-files'] = support_files Please add a short inline comment about the new file format.
Attachment #8741545 -
Flags: review?(gps) → review+
Comment 7•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/70bfda59e76f
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox48:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Updated•6 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•