Split generated webgl-conf mochitest manifest into per-cluster files
Categories
(Core :: Graphics: CanvasWebGL, task)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox153 | --- | fixed |
People
(Reporter: florian, Assigned: florian)
References
Details
Attachments
(2 files)
The webgl2-core and webgl2-ext mochitest jobs (along with webgl1-core, webgl1-ext, webgl2-deqp) share a single generated manifest, dom/canvas/test/webgl-conf/generated-mochitest.toml, that lists all ~3800 wrappers. The jobs differ only by a --subsuite= filter applied at runtime. Dynamic chunking operates at manifest granularity, so with a single manifest there is nothing to distribute and these jobs can't use it - they're stuck on static chunking. This change is the prerequisite for switching them to chunks: dynamic.
The generator (generate-wrappers-and-manifest.py) now emits ~93 smaller manifests grouped by (subsuite, path cluster) under dom/canvas/test/webgl-conf/generated/, alongside a generated moz.build that's pulled in from dom/canvas/moz.build via DIRS += ["test/webgl-conf/generated"]. Cluster sizes are capped via a tree-split on the wrapper path components, merging tiny siblings into _misc buckets.
Other cleanups:
subsuitemoved into each manifest's[DEFAULT]section instead of repeating per entry.support-fileslists changed from enumerating every file incheckout/(5200+ entries) to a small set of per-cluster globs (e.g.../checkout/conformance/textures/**, plus the shared../checkout/js/**and../checkout/resources/**).
Per-cluster support-files mean that when an engineer runs ./mach test <name> locally, the harness only copies the files relevant to that test's cluster from src dir to obj dir, instead of the entire checkout/ tree.
|
Assignee | |
Comment 1•13 days ago
|
||
The webgl1-core, webgl1-ext, webgl2-core, webgl2-ext and webgl2-deqp
mochitest jobs share a single generated manifest listing all ~3800
wrappers. The jobs differ only by a --subsuite= filter applied at
runtime. Dynamic chunking operates at manifest granularity, so with
a single manifest there is nothing to distribute and these jobs are
stuck on static chunking. This change is the prerequisite for moving
them to chunks: dynamic.
The generator now emits ~93 smaller manifests grouped by (subsuite,
path cluster) under dom/canvas/test/webgl-conf/generated/, alongside
a generated moz.build that's pulled in from dom/canvas/moz.build via
DIRS += ["test/webgl-conf/generated"]. Cluster sizes are capped via
a tree-split on the wrapper path components, merging tiny siblings
into _misc buckets.
Other cleanups in the same pass:
- subsuite moved into each manifest's [DEFAULT] section instead of
being repeated per entry. - support-files lists no longer enumerate every file in checkout/
(5200+ entries per manifest); instead each manifest declares a
small set of per-cluster globs. This means ./mach test <name>
locally only copies the files relevant to that test's cluster
from src dir to obj dir, instead of the entire checkout/ tree.
|
|
Assignee | |
Comment 2•13 days ago
|
||
Now that the webgl-conf manifest is split into per-cluster files, the
chunker has enough atomic units to distribute. Switch the four webgl
jobs that were on static chunking to chunks: dynamic, matching the
existing webgl2-deqp configuration. default-chunks is set to each
job's prior static count (1 for webgl1-core, webgl1-ext, webgl2-core;
4 for webgl2-ext) so behavior is unchanged when historical runtime
data isn't available; dynamic chunking will scale from there based on
the manifests' observed runtimes.
Also adds chunked: true to the mozharness block for webgl1-core and
webgl1-ext, which is required once chunks > 1 is possible.
https://hg.mozilla.org/mozilla-central/rev/3d23c2e5c75e
https://hg.mozilla.org/mozilla-central/rev/8458bebaa65c
Description
•