Brian noticed that the `fission-test-info` artifacts don't match up with the spreadsheets listing the skipped tests manually. From his e-mail: ``` I started on this per-component graph and realized the numbers I’m seeing from the artifacts don’t line up with what I see on the spreadsheet which is tracking individual tests (https://docs.google.com/spreadsheets/d/1kjp32JTuB4axM3wKx0iIYL2Ado-HcyeBhoY-siGxYAs/edit#gid=1560718888). For example, on that spreadsheet I see 600+ rows with not too many crossed out. But if I load the artifact from yesterday (https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.pushdate.2019.09.11.latest.source.test-info-fission/artifacts/public/test-info-fission.json) I’m seeing only 198 skipped tests and 27 failed tests. Any ideas what’s going on with the discrepancy here? Is the artifact collecting incomplete data, is the spreadsheet out of date, or am I just misreading it? ``` To test this out, I ran `./mach test-info` both with a build and without and noticed the results are different. I believe this is happening because certain `moz.build` files are only read based on build conditions. E.g: https://searchfox.org/mozilla-central/source/toolkit/toolkit.mozbuild#182 So if any of those directories define test manifests, the test resolver (which `mach test-info` depends on) won't pick them up. So the "simple" solution would be to get the `test-info-fission` tasks to run configure before running `./mach test-info`. But there are two downsides here: A) It will need to use the build image, making it more expensive and slower to run. B) Even with a buildconfig, it might be missing manifests. E.g, here is another condition: https://searchfox.org/mozilla-central/source/toolkit/themes/moz.build#21 So even if we do run `./configure` we'll never consider every `moz.build` file, which means we might be missing out on manifests. Luckily, the build system has something called "file-system mode". This essentially means that instead of following the `DIRS` variables, moz.build traversal will instead simply look for `moz.build` files in the file-system. This mode is designed to pick-up every moz.build file regardless of build config for cases like this. So I propose we add a `file_system_traversal=False` flag to the test resolver: https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py And make sure `./mach test-info` uses it. I can look into switching the `test-info-fission` tasks over to the build image in the meantime.
Bug 1580914 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Brian noticed that the `fission-test-info` artifacts don't match up with the spreadsheets listing the skipped tests manually. From his e-mail: >I started on this per-component graph and realized the numbers I’m seeing from the artifacts don’t line up with what I see on the spreadsheet which is tracking individual tests (https://docs.google.com/spreadsheets/d/1kjp32JTuB4axM3wKx0iIYL2Ado-HcyeBhoY-siGxYAs/edit#gid=1560718888). > >For example, on that spreadsheet I see 600+ rows with not too many crossed out. But if I load the artifact from yesterday (https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.pushdate.2019.09.11.latest.source.test-info-fission/artifacts/public/test-info-fission.json) I’m seeing only 198 skipped tests and 27 failed tests. > >Any ideas what’s going on with the discrepancy here? Is the artifact collecting incomplete data, is the spreadsheet out of date, or am I just misreading it? To test this out, I ran `./mach test-info` both with a build and without and noticed the results are different. I believe this is happening because certain `moz.build` files are only read based on build conditions. E.g: https://searchfox.org/mozilla-central/source/toolkit/toolkit.mozbuild#182 So if any of those directories define test manifests, the test resolver (which `mach test-info` depends on) won't pick them up. So the "simple" solution would be to get the `test-info-fission` tasks to run configure before running `./mach test-info`. But there are two downsides here: A) It will need to use the build image, making it more expensive and slower to run. B) Even with a buildconfig, it might be missing manifests. E.g, here is another condition: https://searchfox.org/mozilla-central/source/toolkit/themes/moz.build#21 So even if we do run `./configure` we'll never consider every `moz.build` file, which means we might be missing out on manifests. Luckily, the build system has something called "file-system mode". This essentially means that instead of following the `DIRS` variables, moz.build traversal will instead simply look for `moz.build` files in the file-system. This mode is designed to pick-up every moz.build file regardless of build config for cases like this. So I propose we add a `file_system_traversal=False` flag to the test resolver: https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py And make sure `./mach test-info` uses it. I can look into switching the `test-info-fission` tasks over to the build image in the meantime.