Closed Bug 1335197 Opened 4 years ago Closed 4 years ago
quantum graphics builds and tests on m-c
From dev-tree-management kgupta's email For the time being the set of extra jobs is pretty small. Right now on the graphics branch the extra jobs that we are doing for QR are: - linux64 QR builds with cppunit, gtest, jittest, reftest, reftest-no-accel tests - macosx64 QR builds - win64 QR builds Each of the above is run in debug and opt versions. At a minimum I'd like to run all of this on m-c (so ~3-4 pushes a day or so). Running the build jobs on the integration branches and making them tier-1 would be nice, but I'd be ok with just leaving them as tier-2 on m-c, because generally I don't think they will break without breaking other builds that are already covered on the integration branches. So I guess it's a question of how much the above-listed set of jobs costs and if it's ok to run it on m-c. Note that the set will probably expand slowly as we bring more tests online. However, longer-term I guess it would be good to figure out the testing strategy for QR. I anticipate that we'll remove the compile-time switch in the not-too-distant future, and leave a runtime switch instead. That will eliminate the "build" jobs. However we'll still want to keep test coverage of both QR and non-QR codepaths, much like we have e10s and non-e10s test coverage. We can probably limit it to test suites that exercise the graphics stack more heavily, but it will probably still be a nontrivial amount of extra infra load. I don't know what to do about that, but I'm open to suggestions.
From mailing list again, conversation between gps and kgupta > This is basically what I did for Stylo in bug 1318200. Extending that > solution to work for QR should be trivial. Although it appears from  that > most of that work is already done on the graphics repo, just without build > peer review. Yeah, I made changes to the in-tree taskcluster configs in the graphics branch that was roughly modeled after the stylo taskcluster changes. I had to go a bit beyond stylo to get the win64 and macosx64 builds as well but the changes are roughly analogous. > So, uh, you may want to verify your automation code matches > what is used for Stylo before submitting that review request because if it > doesn't, it will likely get r- and the review will be "make it more like > what Stylo does." I'm expecting that we might have to make minor tweaks to the taskcluster configs based on the review comments but hopefully nothing major. (And even if they are major, it's a one-time thing, I can deal with it.) > When you say "linux64 QR test jobs," what exactly does that mean? Is there a > new test harness for a new class of tests? Or, is this running a large > subset of existing tests just with QR enabled? If so, which ones? (If we're > running a variation of existing tests, we could run into capacity concerns.) I just mean the existing test jobs that we've enabled to run on the linux64 QR builds. No new harness. The tests we have enabled so far can be seen at https://hg.mozilla.org/projects/graphics/file/12c5788aa7a0/taskcluster/ci/test/test-sets.yml#l72 (or take a look at the graphics branch on TH). >> As QR builds would be tier-1, they would need to run on all >> integration branches (including m-i and autoland), but I think QR test >> jobs could be limited to running on m-c, to reduce the load on our >> infra. Right now there's not a lot of QR test jobs anyway but we're >> actively greening up more test suites and getting them running on the >> QR builds. > > > As Lawrence said, capacity could be problematic. Generally speaking, it is > easier to scale Linux than Windows than OS X. So far we're only running on tests on Linux because TaskCluster isn't really working for the other desktop platforms. (At least not that I'm aware of). > And, builds are easier to > scale than tests because the number of machine hours spent running tests > dwarfs time running builds. What I'm trying to say is that this may > ultimately result with a tiered rollout where some platforms are higher > tiered than others for capacity reasons. That seems fair. We can prioritize test suites so that e.g. Linux runs a larger set and Windows runs just the really graphics-critical ones. But at the moment I have my hands full with just getting the Linux suites turned on, so adding jobs on other platforms is probably at least a month away. >> If you're interested in more details on how we would go about doing >> the merge, please see . Any comments or suggestions are welcome. >> > > My understanding of the development model on the graphics repo is that many > of the commits were throwaway. If this is indeed the case and the history > doesn't provide much value, I'd seriously consider introducing a single, > non-merge changeset in mozilla-central for this "import" instead of > polluting history with value-less changesets. There aren't actually that many throwaway commits in the graphics mercurial repo. Prior to using the graphics branch, we were doing development on github, and that *did* have a lot of throwaway commits. When I migrated the code from github to the graphics branch I squashed it all into four commits (see bug 1317774). At this point we have ~320 non-merge commits on the graphics branch that would get merged over to m-c, and I feel like the history there is mostly pretty useful to have. and further note from gps This set doesn't seem too bad. reftests look to have the largest cost. The capacity impact from those tests will be noticeable, but it's a lot better than say if mochitests were also being run. Out of curiosity, is there a way to limit tests within suites to just those impacted by QR? For example, there are tons of xpcshell tests not at all related to anything Quantum. Test manifests support "tags" and it is possible to configure automation to only run tests having a certain flag. In a TaskCluster world, I would entertain the idea of splitting various suites into separate jobs grouped by tags. This will soon be technically achievable by integrating the TaskCluster "decision" task / task group with build system metadata.
See Also: → 1335748
No longer blocks: 1335455
These have been running for a while
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.