Closed Bug 1335197 Opened 4 years ago Closed 4 years ago

quantum graphics builds and tests on m-c

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kmoir, Assigned: kmoir)

References

Details

From dev-tree-management

kgupta's email

For the time being the set of extra jobs is pretty small. Right now on
the graphics branch the extra jobs that we are doing for QR are:
- linux64 QR builds with cppunit, gtest, jittest, reftest,
reftest-no-accel tests
- macosx64 QR builds
- win64 QR builds

Each of the above is run in debug and opt versions. At a minimum I'd
like to run all of this on m-c (so ~3-4 pushes a day or so). Running
the build jobs on the integration branches and making them tier-1
would be nice, but I'd be ok with just leaving them as tier-2 on m-c,
because generally I don't think they will break without breaking other
builds that are already covered on the integration branches. So I
guess it's a question of how much the above-listed set of jobs costs
and if it's ok to run it on m-c. Note that the set will probably
expand slowly as we bring more tests online.

However, longer-term I guess it would be good to figure out the
testing strategy for QR. I anticipate that we'll remove the
compile-time switch in the not-too-distant future, and leave a runtime
switch instead. That will eliminate the "build" jobs. However we'll
still want to keep test coverage of both QR and non-QR codepaths, much
like we have e10s and non-e10s test coverage. We can probably limit it
to test suites that exercise the graphics stack more heavily, but it
will probably still be a nontrivial amount of extra infra load. I
don't know what to do about that, but I'm open to suggestions.
From mailing list again, conversation between gps and kgupta

> This is basically what I did for Stylo in bug 1318200. Extending that
> solution to work for QR should be trivial. Although it appears from [2] that
> most of that work is already done on the graphics repo, just without build
> peer review.

Yeah, I made changes to the in-tree taskcluster configs in the
graphics branch that was roughly modeled after the stylo taskcluster
changes. I had to go a bit beyond stylo to get the win64 and macosx64
builds as well but the changes are roughly analogous.

> So, uh, you may want to verify your automation code matches
> what is used for Stylo before submitting that review request because if it
> doesn't, it will likely get r- and the review will be "make it more like
> what Stylo does."

I'm expecting that we might have to make minor tweaks to the
taskcluster configs based on the review comments but hopefully nothing
major. (And even if they are major, it's a one-time thing, I can deal
with it.)

> When you say "linux64 QR test jobs," what exactly does that mean? Is there a
> new test harness for a new class of tests? Or, is this running a large
> subset of existing tests just with QR enabled? If so, which ones? (If we're
> running a variation of existing tests, we could run into capacity concerns.)

I just mean the existing test jobs that we've enabled to run on the
linux64 QR builds. No new harness. The tests we have enabled so far
can be seen at https://hg.mozilla.org/projects/graphics/file/12c5788aa7a0/taskcluster/ci/test/test-sets.yml#l72
(or take a look at the graphics branch on TH).

>> As QR builds would be tier-1, they would need to run on all
>> integration branches (including m-i and autoland), but I think QR test
>> jobs could be limited to running on m-c, to reduce the load on our
>> infra. Right now there's not a lot of QR test jobs anyway but we're
>> actively greening up more test suites and getting them running on the
>> QR builds.
>
>
> As Lawrence said, capacity could be problematic. Generally speaking, it is
> easier to scale Linux than Windows than OS X.

So far we're only running on tests on Linux because TaskCluster isn't
really working for the other desktop platforms. (At least not that I'm
aware of).

> And, builds are easier to
> scale than tests because the number of machine hours spent running tests
> dwarfs time running builds. What I'm trying to say is that this may
> ultimately result with a tiered rollout where some platforms are higher
> tiered than others for capacity reasons.

That seems fair. We can prioritize test suites so that e.g. Linux runs
a larger set and Windows runs just the really graphics-critical ones.
But at the moment I have my hands full with just getting the Linux
suites turned on, so adding jobs on other platforms is probably at
least a month away.

>> If you're interested in more details on how we would go about doing
>> the merge, please see [2]. Any comments or suggestions are welcome.
>>
>
> My understanding of the development model on the graphics repo is that many
> of the commits were throwaway. If this is indeed the case and the history
> doesn't provide much value, I'd seriously consider introducing a single,
> non-merge changeset in mozilla-central for this "import" instead of
> polluting history with value-less changesets.

There aren't actually that many throwaway commits in the graphics
mercurial repo. Prior to using the graphics branch, we were doing
development on github, and that *did* have a lot of throwaway commits.
When I migrated the code from github to the graphics branch I squashed
it all into four commits (see bug 1317774). At this point we have ~320
non-merge commits on the graphics branch that would get merged over to
m-c, and I feel like the history there is mostly pretty useful to
have.

and further note from gps

This set doesn't seem too bad. reftests look to have the largest cost. The
capacity impact from those tests will be noticeable, but it's a lot better
than say if mochitests were also being run.

Out of curiosity, is there a way to limit tests within suites to just those
impacted by QR? For example, there are tons of xpcshell tests not at all
related to anything Quantum. Test manifests support "tags" and it is
possible to configure automation to only run tests having a certain flag.
In a TaskCluster world, I would entertain the idea of splitting various
suites into separate jobs grouped by tags. This will soon be technically
achievable by integrating the TaskCluster "decision" task / task group with
build system metadata.
Assignee: nobody → kmoir
These have been running for a while
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.