(In reply to Andrew Halberstadt [:ahal] from comment #1)
I'll upload my WIP patch soon, but there are a few problems:
- The DB doesn't have sufficient platform information. The schema is
files,platform,chunk and each row looks something like this:
js/src/vm/TraceLogging.cpp, windows, jittest-1proc-5
The only two platforms in the DB are
linux. But there are tons of different configurations under these. Are they debug? Asan? QR? The chunking is going to be different under each configuration.
We need to make a call here. Do we run on all configurations? Do we run only under one configuration? Do we use something like SETA to decide the configuration?
This applies in general to any coverage-based scheduler (at chunk level, or manifest level, or test level), as the different configurations are often redundant.
- It looks like only compiled test suites and marionette based suites exist in the DB. I don't see any mochitest/reftest/wpt references anywhere. This isn't really a blocker to getting the shadow-scheduler landed, but it will severely limit the usefulness of this optimizer.
I thought I fixed this, but I guess I didn't :)
- Known issue, but since the DB only updates once a day, tests may get shuffled to new chunks in the interim. We could A) use a more stable chunking algorithm than
chunk-by-runtime, and/or B) figure out how to schedule tasks by test path more reliably in
taskgraph itself. Definitely think we should do B) but that could be a follow-up to this bug.
Ok, if it's the ccov config this data won't be very useful to us then. As the chunks from there will be different than the chunks on the configs we'd like to run. We're basically back to the same old problem we always run up against: how to schedule by test path on autoland (without breaking backfill).
These two are ultimately the same root problem.
As an interim solution, could we schedule all chunks containing the tests that are in the ccov chunk?
- Get list of covered chunks from file_to_chunk;
- Get list of tests from covered chunks using chunk_to_test;
- Build list of chunks to run from the list of tests.
Is 3 feasible? We'd possibly be running more tests than necessary, but at least we'd have a working solution.