1575213 - Create a code coverage optimizer

Status: ASSIGNED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

Now that we have the ability to run "shadow schedulers", we should create an optimizer that tries to use code coverage information.

Marco has set up a database that exists as an artifact here.

This is a sqlite database that has a table called file_to_chunk which is a mapping of "source file" -> "test task". Our optimizer can download this, check if any of the modified files can be mapped to test tasks, and remove any tasks that aren't in the list.

Much of the logic we'll need is already implemented in the try coverage selector. Once we fix this bug, we should refactor that selector to import the optimizer from taskgraph and use that.

Comment 1

[Andrew Halberstadt [:ahal]]

I'll upload my WIP patch soon, but there are a few problems:

1. The DB doesn't have sufficient platform information. The schema is files,platform,chunk and each row looks something like this:
   js/src/vm/TraceLogging.cpp, windows, jittest-1proc-5

The only two platforms in the DB are windows and linux. But there are tons of different configurations under these. Are they debug? Asan? QR? The chunking is going to be different under each configuration.

2. It looks like only compiled test suites and marionette based suites exist in the DB. I don't see any mochitest/reftest/wpt references anywhere. This isn't really a blocker to getting the shadow-scheduler landed, but it will severely limit the usefulness of this optimizer.

3. Known issue, but since the DB only updates once a day, tests may get shuffled to new chunks in the interim. We could A) use a more stable chunking algorithm than chunk-by-runtime, and/or B) figure out how to schedule tasks by test path more reliably in taskgraph itself. Definitely think we should do B) but that could be a follow-up to this bug.

Comment 2

[Joel Maher ( :jmaher ) (UTC-4)]

to answer the question, we only have code coverage data for windows/linux- it is the ccov config, which is basically debug, but slightly different.

We should have clear mappings for wpt, mochitest, xpcshell (maybe reftest). I don't think we have it for the compiled tests and marionette, that seems to be opposite of what I would expect.

Comment 3

[Andrew Halberstadt [:ahal]]

Ok, if it's the ccov config this data won't be very useful to us then. As the chunks from there will be different than the chunks on the configs we'd like to run. We're basically back to the same old problem we always run up against: how to schedule by test path on autoland (without breaking backfill).

Comment 4

[Marco Castelluccio [:marco]]

(In reply to Andrew Halberstadt [:ahal] from comment #1)

I'll upload my WIP patch soon, but there are a few problems:

1. The DB doesn't have sufficient platform information. The schema is files,platform,chunk and each row looks something like this:
   js/src/vm/TraceLogging.cpp, windows, jittest-1proc-5

The only two platforms in the DB are windows and linux. But there are tons of different configurations under these. Are they debug? Asan? QR? The chunking is going to be different under each configuration.

We need to make a call here. Do we run on all configurations? Do we run only under one configuration? Do we use something like SETA to decide the configuration?
This applies in general to any coverage-based scheduler (at chunk level, or manifest level, or test level), as the different configurations are often redundant.

2. It looks like only compiled test suites and marionette based suites exist in the DB. I don't see any mochitest/reftest/wpt references anywhere. This isn't really a blocker to getting the shadow-scheduler landed, but it will severely limit the usefulness of this optimizer.

I thought I fixed this, but I guess I didn't :)

3. Known issue, but since the DB only updates once a day, tests may get shuffled to new chunks in the interim. We could A) use a more stable chunking algorithm than chunk-by-runtime, and/or B) figure out how to schedule tasks by test path more reliably in taskgraph itself. Definitely think we should do B) but that could be a follow-up to this bug.

Ok, if it's the ccov config this data won't be very useful to us then. As the chunks from there will be different than the chunks on the configs we'd like to run. We're basically back to the same old problem we always run up against: how to schedule by test path on autoland (without breaking backfill).

These two are ultimately the same root problem.

As an interim solution, could we schedule all chunks containing the tests that are in the ccov chunk?
That is:

1. Get list of covered chunks from file_to_chunk;
2. Get list of tests from covered chunks using chunk_to_test;
3. Build list of chunks to run from the list of tests.

Is 3 feasible? We'd possibly be running more tests than necessary, but at least we'd have a working solution.