Closed Bug 1568638 Opened 5 years ago Closed 5 years ago

Ability to query skipped tests from manifests

Categories

(Testing :: General, enhancement, P1)

Product:
Testing
Component:
General
Version:
Version 3
Type:
enhancement

Tracking

(firefox70 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla70
Tracking Flags:
Tracking Status
firefox70 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(4 files, 1 obsolete file)

fission-report.json
41.92 KB, application/json
Details
Bug 1568638 - [test-info] Create --output-file directory if it doesn't exist, r?gbrown
47 bytes, text/x-phabricator-request
Details | Review
Bug 1568638 - [build] Create objdir in test_manifest backend if it doesn't exist
47 bytes, text/x-phabricator-request
Details | Review
Bug 1568638 - [ci] Add a task to generate test manifest metadata for fission
47 bytes, text/x-phabricator-request
Details | Review

The fission team is currently triaging their tests in preparation for enabling fission on mozilla-central. They would like to be able to see a snapshot of all the tests that are disabled with fission across all the various platforms and configurations. This is basically what Test Informant used to do.

I think we could write an ActiveData recipe that grabs this information (the old test-informant used to use AD already). But Joel also mentioned that :gbrown might have something (or be working on something) that might help..

Geoff, could you explain what Joel was talking about? Do you know if it would help for this scenario?

Flags: needinfo?(gbrown)

Yes, we are specifically looking to query for the skip-if(s) and fail-if(s) with fission, split by bug component, and collected in a json format.

I have been reviving the work in bug 1404472, in hopes of eventually providing a meaningful summary of the tests associated with a group of bugzilla components; I think it might be beneficial if a bz triage owner could track data like "how many tests are in my components?", "how many of those tests are skipped?" etc. See bug 1519171 for a better idea of my motivations.

The work-in-progress patch on bug 1404472 generates a report of the test manifests associated with each bugzilla component and includes a count of tests found in each manifest and the number of "skip-if" annotations in ini manifests (does not handle reftests). The report looks like:

{
  "Testing::BrowserTest": [
    {
      "manifest": "testing/modules/tests/browser/browser.ini", 
      "skipped": 0, 
      "tests": 1
    }
  ], 
  "Testing::Firefox UI Tests": [
    {
      "manifest": "testing/firefox-ui/tests/functional/keyboard_shortcuts/manifest.ini", 
      "skipped": 0, 
      "tests": 1
    }, 
    {
      "manifest": "testing/firefox-ui/tests/functional/private_browsing/manifest.ini", 
      "skipped": 0, 
      "tests": 1
    }, 
    {
      "manifest": "testing/firefox-ui/tests/functional/safebrowsing/manifest.ini", 
      "skipped": 1, 
      "tests": 3
    }, 
 ...

I think it could be easily modified to report annotations containing "fission" in ini files, but I don't have a solution for reftests.

Flags: needinfo?(gbrown)

(In reply to Geoff Brown [:gbrown] from comment #3)

The work-in-progress patch on bug 1404472 generates a report of the test manifests associated with each bugzilla component and includes a count of tests found in each manifest and the number of "skip-if" annotations in ini manifests (does not handle reftests). The report looks like:
...
I think it could be easily modified to report annotations containing "fission" in ini files

That sounds like exactly what we want (well, except that we also need to take fail-ifs into account, but I don't think that will make things much harder).

but I don't have a solution for reftests.

We can leave reftests as a separate issue for now. The current focus is on mochitests only.

Usage: mach test-info report [<src path(s)>] [--components <bz component>,<bz component>,...]

Attached file fission-report.json —

How's this?

See Also: → 1404472

Nice! I love when requests are already solved ;). Thanks Geoff.

Would it be reasonable to block on bug 1404472 and then use this bug to add the ability to filter on specific configurations in a more robust manner (e.g to avoid picking up strings like !fission)?

Sure!

Depends on: 1404472
See Also: 1404472 →

Hi Geoff,
When should we expect bug 1404472 to land so it can be used to build up and add filter on top for tracking fission skipped tests?

Flags: needinfo?(gbrown)

I expect to request review tomorrow and land as soon as possible.

Flags: needinfo?(gbrown)
Comment on attachment 9080431 [details] [diff] [review]
mach subcommand from bug 1404472, hacked for "fission"

With the changes landed in bug 1404472, it is now possible to generate a report of fission-skipped tests with:

mach test-info report --show-tests --filter-keys=skip-if --filter-values=fission
Attachment #9080431 - Attachment is obsolete: true

One problem is that --filter-values=fission will pick up !fission.

So, e.g:

[test_foo.js]
skip-if = !fission

would get counted. Probably not a huge deal for now as I doubt many tests are annotated with !fission yet. Simple solution would be to allow this value to be a regex.

Filtering on the regex '(?<!!)fission' would be ideal-ish, yes.

I'm landing changes in bug 1572794 to support that sort of regex.

Then we can generate a fission test report with something like

mach test-info report --show-tests --show-summary --filter-keys=fail-if,skip-if --filter-values='(?<!!)fission'

But talking to Neha yesterday, she said 'it requires downloading the tree with every push and running that query. It would be really helpful if we had this info as part of the build metadata'. :ahal -- Do you have an idea of how to implement that? Might it be appropriate to add that on to a task like source-test-file-metadata-bugzilla-components?

Flags: needinfo?(ahal)

Hi Neha, please define "build metadata" (I think it might be the wrong term here). I mentioned in an IRC chat that we could set up a task that runs this command and uploads the output as an artifact. Then your scripts could download these artifacts as needed. Is that what you're looking for?

Flags: needinfo?(ahal) → needinfo?(nkochar)

Yes, an artifact that we could download from each m-c build will help so we don't have to pull the tree and run the query for each build.

Flags: needinfo?(nkochar)
Assignee: nobody → ahal
Status: NEW → ASSIGNED
Depends on: 1572794
Priority: P2 → P1
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a2339aac248c
[test-info] Create --output-file directory if it doesn't exist, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/7d7cdf191fe7
[build] Create objdir in test_manifest backend if it doesn't exist r=mshal
https://hg.mozilla.org/integration/autoland/rev/8f93f148e4fa
[ci] Add a task to generate test manifest metadata for fission r=gbrown

Andrew, I'm trying to query for these artifacts with the API in order to gather them together for a graph and am not having much luck. I'm guessing the artifact path I'm using is wrong, but based on the url I see for the artifact at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=cae93ef1993e02a136ef64d974856071b905997f&selectedJob=264337756 I'm not sure what else it should be. Here is the query I'm trying https://activedata.allizom.org/tools/query.html#query_id=jA_4qkgn:

{
	"sort":{"repo.push.date":"desc"},
	"limit":10,
	"from":"task.task.artifacts",
	"where":{"eq":{"name":"public/test-info-fission.json"}},
	"select":["task.artifacts.url","repo"],
	"format":"list"
}

Do you have any ideas how to get this list, or know who else I could ask?

Flags: needinfo?(ahal)

I created an index for this task, so use the taskcluster index instead. To get the latest:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.source.test-info-fission/artifacts/public/test-info-fission.json

Or by revision:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.revision.cae93ef1993e02a136ef64d974856071b905997f.source.test-info-fission/artifacts/public/test-info-fission.json

Take a look at the index browser for other ways to find it, like by date.

Flags: needinfo?(ahal)

(In reply to Andrew Halberstadt [:ahal] from comment #24)

I created an index for this task, so use the taskcluster index instead. To get the latest:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.source.test-info-fission/artifacts/public/test-info-fission.json

Or by revision:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.revision.cae93ef1993e02a136ef64d974856071b905997f.source.test-info-fission/artifacts/public/test-info-fission.json

Take a look at the index browser for other ways to find it, like by date.

For context, Andrew and I discussed how to collect these artifacts for a graph and decided that we'd like a new index that returns the latest artifact on a given day in order to avoid having to querying active data or scraping the tc index site to get the full timestamp of a push. He made a patch to do this in Bug 1578510.

See Also: → 1578510

Neha: is there a desire to somehow backfill these jobs so that we can show data from july/august in a dashboard, or should it start only once this data started being collected (Sept 2)? I'm not sure if it'd be possible to do through taskcluster or if we'd want to archive the existing data from the spreadsheet as JSON, but if it's something you want to show we could check out the options.

Flags: needinfo?(nkochar)

When we build the graphs, the July/August data should be visible too. As far as I understood, it wasn't possible to query such past data, which is why we were keeping the spreadsheet. If it is possible to backfill this data via taskcluster, then that's great. Otherwise, we can use kmag's spreadsheet data to do a one-time fill for the past numbers.

Flags: needinfo?(nkochar)

(In reply to Neha Kochar [:neha] from comment #28)

When we build the graphs, the July/August data should be visible too. As far as I understood, it wasn't possible to query such past data, which is why we were keeping the spreadsheet. If it is possible to backfill this data via taskcluster, then that's great. Otherwise, we can use kmag's spreadsheet data to do a one-time fill for the past numbers.

Kris, is it possible to get a copy of your daily spreadsheet data, but as JSON matching the structure of test-info-fission.json? We can hardcode those daily measurements up until around Sept 2 and then start gathering from taskcluster after that.

Flags: needinfo?(kmaglione+bmo)

(In reply to Brian Grinstead [:bgrins] from comment #29)

(In reply to Neha Kochar [:neha] from comment #28)

When we build the graphs, the July/August data should be visible too. As far as I understood, it wasn't possible to query such past data, which is why we were keeping the spreadsheet. If it is possible to backfill this data via taskcluster, then that's great. Otherwise, we can use kmag's spreadsheet data to do a one-time fill for the past numbers.

Kris, is it possible to get a copy of your daily spreadsheet data, but as JSON matching the structure of test-info-fission.json? We can hardcode those daily measurements up until around Sept 2 and then start gathering from taskcluster after that.

Nevermind, I ended up figuring out how to export individual csvs and then manually converted into the right json format.

Flags: needinfo?(kmaglione+bmo)
See Also: → 1656372
See Also: → 1748020
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: