1668384 - Filter by test path not working

Status: NEW

(Tree Management :: Treeherder: Frontend, defect)

Description

[Narcis Beleuzu [:NarcisB]]

For example:

• with filter applied:
  https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Crunnable&searchStr=windows%2C10%2Cx64%2Copt%2Cmochitests%2Ctest-windows10-64%2Fopt-mochitest-browser-chrome-e10s%2Cbc1&test_paths=browser%2Fbase%2Fcontent%2Ftest%2Fstatic&fromchange=3336746f11cd7d2fcffac299a4c752dd3564a9e8

• without the filter:
  https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=GdK2ozkpRaqx7NWgy6Uz7w.0&resultStatus=success%2Ctestfailed%2Cbusted%2Crunnable&searchStr=windows%2C10%2Cx64%2Copt%2Cmochitests%2Ctest-windows10-64%2Fopt-mochitest-browser-chrome-e10s%2Cbc1&fromchange=3336746f11cd7d2fcffac299a4c752dd3564a9e8

Comment 1

[Andrew Halberstadt [:ahal]]

I believe this is an intentional regression that Armen made to improve performance before he left. You should be able to insert the manifest into the filter (which is linked from the test groups tab) and it should work then..

Armen was planning to fix this regression next but obviously never got the chance. His solution was going to involve a bunch of backend work, which we would now need to rely on Cam or Sarah for help. But maybe there is something easier we can do to improve it a little bit in the meantime.

Thanks for filling!

Comment 2

[Narcis Beleuzu [:NarcisB]]

Thanks for the info, but, for the below link, if I select "docshell/test/browser/browser.ini" from "Test Groups", why the bc failures from 681bab0c299c are not showing?

https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=fpO1odPpTK-0YxfZwyIe8A.0&resultStatus=testfailed%2Cbusted%2Crunnable&searchStr=linux%2C18.04%2Cx64%2Copt%2Cmochitests%2Ctest-linux1804-64%2Fopt-mochitest-browser-chrome-e10s%2Cbc4&fromchange=1ed849fcf90ca09e5db9917e3d3b7b3dc2a922cf&tochange=adf38a6442cf89e88135dbf0b2d3b53c07a9d800

Comment 3

[Cosmin Sabou [:CosminS]]

The problem here was the test filter was missing an "/" so the updated link would look something like: https://tinyurl.com/y949x9ho
Another issue with this kind of filtering is that it doesn't work for wpts and because of the manifest changes those tests are scheduled rarely and hard to find when they started permafailing having no reliable history for them, eg (with a test filter for a wpt): https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&fromchange=8a73f3d643d45b5022d63cba2279710335d3202d&tochange=efdf5255c248b66c9c132db4087f0ef464d5018a&test_paths=testing%2Fweb-platform%2Ftests%2Fwebdriver%2Ftests%2Felement_clear%2F
or https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Crunnable&fromchange=3336746f11cd7d2fcffac299a4c752dd3564a9e8&tochange=1ed849fcf90ca09e5db9917e3d3b7b3dc2a922cf&test_paths=testing%2Fweb-platform%2Ftests%2Fhtml%2Fcross-origin-embedder-policy%2F
Also on the latest manifest change Bug 1665522 has come up again.

Comment 4

[Andrew Halberstadt [:ahal]]

Ah, so directories do work, just not test files... Seems like we could do some treeherder frontend hackery to handle the missing "/".

As for WPT, you need to use the manifest ID rather than the test path. So e.g, instead of testing/web-platform/tests/webdriver/tests/cookies/path, try using /cookies/path. This is also something we could just convert automatically in the frontend (I thought it was already doing this, but I guess not).

In both cases, the best bet is to click the Test Groups tab and click the links in there (rather than entering test path filters manually). As that should definitely apply the filter in the right format.

Anyway, we should definitely leave this bug open to improve things. I'll see if Cam or Sarah can help out.