1669513 - Backfilled tasks are filtered out by Test Groups/Test path filter

Status: NEW

(Tree Management :: Treeherder: Frontend, defect)

Description

[Cosmin Sabou [:CosminS]]

We had this failure on tree https://tinyurl.com/y2cyuhse for which we backfilled here: https://tinyurl.com/y3lfr3md
if we want to filter by that failing test mobile/android/components/extensions/test/mochitest/test_ext_tabs_create.html to see it's history, where it ran and was green and where it first failed, using test path or test groups the backfill jobs get filtered out. eg: https://tinyurl.com/yysffud2 or https://tinyurl.com/y5fu2zp9
If there's a way to also have the jobs triggered by the backfill action be included in those filters it would be welcomed. Thanks.

Comment 1

[Andrew Halberstadt [:ahal]]

Weird, this definitely looks like a bug in the filtering logic.. If I remove the task name filter then one of the backfill tasks shows up further down:
https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=402686aadec108248fe5c83e5ed6496174889c03&fromchange=b4b7e943b93cdc77a479bd5484f7661985bdb7d4&test_paths=mobile%2Fandroid%2Fcomponents%2Fextensions%2Ftest%2Fmochitest%2Fmochitest.ini&group_state=expanded

But still missing the orange backfill tasks. So it doesn't seem to be simply a case of "backfill tasks are skipped by the filter". Something more subtle is happening here.

Cam, do you think you or Sarah would have time to look at this in the nearish future? Simplified STR below.

STR

Open https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitests%2Ctest-android-em-7.0-x86_64%2Fopt-geckoview-mochitest-plain-e10s%2C4&revision=43467c261c23e6406e06cb0ed7e6bbde4cb3d22c&test_paths=mobile%2Fandroid%2Fcomponents%2Fextensions%2Ftest%2Fmochitest%2Fmochitest.ini

(notice the test path filter)

Expected

This task is displayed (since it contains mobile/android/components/extensions/test/mochitest/mochitest.ini):
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=WKzl1xFQRFWy_uZw_SEXiw.0&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitests%2Ctest-android-em-7.0-x86_64%2Fopt-geckoview-mochitest-plain-e10s%2C4&revision=43467c261c23e6406e06cb0ed7e6bbde4cb3d22c

Actual

That task is not displayed

Comment 2

[Cameron Dawson [:camd]]

I will look into this next week. I have just been swamped for the past couple days.

Comment 3

[Cameron Dawson [:camd]]

I'm sorry for the delay on this. I've been moved into a sprint and just haven't had time to circle back to this. I will keep it on my plate and try to fit it in where I can.

Comment 4

[Cameron Dawson [:camd]]

We would need an artifact from Taskcluster that is manifests_by_label rather than manifests_by_task, as we have now. Alternatively, the backfill-task could have a manifests_by_task artifact of its own. We could load it and merge with all the others from the push.

Barring that, I think our progress on this might be pretty blocked.

Comment 5

[Cameron Dawson [:camd]]

Dustin-- Would you know if my idea in comment 4 was possible? Or if there's already something like this we could access?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I don't, sorry. Maybe Andrew knows, or knows who would?

Comment 7

[Andrew Halberstadt [:ahal]]

Ah, manifests_by_label wouldn't be helpful because backfills explicitly have a different set of manifests than the task with the same label on the current push. So if we used that the test path filter would be wrong for backfills.

So I think we'd need to do your second idea and make a manifests_by_task artifact on the backfill tasks and then merge them together.. Either that or push forward with Armen's plan to handle manifests and filtering therein on the backend. If the latter is still on the table, I'd vote we block on that.. but I understand it's a lot of work and resources are scarce.

Comment 8

[Cameron Dawson [:camd]]

I think we should go with the idea of creating a manifests_by_task artifact for backfill and "add new jobs" actions. I think it would introduce less risk at this time.

This would involve someone upstream to generate those files when we do either of those actions. With Ahal on leave, would that be Marco we would ask? I'm not certain.

This is the function that would need to be modified:
https://github.com/mozilla/treeherder/blob/5242e612f708d8ccd03c119ab0c5af4afc5a99ab/ui/job-view/pushes/Push.jsx#L266

And that function calls fetchGeckoDecisionArtifact here:
https://github.com/mozilla/treeherder/blob/5242e612f708d8ccd03c119ab0c5af4afc5a99ab/ui/job-view/pushes/Push.jsx#L71

We might end up either creating a new version of that which gets manifests_by_task for ALL the decision and action tasks.

Then in transformedPaths we would merge all the files here:
https://github.com/mozilla/treeherder/blob/5242e612f708d8ccd03c119ab0c5af4afc5a99ab/ui/job-view/pushes/Push.jsx#L61

Comment 9

[Razvan Maries]

We noticed that if we set the "times" parameter for a backfill to more than 1 and if the necessary build did not run, it will be requested multiple times.

E.g. https://treeherder.mozilla.org/jobs?repo=autoland&fromchange=99e23329a4ad9aaa13611cf91b3ca80cc8713bce&searchStr=windows%2C10%2Cx64%2Cdebug%2Cmochitests%2Cwithout%2Ce10s%2Ctest-windows10-64%2Fdebug-mochitest-chrome-1proc%2Cc3&tochange=4a6818e6c81a2c0c9221b8e0306d3c47d8cc666e&group_state=expanded&selectedTaskRun=XWne8GiDStmd6uszaAw7vw.0

Here, I requested a backfill for the c3 job and I set the "times" parameter to 5. On this push the Windows 2012 x64 debug B has been requested 5 times.

Cameron, should we file a new bug for this issue?

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

That's bug 1658174.

Comment 11

[Cameron Dawson [:camd]]

Hey Sarah-- This is the bug/issue about needing a new artifact for when backfills happen so that we can filter by test path. Comment 8 was my input.

Comment 12

[Sarah Clements [:sclements]]

ok, thanks