Schedule "new manifests" with the manifest based bugbug optimizers
Categories
(Firefox Build System :: Task Configuration, enhancement, P2)
Tracking
(Not tracked)
People
(Reporter: ahal, Unassigned)
References
Details
Bug 1665713 renamed some mochitest-media manifests, but the associated mach try auto push didn't select them, presumably because bugbug is unaware that they exist.
We handle this at the "task" level, with bugbug sending the entire list of tasks that it is aware of as part of the response. We could potentially do something similar for manifests as well, though the list of manifests is much much larger, so I'm not sure if it's feasible.
Not sure what else we can do, but this is definitely a case that should be handled.
|
Reporter | |
Comment 1•4 years ago
|
||
It's also possible that bugbug is aware that the newly renamed files are manifests, but just lost all previously acquired data on them. Though if this is the case, I'd be surprised that the fact the file was modified wasn't enough to get it selected..
I guess when manifests are renamed we'd ideally be able to transfer the data on it as well, but that is a separate issue (belonging on the bugbug repo).
|
||
Comment 2•4 years ago
|
||
:ahal, Thank you for filing this!
|
||
Comment 3•4 years ago
|
||
Ideally, we should handle the renamings in mozci directly, so the regression detection algorithm can find the same task/group in parent/children pushes. I had filed https://github.com/mozilla/mozci/issues/231 about tasks, but we can do the same for groups.
It's a manual process though, it'd be nice if we found some automated way to do it (even though it is likely rare).
(In reply to Andrew Halberstadt [:ahal] from comment #1)
It's also possible that bugbug is aware that the newly renamed files are manifests, but just lost all previously acquired data on them. Though if this is the case, I'd be surprised that the fact the file was modified wasn't enough to get it selected..
Bugbug won't select anything that it doesn't know about, as it can't know what is a group and what isn't. Currently, it relies on a built-in list of groups that were known at the time of training. If I had a way to dinamically know what manifests are schedulable on a given revision, I could support scheduling previously unseen manifests too (it wouldn't work as good as other manifests given that it would have no information about previous failures, but it might work reasonably well).
By the way, in this specific case, shouldn't they be run via test-verify?
|
|
Reporter | |
Comment 4•4 years ago
|
||
(In reply to Marco Castelluccio [:marco] from comment #3)
Ideally, we should handle the renamings in mozci directly, so the regression detection algorithm can find the same task/group in parent/children pushes.
Agreed, though it's not just renames, it's also when a new manifest is created.
(In reply to Andrew Halberstadt [:ahal] from comment #1)
It's also possible that bugbug is aware that the newly renamed files are manifests, but just lost all previously acquired data on them. Though if this is the case, I'd be surprised that the fact the file was modified wasn't enough to get it selected..
If I had a way to dinamically know what manifests are schedulable on a given revision, I could support scheduling previously unseen manifests too (it wouldn't work as good as other manifests given that it would have no information about previous failures, but it might work reasonably well).
You could grab this from the manifests-by-task (or even tests-by-manifest) artifact on a decision task (you'd need to make a set of all the values in that dict). We could upload a simple text list as well if it helped.
By the way, in this specific case, shouldn't they be run via test-verify?
Yes, looks like it did, but the way the test was invoked didn't reproduce this specific error since I believe test-verify passes in the test file name directly, and this was an issue with manifests or something like that. Also it's pretty easy to miss stuff in the noise of WPT. But good point, the fact that test-verify should run new tests is a mitigating factor.
|
|
||
Comment 5•4 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #4)
(In reply to Andrew Halberstadt [:ahal] from comment #1)
It's also possible that bugbug is aware that the newly renamed files are manifests, but just lost all previously acquired data on them. Though if this is the case, I'd be surprised that the fact the file was modified wasn't enough to get it selected..
If I had a way to dinamically know what manifests are schedulable on a given revision, I could support scheduling previously unseen manifests too (it wouldn't work as good as other manifests given that it would have no information about previous failures, but it might work reasonably well).
You could grab this from the manifests-by-task (or even
tests-by-manifest) artifact on a decision task (you'd need to make a set of all the values in that dict). We could upload a simple text list as well if it helped.
Right now the decision task itself makes requests to bugbug, so bugbug can't retrieve artifacts from the decision task. As discussed on Matrix, we could change the API so it receives the list of manifests via POST. Another option would be to make bugbug generate the list of manifests using mach if it is not too slow, though it'd require bugbug to have a fully working mach environment.
Description
•