1639886 - All android performance tests with `optimization` set are no longer running on mozilla-central

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Greg Mierzwinski [:sparky]]

We have many performance tests that run only on mozilla-central on per-commit, and through crons and these also don't run anywhere else. However, none of these tests are running on mozilla-central anymore since the optimization fields were added.

Here's the patch which caused this issue for all geckoview pageload tests: https://bugzilla.mozilla.org/show_bug.cgi?id=1626962

Here's the patch which caused this issue for all browsertime mobile tests (including the cron tests which are optimized out): https://bugzilla.mozilla.org/show_bug.cgi?id=1615255

So all browsertime mobile tests that ran on mozilla-central (either per-commit or through crons), and geckoview tests that only ran on mozilla-central are no longer running.

Comment 1

[Bob Clary [:bc] (inactive)]

It should only be active on autoland since it is implemented using the Backstop strategy.

ahal: Thoughts? Is the return value for should_remove_task when project != 'autoland', correct?

Comment 2

[Andrew Halberstadt [:ahal]]

It's correct in the use case of the test optimization:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/__init__.py#380

Which is All(<other optimizers>, 'backstop'). In other words, remove this task if all of <other optimizer> + backstop says to remove it. So by returning True on non-autoland projects, we essentially disable the backstop by removing it from that All equation.

But since for these tasks the backstop isn't being used with other optimizers, that line basically just causes the tasks to be removed.

However, even if we removed that line, the optimization would cause tasks to run every 25th mozilla-central push which is also not the desired behaviour (I assume we want them to run on every push there). So the solution here should be figuring out how to only apply that optimization on autoland and nowhere else. Maybe we can implement the ability to use by-project with the optimization key. Or else we could handle it in a transform rather than the .yml.

Comment 3

[Andrew Halberstadt [:ahal]]

Another option would be to create a new strategy that returns False on non-autoland branches. Maybe it could be an argument to the constructor that controls the return value.

But it must return True in the test optimization at least.

Comment 4

[Bob Clary [:bc] (inactive)]

(In reply to Andrew Halberstadt [:ahal] from comment #3)

Another option would be to create a new strategy that returns False on non-autoland branches. Maybe it could be an argument to the constructor that controls the return value.

But it must return True in the test optimization at least.

I like this approach the best though I am conflicted on variable naming and the necessary changes to the docstring. I'll clarify with you on Riot and submit a patch when we have decided on what to do.

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: Bug 1639886 - Allow the Backstop optimization strategy to be used outside of the test optimization, r=ahal.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f615c38b594
Allow the Backstop optimization strategy to be used outside of the test optimization, r=ahal.

Comment 7

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/6f615c38b594

Comment 8

[Greg Mierzwinski [:sparky]]

:bc, :ahal, thanks for the quick response and fix! The patch seems to have fixed the issue for raptor-geckoview, but the browsertime-geckoview tests still aren't running: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=browsertime-tp6m

Here's what we're expecting to see: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=browsertime-tp6m&revision=5415da14ec9a2f4749e8f405d6111a6f40e8138f

Comment 9

[Bob Clary [:bc] (inactive)]

Looking.

Comment 10

[Bob Clary [:bc] (inactive)]

I don't think the issue has to do with the push-interval-25 optimization for these tests. It seems to me that Bug 1623355 is a better candidate.

Comment 11

[Greg Mierzwinski [:sparky]]

Thanks for checking :bc! I think you are correct, I see this patch modified the YML config: https://phabricator.services.mozilla.com/D74306

Comment 12

[Mathew Hodson]

I think the "Regressed by: bug 1615255" should be cleared from this, since the browsertime-geckoview tests issue was covered by bug 1641017.