1888304 - Code changes to firefox-android do not trigger relevant tests on autoland

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

When we originally reviewed task scheduling for firefox-android on mozilla-central in bug 1883499, I noted three perspectives:

1. Aryx: android tests should run as often as possible to enable regression detection, or at least on backstop pushes ("Backstop pushes always run the desktop, geckoview, fenix and focus tasks.")
2. Aaron: android tests should run on autoland for code changes to firefox-android code.
3. Me: we should be careful not to run firebase tests too often.

There's also the perspective of bug 1887747:
4. Joel: roughly, the firefox-android migration should not significantly increase CI costs.

With bug 1886193, firefox-android builds and tests are running on autoland for backstop pushes, but rarely otherwise. I think that's a reasonable accommodation of 1, 3, and 4.

What about 2 - can/should we also run firefox-android tests when particularly relevant code changes (run fenix tests when a push modifies mobile/android/fenix/**)?

:ahal - I see https://searchfox.org/mozilla-central/rev/f8258e49f277c1c881c5c0da95f463dd12b7397e/mobile/android/moz.build#16, for example, but https://treeherder.mozilla.org/jobs?repo=autoland&revision=cd15842a69cf0d620e71db63238d7479f7beff33, which modifies mobile/android/fenix/**, ran no fenix tests. Is that expected and intentional?

Comment 1

[Andrew Halberstadt [:ahal]]

Yes that's expected. There's two optimizations those tasks need to pass:

1. skip-unless-schedules: These are the moz.build rules you see with "exclusive". All these rules do is say "optimize" the task if none of the modified files could possibly affect it. So this optimization is irrelevant to our use case because we're thinking about what to do when highly relevant files are modified.

2. bugbug: This is the machine learning algorithm, and almost certainly what is causing the tasks to be optimized. Put simply, the algorithm isn't very good at determining what files can break these tasks yet. It should get better with time, but I'm not sure how long it will take or how much better it will get.

We don't really have precedent here, but it seems like we want to invent a new composite optimization strategy that does something like:

SCHEDULES.inclusive AND (SCHEDULES.exclusive OR bugbug)

(Inclusive is the run this task when these files change variation of schedules rules)

In English the above translates to:
Optimize this task if no highly relevant files were modified, AND EITHER none of the modified files could possibly affect this task OR bugbug told us to do it.

Let me know if that doesn't make any sense :)

Comment 2

[Andrew Halberstadt [:ahal]]

One thing to beware of is that both the inclusive and exclusive schedules rules are processed by the same optimization strategy currently. For this idea to work, I think they would need to be refactored into independent strategies.

Comment 3

[Aaron Train [:aaronmt]]

Just an example change where I would have expected to see all inclusive test tasks run https://treeherder.mozilla.org/jobs?repo=autoland&revision=263fe8f229dcd608345da2eac2dc51cc54316c49 with mobile/android/fenix/app changes but there was no ui-test-apk scheduling. Can we ensure that when test-apk-fenix/focus is scheduled, so too ui-test-apk?

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Aaron Train [:aaronmt] from comment #3)
That's a good example - thanks! I see now that bug 1886193 missed the ui-test-apk kind. I can fix that...

Comment 5

[Geoff Brown [:gbrown]]

Attachment: Bug 1888304 - Use 'build' optimization strategy for android ui tests also

Extend the build optimization strategy of bug 1886193 to the ui-test-apk and test-components kinds.

Comment 6

[Geoff Brown [:gbrown]]

I'm hopeful that the tweak in comment 5 will be sufficient. Let's go with that and resolve this bug; if there are further concerns, we can re-open and revisit comment 1.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9e5d4021d56c
Use 'build' optimization strategy for android ui tests also r=ahal,taskgraph-reviewers

Comment 8

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/9e5d4021d56c