1882087 - make sure android tests are sheriffable

Status: RESOLVED FIXED

(Fenix :: General, task)

Description

[Julien Cristau [:jcristau]]

As firefox-android moves to the gecko tree, build/test failures need to be sheriffable.

Comment 1

[Julien Cristau [:jcristau]]

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Requirements_for_jobs_shown_in_the_default_Treeherder_view

Comment 2

[Geoff Brown [:gbrown]]

:jmaher pointed me at https://github.com/mozilla/treeherder/blob/1160cd0262e29cae46a39610dd0160aa497d7a58/treeherder/log_parser/parsers.py#L42 as the source of log parsing expressions.

Comment 3

[Geoff Brown [:gbrown]]

What does geckoview do? See bug 1457255 and https://searchfox.org/mozilla-central/rev/202c48686136360a23b73a49b611a19e64f3e1b8/mobile/android/geckoview/build.gradle#165-202

Comment 4

[Geoff Brown [:gbrown]]

As an example, here's a broken geckoview build:
https://treeherder.mozilla.org/logviewer?job_id=449500452&repo=try&lineNumber=2943

[task 2024-03-04T23:34:12.658Z] 23:34:12    ERROR -  /builds/worker/checkouts/gecko/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoSession.java:132: error: cannot find symbol
[task 2024-03-04T23:34:12.658Z] 23:34:12     INFO -        return this == otherz;
[task 2024-03-04T23:34:12.659Z] 23:34:12     INFO -                       ^
[task 2024-03-04T23:34:12.659Z] 23:34:12     INFO -    symbol:   variable otherz
[task 2024-03-04T23:34:12.659Z] 23:34:12     INFO -    location: class State

vs. a broken fenix build:
https://treeherder.mozilla.org/logviewer?job_id=449512944&repo=try&lineNumber=4761

[task 2024-03-05T01:09:51.667Z] e: file:///builds/worker/checkouts/gecko/mobile/android/fenix/app/src/main/java/org/mozilla/fenix/FenixApplication.kt:216:29 Unresolved reference: telemetryEnabledz
[task 2024-03-05T01:09:51.667Z] 
[task 2024-03-05T01:09:51.667Z] > Task :app:compileFenixDebugKotlin FAILED
[task 2024-03-05T01:09:51.767Z] 
[task 2024-03-05T01:09:51.767Z] FAILURE: Build failed with an exception.

Comment 5

[Aaron Train [:aaronmt]]

For UI test tasks, I had written https://wiki.mozilla.org/Sheriffing/How_To/Mobile_Trees/Firebase_Test_Lab

e.g, FAILURE from Oak https://firefoxci.taskcluster-artifacts.net/Q25OuLWpTui5XFd4cSyJ7w/0/public/logs/live_backing.log

Comment 6

[Geoff Brown [:gbrown]]

Thanks Aaron, I wasn't aware of that. I also see now that there is existing documentation about sheriffing tests at https://wiki.mozilla.org/Sheriffing/How_To/Mobile_Trees. I still think we should try to make firefox-android error messages more easily available in treeherder: Provide at least a basic error message for most failures in the treeherder "Failure Summary".

Comment 7

[Titouan Thibaud [:titouan]]

Attachment: WIP: Bug 1882087 - test gradle changes

Comment 8

[Julien Cristau [:jcristau]]

Attachment: Bug 1882087 - Format test failure messages for treeherder

Comment 9

[Julien Cristau [:jcristau]]

Attachment: Bug 1882087 - Format ui-test failure messages for treeherder

Comment 10

[Oana Horvath [:ohorvath]]

Attachment: WIP: Bug 1882087 - Format ui-test failure messages for treeherder

Comment 11

[Geoff Brown [:gbrown]]

Attachment: Bug 1882087 - Format build failure messages for treeherder

Comment 12

[Geoff Brown [:gbrown]]

Attachment: Bug 1882087 - Improve ui-test failure messages for treeherder

Comment 13

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/mozilla-central/rev/8accb37f1739
https://hg.mozilla.org/mozilla-central/rev/35aad4a76aa6
https://hg.mozilla.org/mozilla-central/rev/adf10a81658a

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/26662bd77b86
Improve ui-test failure messages for treeherder r=aaronmt

Comment 16

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/26662bd77b86