1815426 - Intermittent UI test failure - < SettingsPrivacyTest. verifyCookieBannerReductionTest >

Status: RESOLVED FIXED

(Firefox for Android :: UI Tests, defect, P3)

Description

[Andrei Jolțan [:Andi]]

Firebase Test Run: Firebase link

Stacktrace:

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:48)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:927)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyCookieBannerReductionTest$3.invoke(SettingsPrivacyTest.kt:624)
at org.mozilla.fenix.ui.SettingsPrivacyTest$verifyCookieBannerReductionTest$3.invoke(SettingsPrivacyTest.kt:622)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:138)
at org.mozilla.fenix.ui.SettingsPrivacyTest.verifyCookieBannerReductionTest(SettingsPrivacyTest.kt:622)

Build: 2/7 Main

Comment 1

[Andrei Jolțan [:Andi]]

Flaky 1x on 3/14 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:51)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:980)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:28)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:26)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:140)
at org.mozilla.fenix.ui.CookieBannerReductionTest.verifyCookieBannerReductionTest(CookieBannerReductionTest.kt:26)

Comment 2

[Andrei Jolțan [:Andi]]

Flaky 1x on 3/16 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:51)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:992)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:27)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:25)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:140)
at org.mozilla.fenix.ui.CookieBannerReductionTest.verifyCookieBannerReductionTest(CookieBannerReductionTest.kt:25)

Comment 3

[Intermittent Failures Robot]

4 failures in 3206 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• firefox-android: 4

Platform and build breakdown:

• fenix-ui-test: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1815426&startday=2023-04-10&endday=2023-04-16&tree=all

Comment 4

[Andrei Jolțan [:Andi]]

Flaky 1x on 4/18 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:51)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:1019)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:27)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:25)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:140)
at org.mozilla.fenix.ui.CookieBannerReductionTest.verifyCookieBannerReductionTest(CookieBannerReductionTest.kt:25)

Comment 5

[Andrei Jolțan [:Andi]]

Flaky 1x on 5/2 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:54)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:756)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:27)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:25)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:140)
at org.mozilla.fenix.ui.CookieBannerReductionTest.verifyCookieBannerReductionTest(CookieBannerReductionTest.kt:25)

Comment 6

[Andrei Jolțan [:Andi]]

Flaky 1x on 5/8 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:54)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:762)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:27)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$3.invoke(CookieBannerReductionTest.kt:25)
at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:140)
at org.mozilla.fenix.ui.CookieBannerReductionTest.verifyCookieBannerReductionTest(CookieBannerReductionTest.kt:25)

Comment 7

[Andrei Jolțan [:Andi]]

Seems to be cause by 1807440

Comment 8

[Andrei Jolțan [:Andi]]

Flaky 1x on 5/17 Main Firebase link

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertFalse(Assert.java:65)
at org.junit.Assert.assertFalse(Assert.java:75)
at org.mozilla.fenix.helpers.MatcherHelper.assertItemWithResIdExists(MatcherHelper.kt:59)
at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerExists(BrowserRobot.kt:763)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$15.invoke(CookieBannerReductionTest.kt:62)
at org.mozilla.fenix.ui.CookieBannerReductionTest$verifyCookieBannerReductionTest$15.invoke(CookieBannerReductionTest.kt:61)
at org.mozilla.fenix.ui.robots.ThreeDotMenuMainRobot$Transition.refreshPage(ThreeDotMenuMainRobot.kt:313)
at org.mozilla.fenix.ui.CookieBannerReductionTest.verifyCookieBannerReductionTest(CookieBannerReductionTest.kt:61)

Comment 9

[Aaron Train [:aaronmt]]

Turning off in bug 1833933. I didn't see it fail, but it's likely too as the other test did in the same Class with commit https://github.com/mozilla-mobile/firefox-android/commit/8c414a423774576a762d90f6a3c67761daee73c5

Comment 10

[Andrei Jolțan [:Andi]]

Fixed with #2104

Comment 11

[Andrei Jolțan [:Andi]]

► 6/15 Firebase link

► 6/16 Failed 1x
Firebase link
Firebase link

► 6/18 Firebase link

androidx.test.uiautomator.UiObjectNotFoundException: UiSelector[CONTAINS_DESCRIPTION=Refresh]

Comment 12

[BMO Github Automation]

Authored by https://github.com/AndiAJ
https://github.com/mozilla-mobile/firefox-android/commit/b6459a0a47074e2babc1f58389128f92b1b02afd
[main] Bug 1815426 - Fix failing cookie banner reduction UI tests