Closed Bug 1265979 Opened 8 years ago Closed 8 years ago

Intermittent failing tc(test) build Halting on failure while running ['python2.7', 'mach', '--log-no-times', 'gradle', 'app:testAutomationDebugUnitTest']

Categories

(Firefox for Android Graveyard :: General, defect)

defect
Not set
normal

Tracking

(firefox48 fixed)

RESOLVED FIXED
Firefox 48
Tracking Status
firefox48 --- fixed

People

(Reporter: cbook, Assigned: Grisha)

References

()

Details

Attachments

(1 file)

https://treeherder.mozilla.org/logviewer.html#?job_id=8817899&repo=fx-team

 04:21:52     INFO -  :app:testAutomationDebugUnitTest

 04:21:52     INFO -  org.mozilla.gecko.db.BrowserProviderVisitsTest > testDefaultSortingOrder FAILED

 04:21:52     INFO -      java.lang.RuntimeException at BrowserProviderVisitsTest.java:130

 04:21:52     INFO -          Caused by: java.lang.IllegalStateException

 04:22:10     INFO -  423 tests completed, 1 failed, 2 skipped

 04:22:10     INFO -  :app:testAutomationDebugUnitTest FAILED

 04:22:10     INFO -  FAILURE: Build failed with an exception.

 04:22:10     INFO -  * What went wrong:

 04:22:10     INFO -  Execution failed for task ':app:testAutomationDebugUnitTest'.

 04:22:10     INFO -  > There were failing tests. See the report at: file:///home/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/app/reports/tests/automationDebug/index.html

i guess caused by bug 1046709
Assignee: nobody → gkruglov
Blocks: 1046709
Summary: Intermittent failing frontend tc(test) "OSError: [Errno 2] No such file or directory" & "SyntaxError: unexpected EOF while parsing" due to failures in BrowserProviderVisitsTest.java:130 → Intermittent failing frontend tc(test) "org.mozilla.gecko.db.BrowserProviderVisitsTest > testDefaultSortingOrder FAILED"
Summary: Intermittent failing frontend tc(test) "org.mozilla.gecko.db.BrowserProviderVisitsTest > testDefaultSortingOrder FAILED" → Intermittent failing Build "org.mozilla.gecko.db.BrowserProviderVisitsTest > testDefaultSortingOrder FAILED" frontend tc(test)
Trying to get this to show up as a starrable intermittent in Treeherder.
Summary: Intermittent failing Build "org.mozilla.gecko.db.BrowserProviderVisitsTest > testDefaultSortingOrder FAILED" frontend tc(test) → Intermittent failing tc(test) build Halting on failure while running ['python2.7', 'mach', '--log-no-times', 'gradle', 'app:testAutomationDebugUnitTest']
This is an unacceptable intermittent failure rate.  Really, bug 1046709 should have been backed out rather than being allowed to stay in the tree with this level of test failures, but it's been a day at this point, and the patch seems to have other things on top of it now.

Do you have a plan for fixing this bug this week?
Flags: needinfo?(gkruglov)
(In reply to David Baron [:dbaron] ⌚️UTC-7 from comment #4)
> This is an unacceptable intermittent failure rate.  Really, bug 1046709
> should have been backed out rather than being allowed to stay in the tree
> with this level of test failures,

was because tier 2

 Tier 2: Jobs are shown by default on Treeherder, but are not sheriff-managed. Results will be shown on Treeherder "for information only". New test failures/bustage will not result in a backout, but a tracking bug will be filed when observed. 

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
Will have a fix shortly.
Flags: needinfo?(gkruglov)
Nick, I've been going through these tests [0], and the problem seems to be that somehow, db connection isn't cleared up correctly in-between individual tests. See [1]. Something is stuck with an invalid connection pointer. Provided that in an @After method I'm calling BrowserProvider's `shutdown` (which ends up disposing of underlying database connections), I'm not sure how that might be.

As an experiment (and somewhat out of frustration), I've looked closely at any differences in how TabsProvider tests work (Bug 1260478), and modified my BrowserProvider tests accordingly, but of course that didn't help at all. The main difference is that I'm re-creating content provider clients in @Before, and releasing them in @After.

To add to confusion, I would expect this failure to occur for every test run, but it only happens for one of the tests (not any in particular, I think the 2nd test that gets run? I'm not sure actually which). Perhaps that's a hint? Removing that call to shutdown() results in this exact failure in every test, which is what one would expect.

In Android Studio, everything works peachy. How is `./gradlew test` different from running these tests in AS, in a way that's relevant to this?

Until this is figured out, I can back-out these tests in the morning, to at least get build failures to stop. Unless you can, perhaps, point to an obviously stupid thing that I'm doing :-) Or some another form of divine intervention occurs. :/

[0] https://dxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/db/BrowserProviderVisitsTest.java
[1] https://public-artifacts.taskcluster.net/SZGzUVCvTdiw2JpQy1gQyw/0/public/android/unittest/automationDebug/classes/org.mozilla.gecko.db.BrowserProviderVisitsTest.html
Flags: needinfo?(nalexander)
Flags: needinfo?(nalexander)
Attachment #8744476 - Flags: review?(s.kaspari) → review+
Comment on attachment 8744476 [details]
MozReview Request: Bug 1265979 - missing call to provider.shutdown() causing test failures r=sebastian

https://reviewboard.mozilla.org/r/48563/#review45277

::: mobile/android/tests/background/junit4/src/org/mozilla/gecko/sync/repositories/android/VisitsHelperTest.java:89
(Diff revision 1)
> -        Assert.assertEquals(10, recentVisits.size());
> +            Assert.assertEquals(10, recentVisits.size());
> -        for (int i = 0; i < recentVisits.size(); i++) {
> +            for (int i = 0; i < recentVisits.size(); i++) {
> -            JSONObject v = (JSONObject) recentVisits.get(i);
> +                JSONObject v = (JSONObject) recentVisits.get(i);
> -            Long date = (Long) v.get("date");
> +                Long date = (Long) v.get("date");
> -            Integer type = (Integer) v.get("type");
> +                Integer type = (Integer) v.get("type");
> -            Assert.assertEquals(Long.valueOf(baseDate - i * 100), date);
> +                Assert.assertEquals(Long.valueOf(baseDate - i * 101), date);

Without this r+ :)
Comment on attachment 8744476 [details]
MozReview Request: Bug 1265979 - missing call to provider.shutdown() causing test failures r=sebastian

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48563/diff/1-2/
https://hg.mozilla.org/integration/fx-team/rev/b50652d71510d898b7e1f94d8f8d4330b7d1a486
Bug 1265979 - missing call to provider.shutdown() causing test failures r=sebastian
Status: NEW → ASSIGNED
https://hg.mozilla.org/mozilla-central/rev/b50652d71510
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 48
Let's summarize what we have found in case this happens again and someone stumbles upon this bug:

One of the tests did not call shutdown() on the content provider. Robolectric will create a new database instance for every test. Keeping and reusing references between tests will lead to the IllegalStateException ("Illegal connection pointer 1. Current pointers for thread [..]"). In addition to that the test order seems to be randomized (or at least different depending on the machine they are running on): Intermittent instead of perma fail.
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.