Closed Bug 915449 Opened 10 years ago Closed 9 years ago

Intermittent (testNewTab | testShareLink | testSearchSuggestions | testBookmark | testMasterPassword | testBookmarksPage | testBookmarksPanel) | Exception caught - junit.framework.AssertionFailedError: EditText is not found!

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox28 fixed, firefox29 fixed, firefox30 fixed, firefox31 fixed, firefox-esr24 wontfix, b2g-v1.2 wontfix)

Status:
RESOLVED FIXED
Milestone:
Firefox 31
Tracking Flags:
Tracking Status
firefox28 --- fixed
firefox29 --- fixed
firefox30 --- fixed
firefox31 --- fixed
firefox-esr24 --- wontfix
b2g-v1.2 --- wontfix

People

(Reporter: KWierso, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 2 obsolete files)

pause before clicking Save
2.97 KB, patch
jmaher
: review+
Details | Diff | Splinter Review
avoid use of getEditText
1.45 KB, patch
mcomella
: review+
Details | Diff | Splinter Review 
4 INFO TEST-PASS | testNewTab | waiting for add tab view - add tab view available
Exception caught during test!
junit.framework.AssertionFailedError: EditText is not found!
	at junit.framework.Assert.fail(Assert.java:47)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at com.jayway.android.robotium.solo.Waiter.waitForAndGetView(Waiter.java:501)
	at com.jayway.android.robotium.solo.Getter.getView(Getter.java:49)
	at com.jayway.android.robotium.solo.Solo.getEditText(Solo.java:1774)
	at org.mozilla.fennec.tests.BaseTest$1.isSatisfied(BaseTest.java:207)
	at com.jayway.android.robotium.solo.Waiter.waitForCondition(Waiter.java:361)
	at com.jayway.android.robotium.solo.Solo.waitForCondition(Solo.java:420)
	at org.mozilla.fennec.tests.BaseTest.waitForCondition(BaseTest.java:316)
	at org.mozilla.fennec.tests.BaseTest.focusUrlBar(BaseTest.java:204)
	at org.mozilla.fennec.tests.BaseTest.enterUrl(BaseTest.java:221)
	at org.mozilla.fennec.tests.BaseTest.inputAndLoadUrl(BaseTest.java:251)
	at org.mozilla.fennec.tests.BaseTest.addTab(BaseTest.java:578)
	at org.mozilla.fennec.tests.testNewTab.addTab(testNewTab.java:9)
	at org.mozilla.fennec.tests.testNewTab.testNewTab(testNewTab.java:40)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:521)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:162)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
5 INFO TEST-UNEXPECTED-FAIL | testNewTab | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
6 INFO TEST-END | testNewTab | finished in 22629ms
7 INFO TEST-START | Shutdown
https://tbpl.mozilla.org/php/getParsedLog.php?id=28194735&tree=Mozilla-Inbound
Summary: Intermittent TEST-UNEXPECTED-FAIL | testNewTab | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent TEST-UNEXPECTED-FAIL | testNewTab,testShareLink | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
Summary: Intermittent TEST-UNEXPECTED-FAIL | testNewTab,testShareLink | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent testNewTab, testShareLink | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
https://tbpl.mozilla.org/php/getParsedLog.php?id=29576861&tree=Mozilla-Inbound
Summary: Intermittent testNewTab, testShareLink | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent testNewTab,testShareLink,testSearchSuggestions | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
https://tbpl.mozilla.org/php/getParsedLog.php?id=29807478&tree=Fx-Team
Summary: Intermittent testNewTab,testShareLink,testSearchSuggestions | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
Assignee: nobody → markcapella
Status: NEW → ASSIGNED
https://tbpl.mozilla.org/php/getParsedLog.php?id=30004179&tree=Mozilla-Inbound
Summary: Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark,testMasterPassword | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
A lot of our random orange tests seem to display this signature

Exception caught during test!
junit.framework.AssertionFailedError: EditText is not found!
	at junit.framework.Assert.fail(Assert.java:47)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at com.jayway.android.robotium.solo.Waiter.waitForAndGetView(Waiter.java:501)

This tracks into the robotium code here ... 
http://grepcode.com/file/repo1.maven.org/maven2/com.jayway.android.robotium/robotium-solo/4.2/com/jayway/android/robotium/solo/Waiter.java#Waiter.waitForAndGetView%28int%2Cjava.lang.Class%29

We've called robotium to get us a view, he timeouts waiting for some to exist at line #482. At line #486 both views.size() and numberOfUniqueViews) are 0, so he throws at line #494 trying to .get(0), where none (as he already knows) exists.

I'll post more as I get back into it.

Ping gbrown cause he's handy.
Flags: needinfo?(gbrown)
One thing I notice is that this bug seems to happen more on tegra-362 than on other devices: Comments 61, 57, 56, 53, 52, ... It might be time to re-image that device. 

Of the remainder, most (but not all) are in focusUrlBar. Consider Comment 54, for instance. In focusUrlBar, we should have clicked on the url bar and then called mSolo.getEditText(0), which should have waited for the short timeout (10000 ms, I believe) before failing. I find it curious that the screenshot does not show the url bar focused. What did mSolo.getView(BROWSER_TOOLBAR_ID) return? Was the click event received? 

(I am also a little disappointed by the Robotium code here. I would prefer that getEditText returned null rather than asserting. We might need to work around that by getting all the views and searching them ourselves in cases like that in focusUrlBar. But first I would be more interested in exactly how these tests are failing -- why isn't the EditText available, why does the screenshot show an unexpected state?)
Flags: needinfo?(gbrown)
side note: Are we entertaining the thought of supporting local patchs / mozilla enhancements to robotium? Or simply coding around known issues? (I was also slightly unimpressed by the code assertions vs. returning meaninginful test results to callers... can we fix stuff ... ?)

I started out in this direction with bug 925264 which was similar. In that case I proved a fix with a generic waitForObject() routine that returns a value or null on timeout.

I'm still testing... hadn't looked at the vaule returned by mSolo.getView() as the next mSolo.clickOnView() either throws if provided a null view, or performs the click. "Was the click even received" is the current question.
(In reply to Mark Capella [:capella] from comment #69)
> side note: Are we entertaining the thought of supporting local patchs /
> mozilla enhancements to robotium? Or simply coding around known issues? (I
> was also slightly unimpressed by the code assertions vs. returning
> meaninginful test results to callers... can we fix stuff ... ?)

Robotium has been a fairly active project over the last year: They have released several new versions and we have updated several times. Maintaining our own patches and merging them with new releases of Robotium might become time-consuming. I suggest we work-around Robotium limitations and report issues/submit patches to Robotium where appropriate.
See Also: → 939626
The spike in testMasterPassword was caused by bug 846340. I've confirmed that a backout on Try is green after 100+ retriggers. Geoff, what do you want to do here? Backout or disable the test? Or can we get someone to look into what's happening here?
Blocks: 846340
status-b2g-v1.2: --- → wontfix
status-firefox28: --- → affected
status-firefox29: --- → affected
status-firefox-esr24: --- → wontfix
Flags: needinfo?(gbrown)
Ryan -- Thanks so much for tracking that down! That is a very surprising result, since that bug only modifies a different test (testClearPrivateData). Are database changes in one test affecting another? (I think they should not, since each test gets a fresh profile.)

Let's back out bug 846340 to demonstrate the curious result and motivate investigation.
Flags: needinfo?(gbrown)
(In reply to TBPL Robot from comment #193)
> RyanVM
> https://tbpl.mozilla.org/php/getParsedLog.php?id=31813562&tree=Mozilla-
> Inbound
> Android 4.0 Panda mozilla-inbound opt test robocop-2 on 2013-12-11 08:28:18
> revision: 63ae0ef30e3f
> slave: panda-0645
> 
> 16 INFO TEST-UNEXPECTED-FAIL | testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!
> 12-11 08:49:24.664 I/Robocop ( 4272): 16 INFO TEST-UNEXPECTED-FAIL |
> testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!
> 12-11 08:49:26.039 I/TestRunner( 4272):
> junit.framework.AssertionFailedError: 16 INFO TEST-UNEXPECTED-FAIL |
> testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!
> 12-11 08:49:24.664 I/Robocop ( 4272): 16 INFO TEST-UNEXPECTED-FAIL |
> testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!
> 12-11 08:49:26.039 I/TestRunner( 4272):
> junit.framework.AssertionFailedError: 16 INFO TEST-UNEXPECTED-FAIL |
> testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!
> Return code: 1
> 12-11 08:49:24.664 I/Robocop ( 4272): 16 INFO TEST-UNEXPECTED-FAIL |
> testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!
> 12-11 08:49:26.039 I/TestRunner( 4272):
> junit.framework.AssertionFailedError: 16 INFO TEST-UNEXPECTED-FAIL |
> testMasterPassword | Exception caught -
> junit.framework.AssertionFailedError: EditText is not found!

This is post-backout :(. Back to the drawing board.
There is an open robotium issue for this now: http://code.google.com/p/robotium/issues/detail?id=537. No solution, but we should keep an eye on that.
Looks like the robotium folks are looking for a steps to reproduce. Can we help them with that?
Status: ASSIGNED → NEW
Attached patch pause before clicking Save (obsolete) — — Splinter Review 
I do not fully understand what has gone wrong with testMasterPassword, but here's an effective fix, discovered mostly by trial and error. (I suspect that the button clicks are not succeeding, so we end up in an unexpected UI state later in the test when we try to access the EditText).

The waitForText() calls do not contribute to the fix directly, but I think they are a good idea: verify (and leave evidence in the log) that the buttons are present before clicking.

The sleep() turns this test green, and shorter sleeps appear ineffective. Compare:

https://tbpl.mozilla.org/?tree=Try&rev=b443fb2b6da2 (shorter)
https://tbpl.mozilla.org/?tree=Try&rev=96818d32bdcc (this patch)

Obviously this does not address the more general problem of infrequent EditText failures in various tests. I would [leave open] both to address the general, infrequent issue and to follow-up on this sleep().

:capella -- any concerns?
Attachment #8347793 - Flags: review?(jmaher)
Attachment #8347793 - Flags: feedback?(markcapella)
(In reply to Kevin Brosnan [:kbrosnan] from comment #214)
> Looks like the robotium folks are looking for a steps to reproduce. Can we
> help them with that?

testMasterPassword may be too complicated for a robotium issue report, and my recent experience with it suggests that the frequent failure we have here may be unrelated to any bug in robotium. The infrequent failure in various tests may be caused by a robotium bug, but we don't have a way to reproduce that.
Comment on attachment 8347793 [details] [diff] [review]
pause before clicking Save

No concerns from my part ... I was looking here when testShareLink and testNewTab were the issue, but then we disabled it and a few other tests. I was hoping tips I learned would help me return to this new testMasterPassword issue, but I've been off in a different direction and figured someone who'd made a more recent change that perhaps had triggered the oranges might be looking. I should have taken my name off sooner :(

Re;robotium, I had built my own version so I could do a little low level logging. Apparently, there's conditions in the robotium code that causes it to give up and throw an exception, vs. passing back some actionable return code, so patching the 3d party code became a small (but not attractive) idea.

I can repro this in volume unit-tests, either on my local machine or bunches of push to try. I get about one error for every 60-80 runs.

Last I remember, robotium is called to return an editText view to us when there should be one ! But, it searchs it's list of views and finds the list completely empty of an editText or any other view. (This was back a month ago)
Attachment #8347793 - Flags: feedback?(markcapella)
Assignee: markcapella → nobody
Comment on attachment 8347793 [details] [diff] [review]
pause before clicking Save

Review of attachment 8347793 [details] [diff] [review]:
-----------------------------------------------------------------

r- for the use of two different strings for comparison vs clicking.

::: mobile/android/base/tests/testMasterPassword.java
@@ +185,5 @@
>  
>          doorhangerDisplayed(LOGIN_URL);// Check that the doorhanger is displayed
> +
> +        // TODO: Remove this hack -- see bug 915449
> +        mSolo.sleep(2000);

thanks for the comment here- nobody likes this stuff, but this is automation.

@@ +194,1 @@
>                  mSolo.clickOnButton(item);

here we check for 'Save' and '^Save$'.  Can we check for the same thing?

@@ +199,1 @@
>                  mSolo.clickOnButton("OK");

Why do we wait for "^OK$" and then click on "OK"?

I would like to know if these can be the same.

I would like to see them defined as variables so if we change them in the future (as we did just now) we can edit one line of text.
Attachment #8347793 - Flags: review?(jmaher) → review-
I wasn't sure if a regex is valid in clickButton...it is.
Assignee: nobody → gbrown
Attachment #8347793 - Attachment is obsolete: true
Attachment #8348105 - Flags: review?(jmaher)
Attachment #8348105 - Flags: review?(jmaher) → review+
Assignee: gbrown → nobody
https://tbpl.mozilla.org/php/getParsedLog.php?id=32323475&tree=Fx-Team
Summary: Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark,testMasterPassword | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark,testMasterPassword,testBookmarksPage | Exception caught - junit.framework.AssertionFailedError: EditText is not found!
Summary: Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark,testMasterPassword,testBookmarksPage | Exception caught - junit.framework.AssertionFailedError: EditText is not found! → Intermittent testNewTab,testShareLink,testSearchSuggestions,testBookmark,testMasterPassword,testBookmarksPage,testBookmarksPanel | Exception caught - junit.framework.AssertionFailedError: EditText is not found!