773074 - Robocop verifyUrl is timing sensitive

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

Robocop's BaseTest.verifyUrl checks to see whether the url in the awesome bar matches a specified string. Expected usage might be:
<do something to load a page with url X>
blockForEvent("DOMContentLoaded")
verifyUrl(X)

And that usually works...but not always, since the awesome bar view may not be updated yet. 

Most of the tests that currently use verifyUrl do something else between loading the page and verifying the url:

<do something to load a page with url X>
blockForEvent("DOMContentLoaded")
<scroll page, or something>
verifyUrl(X)

so that increases the delay before reading the awesome bar, and increases the chances of success.

testHistoryTab does not have that sort of delay and sometimes encounters a problem, where the previous url is read:

20 INFO TEST-UNEXPECTED-FAIL | testHistoryTab | Awesomebar URL stayed the same - got http://mochi.test:8888/tests/robocop/robocop_big_link.html, expected http://mochi.test:8888/tests/robocop/robocop_blank_01.html
Exception caught during test!
junit.framework.AssertionFailedError: 20 INFO TEST-UNEXPECTED-FAIL | testHistoryTab | Awesomebar URL stayed the same - got http://mochi.test:8888/tests/robocop/robocop_big_link.html, expected http://mochi.test:8888/tests/robocop/robocop_blank_01.html
        at junit.framework.Assert.fail(Assert.java:47)
        at org.mozilla.fennec_mozdev.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
        at org.mozilla.fennec_mozdev.FennecMochitestAssert.ok(FennecMochitestAssert.java:137)
        at org.mozilla.fennec_mozdev.FennecMochitestAssert.is(FennecMochitestAssert.java:143)
        at org.mozilla.fennec_mozdev.tests.BaseTest.verifyUrl(BaseTest.java:197)
        at org.mozilla.fennec_mozdev.tests.testHistoryTab.testClick(testHistoryTab.java:178)
        at org.mozilla.fennec_mozdev.tests.testHistoryTab.testHistoryTab(testHistoryTab.java:61)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:521)
        at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
        at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
        at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
        at org.mozilla.fennec_mozdev.tests.BaseTest.runTest(BaseTest.java:93)
        at junit.framework.TestCase.runBare(TestCase.java:127)
        at junit.framework.TestResult$1.protect(TestResult.java:106)
        at junit.framework.TestResult.runProtected(TestResult.java:124)
        at junit.framework.TestResult.run(TestResult.java:109)
        at junit.framework.TestCase.run(TestCase.java:118)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
        at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
        at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1449)


I think the proper fix is to make verifyUrl more robust -- wait for some time for the expected response before failing.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: wait for a short time in BaseTest.verifyUrl for the expected text

I haven't been able to demonstrate that this change helps testHistoryTab -- there may be something else going wrong there. Still, I think this is a good change and eliminates a possible source of failure.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 643214 [details] [diff] [review]
wait for a short time in BaseTest.verifyUrl for the expected text

Review of attachment 643214 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/base/tests/BaseTest.java.in
@@ +199,3 @@
>              urlbarText = urlbar.getText();
>          }
>          mAsserter.is(urlbarText, url, "Awesomebar URL stayed the same");

This effectively does the same thing twice.  Not sure I like it so much, but I can't think of a better way to detect what is in the url bar since it can be blank.

Comment 3

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3923522569e1

Comment 4

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/3923522569e1