Make GeckoSessionTestRule more reliable on automation

RESOLVED FIXED in Firefox 61

Status

defect
P1
normal
RESOLVED FIXED
a year ago
5 months ago

People

(Reporter: jchen, Assigned: jchen)

Tracking

Trunk
mozilla61
All
Android

Firefox Tracking Flags

(firefox61 fixed)

Details

Attachments

(3 attachments)

Fix several aspects of GeckoSessionTestRule to make it more reliable when running under the ARM emulators on automation.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 4

a year ago
mozreview-review
Comment on attachment 8970342 [details]
Bug 1456209 - 3. Scale timeouts based on device;

https://reviewboard.mozilla.org/r/239134/#review244802

::: mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java:80
(Diff revision 1)
>  public class GeckoSessionTestRule extends UiThreadTestRule {
>      private static final String LOGTAG = "GeckoSessionTestRule";
>  
>      private static final long DEFAULT_TIMEOUT_MILLIS = 10000;
> +    private static final long DEFAULT_ARM_DEVICE_MILLIS = 30000;
> +    private static final long DEFAULT_ARM_EMULATOR_MILLIS = 120000;

Seems like a long time. If you can reliably use something shorter, I encourage it.

::: mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java:436
(Diff revision 1)
>          public boolean isDebugging() {
>              return Debug.isDebuggerConnected();
>          }
> +
> +        public boolean isEmulator() {
> +            return "generic".equals(Build.DEVICE) || Build.DEVICE.startsWith("generic_");

Why not startsWith("generic")?
Attachment #8970342 - Flags: review?(gbrown) → review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(In reply to Geoff Brown [:gbrown] from comment #4)
> Comment on attachment 8970342 [details]
> Bug 1456209 - 3. Scale timeouts based on device;
> 
> >      private static final long DEFAULT_TIMEOUT_MILLIS = 10000;
> > +    private static final long DEFAULT_ARM_DEVICE_MILLIS = 30000;
> > +    private static final long DEFAULT_ARM_EMULATOR_MILLIS = 120000;
> 
> Seems like a long time. If you can reliably use something shorter, I
> encourage it.

I measured a debug test run on ARM emulator, and the actual maximum wait was about 60s, so a 120s timeout seems to be a safe choice.

> :::
> mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/
> rule/GeckoSessionTestRule.java:436
> (Diff revision 1)
> >          public boolean isDebugging() {
> >              return Debug.isDebuggerConnected();
> >          }
> > +
> > +        public boolean isEmulator() {
> > +            return "generic".equals(Build.DEVICE) || Build.DEVICE.startsWith("generic_");
> 
> Why not startsWith("generic")?

Just to be safe I guess. The emulator devices are either "generic", or "generic_foo".
(Assignee)

Comment 9

a year ago
mozreview-review
Comment on attachment 8970340 [details]
Bug 1456209 - 1. Fix test license headers;

https://reviewboard.mozilla.org/r/239130/#review244878
Attachment #8970340 - Flags: review+
(Assignee)

Comment 10

a year ago
mozreview-review
Comment on attachment 8970341 [details]
Bug 1456209 - 2. Miscellaneous GeckoSessionTestRule changes;

https://reviewboard.mozilla.org/r/239132/#review244880
Attachment #8970341 - Flags: review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 17

a year ago
Pushed by nchen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/041289f66ad9
1. Fix test license headers; r=jchen
https://hg.mozilla.org/integration/autoland/rev/2a11c1864d44
2. Miscellaneous GeckoSessionTestRule changes; r=jchen
https://hg.mozilla.org/integration/autoland/rev/b89f665eed0f
3. Scale timeouts based on device; r=gbrown
Backed out 3 changesets (bug 1456209) for bustage in /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoSession.java:22 on a CLOSED TREE

Problematic push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=d3dc4f7a2080215ba14b0cc8e15a4e323c34a88a&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception
Bustage: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&selectedJob=175266838
Backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=d3dc4f7a2080215ba14b0cc8e15a4e323c34a88a&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception
Log: https://treeherder.mozilla.org/logviewer.html#?job_id=175266838&repo=autoland&lineNumber=34678

[task 2018-04-24T05:57:09.732Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoSession.java:22: error: cannot find symbol
[task 2018-04-24T05:57:09.732Z] 05:57:09     INFO -  import org.mozilla.gecko.IGeckoEditableParent;
[task 2018-04-24T05:57:09.732Z] 05:57:09     INFO -                          ^
[task 2018-04-24T05:57:09.732Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.732Z] 05:57:09     INFO -    location: package org.mozilla.gecko
[task 2018-04-24T05:57:09.772Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/SessionTextInput.java:10: error: cannot find symbol
[task 2018-04-24T05:57:09.772Z] 05:57:09     INFO -  import org.mozilla.gecko.IGeckoEditableParent;
[task 2018-04-24T05:57:09.773Z] 05:57:09     INFO -                          ^
[task 2018-04-24T05:57:09.773Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.774Z] 05:57:09     INFO -    location: package org.mozilla.gecko
[task 2018-04-24T05:57:09.793Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:16: error: cannot find symbol
[task 2018-04-24T05:57:09.793Z] 05:57:09     INFO -  import org.mozilla.gecko.IGeckoEditableChild;
[task 2018-04-24T05:57:09.793Z] 05:57:09     INFO -                          ^
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -    location: package org.mozilla.gecko
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:17: error: cannot find symbol
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -  import org.mozilla.gecko.IGeckoEditableParent;
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -                          ^
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -    location: package org.mozilla.gecko
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:53: error: package IGeckoEditableParent does not exist
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -      extends IGeckoEditableParent.Stub
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -                                  ^
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoEditableChild.java:24: error: cannot find symbol
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -  public final class GeckoEditableChild extends JNIObject implements IGeckoEditableChild {
[task 2018-04-24T05:57:09.796Z] 05:57:09     INFO -                                                                     ^
[task 2018-04-24T05:57:09.797Z] 05:57:09     INFO -    symbol: class IGeckoEditableChild
[task 2018-04-24T05:57:09.803Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoSession.java:577: error: cannot find symbol
[task 2018-04-24T05:57:09.803Z] 05:57:09     INFO -          public native void attachEditable(IGeckoEditableParent parent,
[task 2018-04-24T05:57:09.803Z] 05:57:09     INFO -                                            ^
[task 2018-04-24T05:57:09.804Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.804Z] 05:57:09     INFO -    location: class Window
[task 2018-04-24T05:57:09.813Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:76: error: cannot find symbol
[task 2018-04-24T05:57:09.814Z] 05:57:09     INFO -      /* package */ IGeckoEditableChild mDefaultChild; // Used by IC thread.
[task 2018-04-24T05:57:09.815Z] 05:57:09     INFO -                    ^
[task 2018-04-24T05:57:09.815Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.816Z] 05:57:09     INFO -    location: class GeckoEditable
[task 2018-04-24T05:57:09.816Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:78: error: cannot find symbol
[task 2018-04-24T05:57:09.816Z] 05:57:09     INFO -      /* package */ IGeckoEditableChild mFocusedChild; // Used by IC thread.
[task 2018-04-24T05:57:09.817Z] 05:57:09     INFO -                    ^
[task 2018-04-24T05:57:09.817Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.817Z] 05:57:09     INFO -    location: class GeckoEditable
[task 2018-04-24T05:57:09.818Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:108: error: cannot find symbol
[task 2018-04-24T05:57:09.818Z] 05:57:09     INFO -      private void onKeyEvent(final IGeckoEditableChild child, KeyEvent event, int action,
[task 2018-04-24T05:57:09.818Z] 05:57:09     INFO -                                    ^
[task 2018-04-24T05:57:09.819Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.819Z] 05:57:09     INFO -    location: class GeckoEditable
[task 2018-04-24T05:57:09.819Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:626: error: cannot find symbol
[task 2018-04-24T05:57:09.819Z] 05:57:09     INFO -      public void setDefaultEditableChild(final IGeckoEditableChild child) {
[task 2018-04-24T05:57:09.820Z] 05:57:09     INFO -                                                ^
[task 2018-04-24T05:57:09.820Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.820Z] 05:57:09     INFO -    location: class GeckoEditable
[task 2018-04-24T05:57:09.820Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java:1129: error: cannot find symbol
[task 2018-04-24T05:57:09.820Z] 05:57:09     INFO -      public void notifyIME(final IGeckoEditableChild child, final int type) {
[task 2018-04-24T05:57:09.820Z] 05:57:09     INFO -                                  ^
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -    location: class GeckoEditable
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoEditableChild.java:73: error: cannot find symbol
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -      private final IGeckoEditableParent mEditableParent;
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -                    ^
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.821Z] 05:57:09     INFO -    location: class GeckoEditableChild
[task 2018-04-24T05:57:09.822Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoEditableChild.java:74: error: cannot find symbol
[task 2018-04-24T05:57:09.822Z] 05:57:09     INFO -      private final IGeckoEditableChild mEditableChild;
[task 2018-04-24T05:57:09.822Z] 05:57:09     INFO -                    ^
[task 2018-04-24T05:57:09.822Z] 05:57:09     INFO -    symbol:   class IGeckoEditableChild
[task 2018-04-24T05:57:09.822Z] 05:57:09     INFO -    location: class GeckoEditableChild
[task 2018-04-24T05:57:09.823Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoEditableChild.java:79: error: cannot find symbol
[task 2018-04-24T05:57:09.823Z] 05:57:09     INFO -      public GeckoEditableChild(final IGeckoEditableParent editableParent) {
[task 2018-04-24T05:57:09.823Z] 05:57:09     INFO -                                      ^
[task 2018-04-24T05:57:09.823Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.824Z] 05:57:09     INFO -    location: class GeckoEditableChild
[task 2018-04-24T05:57:09.945Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoEditableChild.java:31: error: package IGeckoEditableChild does not exist
[task 2018-04-24T05:57:09.945Z] 05:57:09     INFO -      private final class RemoteChild extends IGeckoEditableChild.Stub {
[task 2018-04-24T05:57:09.946Z] 05:57:09     INFO -                                                                 ^
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/process/GeckoProcessManager.java:9: error: cannot find symbol
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -  import org.mozilla.gecko.IGeckoEditableParent;
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -                          ^
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -    location: package org.mozilla.gecko
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/process/GeckoProcessManager.java:28: error: package IProcessManager does not exist
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -  public final class GeckoProcessManager extends IProcessManager.Stub {
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -                                                                ^
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/process/GeckoProcessManager.java:37: error: cannot find symbol
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -      private static native IGeckoEditableParent nativeGetEditableParent(long contentId,
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -                            ^
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -    location: class GeckoProcessManager
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/process/GeckoProcessManager.java:41: error: cannot find symbol
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -      public IGeckoEditableParent getEditableParent(final long contentId, final long tabId) {
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -             ^
[task 2018-04-24T05:57:09.959Z] 05:57:09     INFO -    symbol:   class IGeckoEditableParent
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -    location: class GeckoProcessManager
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/process/GeckoProcessManager.java:51: error: cannot find symbol
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -          private IChildProcess mChild;
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -                  ^
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -    symbol:   class IChildProcess
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -    location: class ChildConnection
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -  /builds/worker/workspace/build/src/mobile/android/geckoview/src/main/java/org/mozilla/gecko/process/GeckoProcessManager.java:69: error: cannot find symbol
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -          public synchronized IChildProcess bind() {
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -                              ^
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -    symbol:   class IChildProcess
[task 2018-04-24T05:57:09.960Z] 05:57:09     INFO -    location: class ChildConnection
[task 2018-04-24T05:57:10.972Z] 05:57:10     INFO -  22 warnings
[task 2018-04-24T05:57:11.023Z] 05:57:11     INFO -  :geckoview:javadocJarOfficialWithGeckoBinariesNoMinApiRelease
[task 2018-04-24T05:57:11.164Z] 05:57:11     INFO -  :geckoview:sourcesJarOfficialWithGeckoBinariesNoMinApiRelease
[task 2018-04-24T05:57:11.869Z] 05:57:11     INFO -  :geckoview:uploadArchivesCould not find metadata org.mozilla:geckoview-default-arm64-v8a/maven-metadata.xml in remote (file:///builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/maven)
[task 2018-04-24T05:57:11.890Z] 05:57:11     INFO -  FAILURE: Build failed with an exception.
[task 2018-04-24T05:57:11.890Z] 05:57:11     INFO -  * What went wrong:
[task 2018-04-24T05:57:11.890Z] 05:57:11     INFO -  Execution failed for task ':geckoview:compileOfficialWithGeckoBinariesNoMinApiDebugAndroidTestKotlin'.
[task 2018-04-24T05:57:11.890Z] 05:57:11     INFO -  > Compilation error. See log for more details
[task 2018-04-24T05:57:11.890Z] 05:57:11     INFO -  * Try:
[task 2018-04-24T05:57:11.891Z] 05:57:11     INFO -  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
[task 2018-04-24T05:57:11.891Z] 05:57:11     INFO -  * Get more help at https://help.gradle.org
[task 2018-04-24T05:57:11.891Z] 05:57:11     INFO -  BUILD FAILED in 37s
[task 2018-04-24T05:57:11.891Z] 05:57:11     INFO -  123 actionable tasks: 110 executed, 13 up-to-date
[task 2018-04-24T05:57:12.504Z] 05:57:12    ERROR - Return code: 1
[task 2018-04-24T05:57:12.504Z] 05:57:12    ERROR - 1 not in success codes: [0]
[task 2018-04-24T05:57:12.504Z] 05:57:12  WARNING - setting return code to 2
[task 2018-04-24T05:57:12.504Z] 05:57:12    FATAL - Halting on failure while running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'android', 'archive-geckoview']
[task 2018-04-24T05:57:12.505Z] 05:57:12    FATAL - Running post_fatal callback...
[task 2018-04-24T05:57:12.505Z] 05:57:12    FATAL - Exiting 2
[task 2018-04-24T05:57:12.505Z] 05:57:12     INFO - [mozharness: 2018-04-24 05:57:12.505277Z] Finished build step (failed)
[task 2018-04-24T05:57:12.505Z] 05:57:12     INFO - Running post-run listener: _summarize
[task 2018-04-24T05:57:12.505Z] 05:57:12    ERROR - # TBPL FAILURE #
[task 2018-04-24T05:57:12.505Z] 05:57:12     INFO - [mozharness: 2018-04-24 05:57:12.505590Z] FxDesktopBuild summary:
[task 2018-04-24T05:57:12.505Z] 05:57:12    ERROR - # TBPL FAILURE #
Flags: needinfo?(nchen)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 22

a year ago
Pushed by nchen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/65fb23b3c5e0
1. Fix test license headers; r=jchen
https://hg.mozilla.org/integration/autoland/rev/f6dd07476c05
2. Miscellaneous GeckoSessionTestRule changes; r=jchen
https://hg.mozilla.org/integration/autoland/rev/bb4f5395f6dc
3. Scale timeouts based on device; r=gbrown
Flags: needinfo?(nchen)

Updated

5 months ago
Product: Firefox for Android → GeckoView
Target Milestone: Firefox 61 → mozilla61
You need to log in before you can comment on or make changes to this bug.