Closed Bug 1401035 Opened 7 years ago Closed 7 years ago

Android reftest "application ran for longer than allowed maximum time" handled poorly

Categories

(Firefox for Android Graveyard :: Testing, defect, P1)

Product:
Firefox for Android Graveyard
Component:
Testing
Type:
defect

Tracking

(firefox57 fixed, firefox58 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 58
Tracking Flags:
Tracking Status
firefox57 --- fixed
firefox58 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(1 file)

avoid restart after application time out
827 bytes, patch
jmaher
: review+
Details | Diff | Splinter Review 
Consider

https://treeherder.mozilla.org/logviewer.html#?job_id=128553644&repo=mozilla-inbound&lineNumber=3153

The time out is reported, the browser is killed, a dump generated and reported -- and then the browser is restarted and the whole suite restarted:

[task 2017-09-05T17:33:40.110348Z] 17:33:40     INFO -  0x5a276000 - 0x5a27afff  gralloc.goldfish.so  ???
[task 2017-09-05T17:33:40.111914Z] 17:33:40     INFO -  0x5a608000 - 0x5a624fff  Roboto-Light.ttf  ???
[task 2017-09-05T17:33:40.113219Z] 17:33:40     INFO -  0x5ab42000 - 0x5ab49fff  browser.db-shm  ???
[task 2017-09-05T17:33:40.114646Z] 17:33:40     INFO -  0x5ad4a000 - 0x5f09bfff  libxul.so  ???
[task 2017-09-05T17:33:40.116434Z] 17:33:40     INFO -  REFTEST INFO | Process mode: non-e10s
[task 2017-09-05T17:33:40.120428Z] 17:33:40     INFO -  leakcheck | refcount logging is off, so leaks can't be detected!
[task 2017-09-05T17:34:11.819431Z] 17:34:11     INFO -  WARNING: Continuing after a crash is not supported for remote reftest yet.
[task 2017-09-05T17:34:33.880580Z] 17:34:33     INFO -  REFTEST INFO | Running with e10s: False
[task 2017-09-05T17:34:56.716972Z] 17:34:56     INFO -  INFO | automation.py | Application pid: 3111
[task 2017-09-05T17:39:38.395717Z] 17:39:38     INFO -  REFTEST INFO | Reading manifest http://10.0.2.2:8854/tests/layout/reftests/reftest.list
[task 2017-09-05T17:40:43.121077Z] 17:40:43     INFO -  REFTEST WARNING | Failed to find the test-plugin.
[task 2017-09-05T17:40:43.129021Z] 17:40:43     INFO -  REFTEST INFO | Dumping JSON representation of sandbox
[task 2017-09-05T17:40:43.133959Z] 17:40:43     INFO -  REFTEST INFO | {"isDebugBuild":true,"xulRuntime":{"widgetToolkit":"android","OS":"Android","XPCOMABI":"arm-eabi-gcc3"},"smallScreen":false,"d2d":false,"dwrite":false,"gpuProcess":false,"azureCairo":false,"azureSkia":true,"skiaContent":true,"azureSkiaGL":1,"contentSameGfxBackendAsCanvas":true,"layersGPUAccelerated":true,"d3d11":false,"d3d9":false,"layersOpenGL":true,"webrender":false,"layersOMTC":true,"advancedLayers":false,"Android":true,"cocoaWidget":false,"gtkWidget":false,"qtWidget":false,"winWidget":false,"transparentScrollbars":false,"AndroidVersion":18,"AddressSanitizer":false,"webrtc":true,"stylo":false,"styloVsGecko":false,"release_or_beta":false,"http":{"userAgent":"Mozilla/5.0 (Android 4.3.1; Mobile; rv:57.0) Gecko/57.0 Firefox/57.0","appName":"Mozilla","appVersion":"5.0","platform":"Android 4.3.1","oscpu":"Linux armv7l","misc":"rv:57.0"},"haveTestPlugin":false,"windowsDefaultTheme":false,"nativeThemePref":false,"gpuProcessForceEnabled":false,"prefs":{},"browserIsRemote":false,"asyncPan":true,"usesRepeatResampling":false}
[task 2017-09-05T17:40:53.819360Z] 17:40:53     INFO -  REFTEST WARNING | Failed to find the test-plugin.
[task 2017-09-05T17:41:21.854073Z] 17:41:21     INFO -  REFTEST WARNING | Failed to find the test-plugin.

...

[task 2017-09-05T17:49:33.686685Z] 17:49:33     INFO -  REFTEST WARNING | Failed to find the test-plugin.
[task 2017-09-05T17:49:44.083073Z] 17:49:44     INFO -  REFTEST WARNING | Failed to find the test-plugin.
[task 2017-09-05T17:50:07.408293Z] 17:50:07     INFO -  REFTEST ERROR | Got second suite_start message before suite_end. Logged with data: {"tests": {"default": ["data:text/html,<body> == about:blank", "data:text/plain, == about:blank", "d
[task 2017-09-05T17:50:07.412700Z] 17:50:07     INFO -  REFTEST INFO | Running chunk 21 out of 48 chunks.  tests 6680-6989/15642
[task 2017-09-05T17:50:07.419392Z] 17:50:07     INFO -  REFTEST TEST-START | http://10.0.2.2:8854/tests/layout/reftests/css-blending/background-blending-image-gradient.html == http://10.0.2.2:8854/tests/layout/reftests/css-blending/background-blending-gradient-color-ref.html
[task 2017-09-05T17:50:07.420136Z] 17:50:07     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/css-blending/background-blending-image-gradient.html | 0 / 310 (0%)

...

until the task runs out of time

[taskcluster:error] Task timeout after 10800 seconds. Force killing container.
[taskcluster 2017-09-05 18:15:35.920Z] === Task Finished ===
[taskcluster 2017-09-05 18:15:35.921Z] Unsuccessful task run with exit code: -1 completed in 11708.578 seconds

so then artifacts are not collected and I cannot diagnose the performance stats!!
This particularly looks suspect:

  WARNING: Continuing after a crash is not supported for remote reftest yet.
Blocks: 1344991
See Also: → 1397017
See Also: → 1397143
See Also: → 1397999
See Also: → 1397997
See Also: → 1401010
See Also: → 1398014
See Also: → 1397567
See Also: → 1395055
See Also: → 1395050
Blocks: 1204281
It appears that the desktop reftest harness has no "application max run time" time out, so this is only an Android problem.

By returning TEST_SEEN_FINAL here, the restart logic is tricked into not restarting.

In https://treeherder.mozilla.org/#/jobs?repo=try&revision=afa8819156e88cf140e7ba436b061e9bfa019628, I force an application time out by setting the max run time to a small value. Instead of restarting, the test job ends after the time out - great!
Attachment #8911974 - Flags: review?(jmaher)
Comment on attachment 8911974 [details] [diff] [review]
avoid restart after application time out

Review of attachment 8911974 [details] [diff] [review]:
-----------------------------------------------------------------

so many exceptions!
Attachment #8911974 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e18f56d0dbd3
Avoid Android reftest restart after max run time exceeded; r=jmaher
Priority: -- → P1
See Also: → 1402639
https://hg.mozilla.org/mozilla-central/rev/e18f56d0dbd3
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox58: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 58
Blocks: 1411358
No longer blocks: 1411358
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: