1472032 - Intermittent reftest and jsreftest | application crashed [@ libc.so + 0x1c3dc] after application ran for longer than allowed maximum time

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: cbrindusan [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=185431376&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/UkwA9iRBSEqzbpEMM_kzkw/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/UkwA9iRBSEqzbpEMM_kzkw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2018-06-28T21:39:59.884Z]     INFO -  REFTEST TEST-START | http://10.0.2.2:8888/jsreftest/tests/jsreftest.html?test=test262/built-ins/AsyncFromSyncIteratorPrototype/return/result-object-error.js
[task 2018-06-28T21:39:59.886Z]     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8888/jsreftest/tests/jsreftest.html?test=test262/built-ins/AsyncFromSyncIteratorPrototype/return/result-object-error.js | 275 / 311 (88%)
[task 2018-06-28T21:40:11.421Z]     INFO -  REFTEST TEST-PASS | http://10.0.2.2:8888/jsreftest/tests/jsreftest.html?test=test262/built-ins/AsyncFromSyncIteratorPrototype/return/result-object-error.js | (LOAD ONLY)
[task 2018-06-28T21:40:11.421Z]     INFO -  REFTEST TEST-END | http://10.0.2.2:8888/jsreftest/tests/jsreftest.html?test=test262/built-ins/AsyncFromSyncIteratorPrototype/return/result-object-error.js
[task 2018-06-28T21:40:12.630Z]     INFO -  Browser unexpectedly found running. Killing...
[task 2018-06-28T21:40:12.630Z]     INFO -  REFTEST TEST-INFO | started process screentopng
[task 2018-06-28T21:40:13.273Z]     INFO -  REFTEST TEST-INFO | screentopng: exit 0
[task 2018-06-28T21:40:29.056Z]  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/jsreftest/tests/jsreftest.html?test=test262/built-ins/AsyncFromSyncIteratorPrototype/return/result-object-error.js | application ran for longer than allowed maximum time

Comment 1

[Cristian Brindusan [:cbrindusan]]

android-em-4-3-armv7-api16 opt reftest failure: 
https://treeherder.mozilla.org/logviewer.html#?job_id=185433633&repo=mozilla-inbound&lineNumber=3881

[task 2018-06-28T21:52:08.995Z]     INFO -  REFTEST TEST-START | http://10.0.2.2:8888/tests/layout/reftests/selection/extend-3a.html == http://10.0.2.2:8888/tests/layout/reftests/selection/extend-3-ref.html
[task 2018-06-28T21:52:08.996Z]     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/layout/reftests/selection/extend-3a.html | 531 / 535 (99%)
[task 2018-06-28T21:52:10.210Z]     INFO -  Browser unexpectedly found running. Killing...
[task 2018-06-28T21:52:10.211Z]     INFO -  REFTEST TEST-INFO | started process screentopng
[task 2018-06-28T21:52:10.683Z]     INFO -  REFTEST TEST-INFO | screentopng: exit 0
[task 2018-06-28T21:52:25.536Z]  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/selection/extend-3a.html == http://10.0.2.2:8888/tests/layout/reftests/selection/extend-3-ref.html | application ran for longer than allowed maximum time

https://treeherder.mozilla.org/logviewer.html#?job_id=185439089&repo=mozilla-inbound&lineNumber=3472 

[task 2018-06-28T22:27:44.707Z]     INFO -  REFTEST TEST-START | http://10.0.2.2:8888/tests/layout/reftests/scrolling/fixed-opacity-2.html == http://10.0.2.2:8888/tests/layout/reftests/scrolling/fixed-opacity-2.html?ref
[task 2018-06-28T22:27:44.708Z]     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/layout/reftests/scrolling/fixed-opacity-2.html | 424 / 535 (79%)
[task 2018-06-28T22:27:45.619Z]     INFO -  Browser unexpectedly found running. Killing...
[task 2018-06-28T22:27:45.619Z]     INFO -  REFTEST TEST-INFO | started process screentopng
[task 2018-06-28T22:27:46.155Z]     INFO -  REFTEST TEST-INFO | screentopng: exit 0
[task 2018-06-28T22:28:01.508Z]  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/scrolling/fixed-opacity-2.html == http://10.0.2.2:8888/tests/layout/reftests/scrolling/fixed-opacity-2.html?ref | application ran for longer than allowed

android-em-4-3-armv7-api16 debug
https://treeherder.mozilla.org/logviewer.html#?job_id=185439617&repo=mozilla-inbound&lineNumber=2731 

[task 2018-06-28T22:34:20.562Z]     INFO -  REFTEST TEST-START | http://10.0.2.2:8888/tests/layout/reftests/position-dynamic-changes/horizontal/toauto-leftN-widthA-rightA-2.html == http://10.0.2.2:8888/tests/layout/reftests/position-dynamic-changes/horizontal/toauto-leftN-widthA-rightA-2-ref.html
[task 2018-06-28T22:34:20.563Z]     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/layout/reftests/position-dynamic-changes/horizontal/toauto-leftN-widthA-rightA-2.html | 285 / 291 (97%)
[task 2018-06-28T22:34:21.472Z]     INFO -  Browser unexpectedly found running. Killing...
[task 2018-06-28T22:34:21.473Z]     INFO -  REFTEST TEST-INFO | started process screentopng
[task 2018-06-28T22:34:21.962Z]     INFO -  REFTEST TEST-INFO | screentopng: exit 0
[task 2018-06-28T22:34:36.817Z]  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/position-dynamic-changes/horizontal/toauto-leftN-widthA-rightA-2.html == http://10.0.2.2:8888/tests/layout/reftests/position-dynamic-changes/horizontal/toauto-leftN-widthA-rightA-2-ref.html | application ran for longer than allowed maximum time
[task 2018-06-28T22:34:36.818Z]     INFO -  INFO | automation.py | Application ran for: 1:30:40.692389

Comment 5

[Geoff Brown [:gbrown]]

Android jsreftests time out with "application ran for longer than allowed maximum time" when firefox runs for longer than 90 minutes; in this case we expect to also see:

  INFO | automation.py | Application ran for: 1:30+...

and see timestamps consistent with firefox running for 90 minutes or more. Following the timeout, firefox is killed and a crash report is generated. That mechanism appears to be working as expected: Intermittently, some jsreftest and reftest jobs are running for longer than intended.

One cause of long run-time is running too many tests in a test chunk. I reviewed several recent mozilla-central pushes and found all Android/opt jsreftest chunks routinely running in under 50 minutes (55% of max) and all Android/debug jsreftest chunks routinely running in under 65 minutes (72% of max). Although these are long-running tests, I resist adding more chunks because:
 -- even at 72% that's not really close to a timeout
 -- we already run opt in 40 chunks and debug in 100 chunks and there is overhead associated with each chunk

Another cause of long run-time is test hangs. I don't see any sign of test hangs in these logs (including dups). Tests are making progress when the time-out occurs -- they just needed more time.

Another cause of long run-time is slow execution. Sometimes tests on aws take longer than other times. This effect seems more pronounced in the Android emulator. See bug 1321605. Sometimes we have seen correlation with machine type, but these recent failures seem equally distributed between m3.xlarge and c3.xlarge. Sometimes we have seen correlation with aws region, but recent failures seem distributed across us-west-1, us-west-2, us-east-1, etc. We check the Android emulator bogomips at the start of the run to try to guard against this: If the bogomips are < 250, we abandon the test run and automatically retry with a new instance. Most of these recent failures report bogomips in the 300-350 range, which is a little slow (other runs are in the 400-500 range) but are often correlated with successful test runs.

Another possibility is that there is an intermittent slow-down in firefox - a background thread working too hard, or something like that. I'm not sure how we can definitely distinguish between intermittently slow machine and intermittently slow firefox.

I think these failures are basically bug 1321605, but I'll keep looking over logs.

If it is bug 1321605 and it is causing trouble, possible mitigations that come to mind are:
 - relax the 90 minute timeout, perhaps to ~120 minutes
 - a stricter (or better?) bogomips check
 - abandoning aws and running android reftests/jsreftest only on a faster platform, like the in-progress x86 emulator on packet.net

Comment 6

[Geoff Brown [:gbrown]]

{
	"from":"task",
	"groupby":["run.name"],
	"limit":1000,
	"select":[
		{
			"value":"action.duration",
			"aggregate":"average",
			"name":"average"
		},
		{"value":"action.duration","aggregate":"max","name":"max"},
		{"aggregate":"count"}
	],
	"where":{"and":[
		{"gte":{"task.run.start_time":{"date":"2018-06-26"}}},
		{"eq":{"state":"completed"}},
		{"eq":{"kind":"test"}},
		{"in":{"build.branch":["autoland","mozilla-inbound","mozilla-central"]}},
		{"regexp":{"run.name":"test-android.*reftest.*"}}
	]}
}

"run.name"	"average"	"max"	"count"
"test-android-em-4.3-arm7-api-16/debug-reftest-25"	2541.428085423594	3646.3629999160767	398
"test-android-em-4.3-arm7-api-16/opt-reftest-8"	2464.787898981812	3826.454999923706	396
"test-android-em-4.3-arm7-api-16/debug-reftest-22"	2926.809345180613	5529.375	394
"test-android-em-4.3-arm7-api-16/debug-reftest-24"	2696.062421320659	3641.9759998321533	394
"test-android-em-4.3-arm7-api-16/debug-reftest-1"	2876.1562117374674	6560.763000011444	392
"test-android-em-4.3-arm7-api-16/opt-reftest-14"	2694.440418366875	3920.167999982834	392
"test-android-em-4.3-arm7-api-16/debug-reftest-30"	3438.3234450048803	4258.057000160217	391
"test-android-em-4.3-arm7-api-16/debug-reftest-31"	2605.998634270085	4213.029000043869	391
"test-android-em-4.3-arm7-api-16/opt-reftest-17"	2599.7997365680803	4408.846999883652	391
"test-android-em-4.3-arm7-api-16/opt-reftest-27"	2476.5298005211384	3556.9619998931885	391
"test-android-em-4.3-arm7-api-16/opt-reftest-4"	1975.7552046031904	4832.473000049591	391
"test-android-em-4.3-arm7-api-16/debug-reftest-21"	2695.3026820580167	4399.5	390
"test-android-em-4.3-arm7-api-16/debug-reftest-26"	2465.991074362168	3497.4379999637604	390
"test-android-em-4.3-arm7-api-16/opt-reftest-10"	2415.5225128277752	4302.25	390
"test-android-em-4.3-arm7-api-16/opt-reftest-2"	2571.195989738978	3880.486000061035	390
"test-android-em-4.3-arm7-api-16/opt-reftest-24"	2414.8079820559574	3232.265000104904	390
"test-android-em-4.3-arm7-api-16/opt-reftest-5"	2069.864553844012	3056.093999862671	390
"test-android-em-4.3-arm7-api-16/debug-reftest-18"	2483.466676093006	4038.858999967575	389
"test-android-em-4.3-arm7-api-16/debug-reftest-4"	3203.7195141407396	6230.243000030518	389
"test-android-em-4.3-arm7-api-16/debug-reftest-42"	3401.617820045329	4785.923999786377	389
"test-android-em-4.3-arm7-api-16/debug-reftest-51"	2483.350079684147	4332.18799996376	389
"test-android-em-4.3-arm7-api-16/debug-reftest-12"	2438.569023195001	4116.746999979019	388
"test-android-em-4.3-arm7-api-16/debug-reftest-20"	2328.9040128816036	3178.946000099182	388
"test-android-em-4.3-arm7-api-16/opt-reftest-12"	2608.9833170101815	4772.955999851227	388
"test-android-em-4.3-arm7-api-16/opt-reftest-21"	4111.416118559149	5078.691999912262	388
"test-android-em-4.3-arm7-api-16/opt-reftest-23"	2214.461752579384	3193.8239998817444	388
"test-android-em-4.3-arm7-api-16/opt-reftest-6"	2362.3733634027008	4733.253999948502	388
"test-android-em-4.3-arm7-api-16/debug-reftest-14"	2699.633682170888	3966.3440001010895	387
"test-android-em-4.3-arm7-api-16/debug-reftest-32"	2659.357656324865	3490.151999950409	387
"test-android-em-4.3-arm7-api-16/debug-reftest-38"	2293.147609816657	3494.1310000419617	387
"test-android-em-4.3-arm7-api-16/debug-reftest-39"	2425.887705440669	5052.051999807358	387
"test-android-em-4.3-arm7-api-16/debug-reftest-41"	4161.678155042712	4922.050999879837	387
"test-android-em-4.3-arm7-api-16/debug-reftest-45"	2350.0298087794035	3398.2330000400543	387
"test-android-em-4.3-arm7-api-16/debug-reftest-53"	2486.1679638232986	3747.426999807358	387
"test-android-em-4.3-arm7-api-16/debug-reftest-7"	1987.5015503885827	4504.882999897003	387
"test-android-em-4.3-arm7-api-16/opt-reftest-28"	2833.2817054201464	3905.4779999256134	387
"test-android-em-4.3-arm7-api-16/debug-reftest-13"	2371.7883575147916	4728.614000082016	386
"test-android-em-4.3-arm7-api-16/debug-reftest-15"	2553.861098445759	5316.328000068665	386
"test-android-em-4.3-arm7-api-16/debug-reftest-23"	2600.8427901539776	3662.598999977112	386
"test-android-em-4.3-arm7-api-16/debug-reftest-28"	2612.6036968984754	3618.996999979019	386
"test-android-em-4.3-arm7-api-16/debug-reftest-29"	2487.7071113975553	5372.289000034332	386
"test-android-em-4.3-arm7-api-16/debug-reftest-34"	2603.599277201094	5447.898000001907	386
"test-android-em-4.3-arm7-api-16/debug-reftest-35"	2493.4661554363724	3936.8970000743866	386
"test-android-em-4.3-arm7-api-16/debug-reftest-37"	1932.2034974172325	3349.4249999523163	386
"test-android-em-4.3-arm7-api-16/debug-reftest-48"	2400.208124349772	3968.4919998645782	386
"test-android-em-4.3-arm7-api-16/debug-reftest-52"	2378.1729067298415	3641.8359999656677	386
"test-android-em-4.3-arm7-api-16/opt-reftest-16"	2677.7089067382517	5058.776999950409	386
"test-android-em-4.3-arm7-api-16/opt-reftest-19"	1991.3898678709188	4055.2380001544952	386
"test-android-em-4.3-arm7-api-16/debug-reftest-47"	2466.6519168791833	5321.451000213623	385
"test-android-em-4.3-arm7-api-16/debug-reftest-55"	2454.055664938766	5399.361999988556	385
"test-android-em-4.3-arm7-api-16/opt-reftest-18"	2688.889444155507	3882.239000082016	385
"test-android-em-4.3-arm7-api-16/opt-reftest-22"	2512.5047064905043	4514.195000171661	385
"test-android-em-4.3-arm7-api-16/opt-reftest-25"	2373.1805246755675	3101.713000059128	385
"test-android-em-4.3-arm7-api-16/opt-reftest-9"	2481.860166235713	5008.281000137329	385
"test-android-em-4.3-arm7-api-16/debug-reftest-11"	2338.1390312463045	3604.5849997997284	384
"test-android-em-4.3-arm7-api-16/debug-reftest-17"	2537.793708330641	3445.790999889374	384
"test-android-em-4.3-arm7-api-16/debug-reftest-54"	2603.7104010352245	4006.3420000076294	384
"test-android-em-4.3-arm7-api-16/debug-reftest-6"	2298.723223951956	3435.869999885559	384
"test-android-em-4.3-arm7-api-16/debug-reftest-16"	2474.3973472510556	3474.513000011444	383
"test-android-em-4.3-arm7-api-16/debug-reftest-2"	3141.081096606217	5844.016000032425	383
"test-android-em-4.3-arm7-api-16/debug-reftest-36"	2791.7279294956447	3733.2039999961853	383
"test-android-em-4.3-arm7-api-16/debug-reftest-49"	2360.305138378168	4207.313999891281	383
"test-android-em-4.3-arm7-api-16/debug-reftest-5"	2802.2063263583123	4179.328999996185	383
"test-android-em-4.3-arm7-api-16/debug-reftest-9"	2197.157973882735	3839.191999912262	383
"test-android-em-4.3-arm7-api-16/opt-reftest-13"	2497.635054824869	4076.8420000076294	383
"test-android-em-4.3-arm7-api-16/opt-reftest-20"	2370.3376292416697	3716.5759999752045	383
"test-android-em-4.3-arm7-api-16/opt-reftest-26"	2393.5348250654283	5074.802000045776	383
"test-android-em-4.3-arm7-api-16/opt-reftest-7"	2528.0218590071554	4047.680999994278	383
"test-android-em-4.3-arm7-api-16/debug-reftest-19"	2596.3299764332346	3944.816999912262	382
"test-android-em-4.3-arm7-api-16/debug-reftest-3"	2483.0629345553084	3639.9430000782013	382
"test-android-em-4.3-arm7-api-16/debug-reftest-8"	2260.85906283456	4264.162000179291	382
"test-android-em-4.3-arm7-api-16/opt-reftest-11"	2853.9014319393646	4859.631000041962	382
"test-android-em-4.3-arm7-api-16/opt-reftest-15"	2912.435589011427	5139.641999959946	382
"test-android-em-4.3-arm7-api-16/opt-reftest-3"	2397.0950575935904	3363.7009999752045	382
"test-android-em-4.3-arm7-api-16/debug-reftest-43"	3345.1236929123797	5551.978000164032	381
"test-android-em-4.3-arm7-api-16/debug-reftest-50"	2515.6869737488705	3751.930999994278	381
"test-android-em-4.3-arm7-api-16/debug-reftest-44"	2158.0683973701375	3233.9649999141693	380
"test-android-em-4.3-arm7-api-16/debug-reftest-10"	2208.614000001495	4352.276000022888	379
"test-android-em-4.3-arm7-api-16/debug-reftest-40"	2496.457292879477	3927.1100001335144	379
"test-android-em-4.3-arm7-api-16/debug-reftest-46"	2179.8429076564657	3071.9930000305176	379
"test-android-em-4.3-arm7-api-16/debug-reftest-56"	3131.841749334713	5176.191999912262	379
"test-android-em-4.3-arm7-api-16/opt-reftest-1"	2946.7607592533504	4957.029000043869	378
"test-android-em-4.3-arm7-api-16/debug-reftest-33"	2646.2549098417408	4249.284999847412	366
"test-android-em-4.3-arm7-api-16/debug-reftest-27"	2743.2106912244144	3762.1259999275208	353
"test-android-em-4.3-arm7-api-16/debug-jsreftest-45"	3129.725372876151	3870.486000061035	59
"test-android-em-4.3-arm7-api-16/debug-jsreftest-50"	3200.071779671362	3824.5499999523163	59
"test-android-em-4.3-arm7-api-16/debug-jsreftest-87"	3068.9598813461043	3724.6419999599457	59
"test-android-em-4.3-arm7-api-16/debug-jsreftest-21"	3179.2829138246075	3783.0620000362396	58
"test-android-em-4.3-arm7-api-16/debug-jsreftest-34"	3016.215000000493	3763.186000108719	58
"test-android-em-4.3-arm7-api-16/debug-jsreftest-88"	3044.0125344745043	4026.834000110626	58
"test-android-em-4.3-arm7-api-16/debug-jsreftest-20"	3034.772842114432	3837.1419999599457	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-27"	3037.0123508896745	3760.9140000343323	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-52"	3723.7164210687606	4420.669000148773	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-57"	3086.3768596690998	3939.8630001544952	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-60"	3047.934666671251	3520.0390000343323	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-66"	3070.805842136082	3698.2139999866486	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-67"	3056.706333344443	3845.173000097275	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-70"	3057.1187017675034	3689.146000146866	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-75"	3178.1447894740522	5627.37299990654	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-80"	3134.6883508698984	5427.599999904633	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-81"	3123.605701760242	4217.309000015259	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-89"	3030.7381754339785	3857.989000082016	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-90"	3135.257087720068	4731.2809998989105	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-93"	3031.014982453564	3486.061999797821	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-96"	3091.5808596569195	4876.71799993515	57
"test-android-em-4.3-arm7-api-16/opt-jsreftest-22"	2755.1487719427078	3693.573000192642	57
"test-android-em-4.3-arm7-api-16/opt-jsreftest-23"	2820.276771930226	4008.173000097275	57
"test-android-em-4.3-arm7-api-16/opt-jsreftest-34"	2709.0722105084806	3379.3849999904633	57
"test-android-em-4.3-arm7-api-16/opt-jsreftest-36"	2610.1882456060043	3099.427999973297	57
"test-android-em-4.3-arm7-api-16/debug-jsreftest-10"	3103.5446785986423	3906.986999988556	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-11"	3032.3543928563595	3638.4550001621246	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-17"	3155.631928563118	5228.082000017166	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-23"	3186.024839247976	3944.476000070572	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-28"	3069.9383036068507	4757.274000167847	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-29"	3031.5063571248734	3823.758999824524	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-3"	3487.1828392744064	4543.135999917984	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-30"	3080.8880714007787	4232.431999921799	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-32"	2970.900910696813	3682.621999979019	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-36"	3071.83032143116	4231.981999874115	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-37"	3050.786821424961	3816.481999874115	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-38"	3049.551017854895	3518.3129999637604	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-39"	3071.498000004462	5646.323999881744	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-4"	3013.1241249995574	3446.9000000953674	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-40"	3019.098071417638	3853.789999961853	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-43"	2954.9514821469784	3587.2750000953674	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-44"	3041.016946417945	3653.8289999961853	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-48"	3097.583089287792	5276.164000034332	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-53"	2904.2728214178765	3927.9299998283386	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-54"	3386.3259107087338	4593.8369998931885	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-56"	3157.447124992098	3765.5319998264313	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-58"	3037.9222857228347	4053.021000146866	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-6"	3169.1982321483747	4227.717000007629	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-61"	3154.0426607089385	4990.361999988556	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-64"	3113.350660720042	4033.4649999141693	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-68"	3010.582339282547	3749.4900000095367	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-69"	3034.3366964246547	3605.398999929428	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-7"	2954.8787142719543	3731.766000032425	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-72"	3145.8375357048853	4421.6890001297	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-73"	3102.422553603138	4057.339000225067	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-76"	3007.832178558622	3560.059000015259	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-78"	3514.674285718373	4474.5490000247955	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-79"	3102.457821411746	4194.055999994278	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-91"	3029.6781250025547	3816.1900000572205	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-92"	3009.426999998944	3893.941999912262	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-1"	2731.1947321466037	3609.2730000019073	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-11"	2683.6924464234285	3679.098999977112	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-12"	2612.1790714221343	3174.901000022888	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-13"	2694.507428586483	4023.3869998455048	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-14"	2618.1557142862252	3344.2599999904633	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-15"	2615.186535720314	3066.598000049591	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-16"	2669.834446417434	3294.927999973297	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-18"	2668.6415178392613	3435.010999917984	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-19"	2827.584124973842	4357.746999979019	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-2"	2682.1361607228005	3762.2030000686646	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-21"	3002.7039107212	3868.065999984741	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-24"	2692.4779464474746	3393.1330001354218	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-25"	2755.082142846925	3478.826999902725	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-26"	2759.0698035827704	4138.880000114441	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-28"	2730.6890000104904	5360.798000097275	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-29"	2746.2038035690784	3302.6689999103546	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-3"	2562.779410724129	3230.5620000362396	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-33"	2756.306249993188	3639.5009999275208	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-35"	2724.8659643062524	3425.6310000419617	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-37"	2634.1773035654	 3349.784999847412	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-39"	2677.431464288916	3358.111999988556	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-40"	2673.796732114894	3249.746999979019	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-5"	2755.8726964252337	3373.8940000534058	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-6"	2816.401017853192	3552.6240000724792	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-7"	2788.929732152394	3312.648999929428	56
"test-android-em-4.3-arm7-api-16/opt-jsreftest-8"	2775.686482148511	3287.986999988556	56
"test-android-em-4.3-arm7-api-16/debug-jsreftest-1"	3379.4031636628238	4244.158999919891	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-100"	2969.1492545344613	3447.1930000782013	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-15"	3136.00907274593	3968.941999912262	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-16"	3130.899672716314	3962.893000125885	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-18"	3154.1278545596383	5052.557000160217	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-2"	3083.2751272678374	3589.6029999256134	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-22"	3194.36992725459	4904.9800000190735	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-24"	3056.3463454549965	3666.159999847412	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-25"	3139.239981807362	3576.621999979019	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-33"	3008.784672715447	3823.7599999904633	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-35"	2953.0877999999307	3574.25200009346	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-41"	2953.1825999866833	3650.0220000743866	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-42"	3168.4212181698194	3943.7269999980927	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-46"	3193.478072712638	4564.003999948502	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-47"	3148.135818186673	 4114.2119998931885	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-51"	3169.383436350389	3826.8369998931885	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-63"	3011.385981802507	3378.036999940872	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-65"	3025.5042545361953	3467.6180000305176	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-71"	3080.2684363581916	3821.3339998722076	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-74"	3023.328527259827	5424.154000043869	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-77"	2941.2554363554173	3671.4020001888275	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-82"	3123.9308181762694	3909.941999912262	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-85"	3047.467854538831	3875.7560000419617	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-86"	3044.068581832539	3598.8980000019073	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-94"	3048.4582181843844	3577.771000146866	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-95"	3048.4757454655387	3549.816999912262	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-97"	3133.817890925841	4021.746999979019	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-98"	3139.09869091294	5410.856000185013	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-99"	3066.3333454652266	4716.483000040054	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-17"	2757.7050545562397	3360.8899998664856	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-27"	2693.2292545318605	3725.351000070572	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-30"	2683.342672733827	3148.326999902725	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-31"	2578.8140181758185	3317.927999973297	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-32"	2887.571054549651	3470.4530000686646	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-38"	2709.5416545347734	3348.4779999256134	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-4"	2960.8640908934854	3512.4609999656677	55
"test-android-em-4.3-arm7-api-16/opt-jsreftest-9"	2825.4630545399405	3183.1480000019073	55
"test-android-em-4.3-arm7-api-16/debug-jsreftest-12"	3121.5929629581947	3604.044000148773	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-13"	3185.9063518444696	3853.220999956131	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-14"	3119.1135185250528	4064.0570001602173	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-26"	2946.9964259244775	3386.6399998664856	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-31"	3045.436685200091	3462.608999967575	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-49"	3239.5299074208297	4161.5090000629425	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-55"	2986.752925934615	3600.433000087738	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-59"	3178.317055552094	5115.528000116348	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-62"	3072.6309814762185	3903.6760001182556	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-83"	2967.306518514951	3768.3209998607635	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-84"	3033.0920925979262	3714.8010001182556	54
"test-android-em-4.3-arm7-api-16/opt-jsreftest-20"	2892.3903148085983	4424.911000013351	54
"test-android-em-4.3-arm7-api-16/debug-jsreftest-19"	3139.2147547074082	3808.3519999980927	53
"test-android-em-4.3-arm7-api-16/debug-jsreftest-5"	3140.994415089769	3799.6630001068115	53
"test-android-em-4.3-arm7-api-16/debug-jsreftest-9"	3400.7522452867256	4194.648000001907	53
"test-android-em-4.3-arm7-api-16/opt-jsreftest-10"	2838.124490584967	4945.058000087738	53
"test-android-em-4.3-arm7-api-16/debug-jsreftest-8"	3061.086653846961	4297.061000108719	52

Comment 16

[Geoff Brown [:gbrown]]

Let's try a simple time-out increase and see where that gets us...

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/75290027d14b
Increase Android browser time-out to 110 minutes; r=me,a=test-only

Comment 18

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/75290027d14b

Comment 19

[Intermittent Failures Robot]

10 failures in 718 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 5
* try: 4
* autoland: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 6
* linux64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1472032&startday=2018-06-25&endday=2018-07-01&tree=trunk

Comment 33

[Geoff Brown [:gbrown]]

Since the change in comment 18, there have been no new failures of this type (there have been some mis-stars).

This type of failure remains a risk and will re-occur, but this "outbreak" appears to be under control.

Comment 34

[Intermittent Failures Robot]

1 failures in 622 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1472032&startday=2018-07-02&endday=2018-07-08&tree=trunk

Comment 35

[Intermittent Failures Robot]

1 failures in 653 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1472032&startday=2018-07-16&endday=2018-07-22&tree=trunk

Comment 36

[Intermittent Failures Robot]

1 failures in 760 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1472032&startday=2018-07-23&endday=2018-07-29&tree=trunk