1164996 - Some Android 4.3 Debug xpcshell tests run for too long

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

A few xpcshell tests fail on Android 4.3 Debug because of timeouts. 

http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/[e-mail]-0eeb9fffb720/try-android-api-11-debug/try_ubuntu64_vm_armv7_mobile-debug_test-xpcshell-1-bm123-tests1-linux64-build30.txt.gz

10:34:29     INFO -  TEST-START | intl/uconv/tests/unit/test_decode_EUCKR_Hangul.js
10:39:29  WARNING -  TEST-UNEXPECTED-TIMEOUT | intl/uconv/tests/unit/test_decode_EUCKR_Hangul.js | Test timed out
10:39:29     INFO -  TEST-INFO took 300047ms

There are about 5 tests that frequently time out this way on 4.3 Debug.

Unlike mochitests and reftests on this platform, xpcshell *jobs* take about the same time as they do on Opt (just 5 to 10% longer).

If I increase the xpcshell test timeout from 5 minutes to 10 minutes, all timeouts are avoided and the jobs are green.

Comment 1

[Geoff Brown [:gbrown] (pto Apr 11-18)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da7e57d7e7ef&exclusion_profile=false has the per-test timeout increased to 10 minutes, instead of the normal 5 minutes. This avoids all the per-test timeouts.


> Unlike mochitests and reftests on this platform, xpcshell *jobs* take about the same time as 
> they do on Opt (just 5 to 10% longer).

Oops -- that's not right at all. That same try job demonstrates that each Debug chunk takes about twice the time of the corresponding Opt chunk. eg. Opt X1 took 50 minutes; various Debug X1 retries took between 94 and 109 minutes.

Comment 2

[Geoff Brown [:gbrown] (pto Apr 11-18)]

:kmoir -- All remaining 4.3 Debug xpcshell test failures are timing related. There are only a few individual test timeouts, which could be handled with increased timeouts in the harness. However, because the overall job times are also quite long -- about 2 hours per chunk for 3 chunks -- I think it would be best to run 4.3 Debug xpcshell on c3.xlarge instead. Is that okay?

Comment 3

[Kim Moir [:kmoir] ET]

I think that's fine.  Do you want me to enable them on c3.xlarge on try before we enable them on trunk?

Comment 4

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Yes please -- try only for now! 

I have verified that they run green if I increase timeouts, and I assume that a faster cpu will have the same effect, but I have not actually seen them run on c3.xlarge.

Comment 5

[Kim Moir [:kmoir] ET]

gbrown: They seem to be green on try, do you want them to be unhidden or enabled on other branches?

Also, I'll keep on eye on the load for this pool of machines and see if we need to enable more in our configs.

Comment 6

[Geoff Brown [:gbrown] (pto Apr 11-18)]

They look good to me. Thanks for switching to c3.xlarge.

Yes, please unhide, run on all trunk trees, and retire Android 4.0 Debug xpcshell.