1345350 - Intermittent toolkit/components/windowcreator/test/test_private_window_from_content.html | any mochitest-32 | application ran for longer than allowed maximum time

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=82321584&repo=autoland

https://queue.taskcluster.net/v1/task/cfOk86KfR8-4Dv1znro3yQ/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Geoff Brown [:gbrown]]

Screenshot as in bug 1336541 - related?

Also notice that the first run dialog for tab queues is displayed...that seems unexpected.

Comment 2

[Geoff Brown [:gbrown]]

The job here is Android Debug mochitest-32. This is perma-failing on autoland starting with https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4dc0edd4681c71fb0e67bc19bbc290616952fdf7&filter-searchStr=mochitest-32...but it was running long (90-100 minutes) in earlier pushes.

There was a significant increase in run time a few days earlier, at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=mochitest-32&tochange=a793136c90bc5d32f2c82aa7dea4bc300c4f1836&fromchange=31c09bb63b697089f9866b7cbfa57b8900df2b1d.

Comment 3

[Geoff Brown [:gbrown]]

Most of the Android 4.3 Debug mochitest jobs run in 40 to 60 minutes, but mochitest-31 and mochitest-32 are closer to 90 minutes.

Comment 4

[Geoff Brown [:gbrown]]

This looks to be a regression from bug 1335751: at least some Android tests started running a little slower with that landing:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=android%20mochitest-32&tochange=c10963d3a6872149f753738c265ef56b4394dda0&fromchange=340e7f8a7229786fc8808d4a78983338997f81f3

There was an earlier landing (later backed out) with the same result:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=3c4a6d36149c38695e0cdccd9e7bbe261d1e17c8&fromchange=8ab76e34613bf4af022e05d14ed5f75529ed951d&filter-searchStr=mochitest-32


I don't think there's anything "wrong" with bug 1335751, but let's just check in there. :jonco - Just wanted to make you aware that some Android mochitests run slower with your patches; if that is surprising or concerning to you, let me know.


Probably the best action here is to add more test chunks so we have the capacity to run longer.

Comment 5

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f45aae3c2ea0b4b073fd4c98f144a075277a2a11

Comment 7

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c4756660961d9548aa8cbbf96784a273d2f70f0c

Comment 8

[Geoff Brown [:gbrown]]

Attachment: increase android debug mochitest chunks to 48

At 40 chunks there are still several chunks taking longer than 70 minutes.

48 chunks seems to bring almost all jobs into the 30 - 60 minute range.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=97798692b4534acc5f68fc10d0e300790edb4bf5

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8844988 [details] [diff] [review]
increase android debug mochitest chunks to 48

Review of attachment 8844988 [details] [diff] [review]:
-----------------------------------------------------------------

insane, but needed.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ec475e2b1915
Increase Android debug mochitest chunks; r=jmaher

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Looks like it worked. No chunks exceeded the max runtime.

Comment 12

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ec475e2b1915

Comment 13

[Steve Fink [:sfink] [:s:]]

(In reply to Geoff Brown [:gbrown] from comment #4)
> I don't think there's anything "wrong" with bug 1335751, but let's just
> check in there. :jonco - Just wanted to make you aware that some Android
> mochitests run slower with your patches; if that is surprising or concerning
> to you, let me know.

(jonco is on PTO) Yes, it added some slower but more complete gray marking checks.

Comment 14

[Intermittent Failures Robot]

27 failures in 790 pushes (0.034 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 20
* mozilla-inbound: 6
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1345350&startday=2017-03-06&endday=2017-03-12&tree=all