1476893 - Intermittent gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output

Status: RESOLVED FIXED

(Core :: Panning and Zooming, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ebalazs [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=188882070&repo=autoland

https://queue.taskcluster.net/v1/task/JCOPe5VIQeG1B8KXw3sYXw/runs/0/artifacts/public/logs/live_backing.log

WARNING -  TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output
[task 2018-07-19T07:51:57.091Z] 07:51:57     INFO -  INFO | automation.py | Application ran for: 0:21:22.108770
[task 2018-07-19T07:51:57.092Z] 07:51:57     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpHehGx0pidlog
[task 2018-07-19T07:51:57.408Z] 07:51:57     INFO -  Contents of /data/anr/traces.txt:
[task 2018-07-19T07:51:57.410Z] 07:51:57     INFO -  ----- pid 2466 at 2018-07-19 00:51:42 -----
[task 2018-07-19T07:51:57.410Z] 07:51:57     INFO -  Cmd line: org.mozilla.fennec_aurora
[task 2018-07-19T07:51:57.410Z] 07:51:57     INFO -  JNI: CheckJNI is on; workarounds are off; pins=0; globals=254
[task 2018-07-19T07:51:57.410Z] 07:51:57     INFO -  DALVIK THREADS:
[task 2018-07-19T07:51:57.410Z] 07:51:57     INFO -  (mutexes: tll=0 tsl=0 tscl=0 ghl=0)
[task 2018-07-19T07:51:57.411Z] 07:51:57     INFO -  "main" prio=5 tid=1 NATIVE
[task 2018-07-19T07:51:57.412Z] 07:51:57     INFO -    | group="main" sCount=1 dsCount=0 obj=0x414c9578 self=0x2a00d090
[task 2018-07-19T07:51:57.412Z] 07:51:57     INFO -    | sysTid=2466 nice=0 sched=0/0 cgrp=apps handle=1073811452

Comment 1

[Intermittent Failures Robot]

9 failures in 653 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 5
* autoland: 3
* mozilla-central: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-07-16&endday=2018-07-22&tree=trunk

Comment 2

[Intermittent Failures Robot]

2 failures in 2561 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 2

Platform breakdown:
* android-em-4-3-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-08-05&endday=2018-08-11&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 2568 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-08-20&endday=2018-08-26&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 2545 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-08-27&endday=2018-09-02&tree=all

Comment 5

[Intermittent Failures Robot]

4 failures in 2667 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 1
* mozilla-inbound: 1
* autoland: 2

Platform breakdown:
* android-em-4-3-armv7-api16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 7

[Intermittent Failures Robot]

4 failures in 2684 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 2
* try: 1
* mozilla-central: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-09-10&endday=2018-09-16&tree=all

Comment 8

[Intermittent Failures Robot]

2 failures in 2902 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 2

Platform breakdown:
* android-em-4-3-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-09-17&endday=2018-09-23&tree=all

Comment 9

[Intermittent Failures Robot]

2 failures in 2773 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-09-24&endday=2018-09-30&tree=all

Comment 10

[Intermittent Failures Robot]

1 failures in 2740 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-10-08&endday=2018-10-14&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 2759 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-10-15&endday=2018-10-21&tree=all

Comment 12

[Intermittent Failures Robot]

4 failures in 2584 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 1
* autoland: 1
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 1
* android-em-4-3-armv7-api16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-10-22&endday=2018-10-28&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 3007 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-11-05&endday=2018-11-11&tree=all

Comment 14

[Intermittent Failures Robot]

3 failures in 2917 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1
* try: 1
* autoland: 1

Platform breakdown:
* osx-10-10: 2
* macosx64-nightly: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-11-12&endday=2018-11-18&tree=all

Comment 15

[Intermittent Failures Robot]

3 failures in 2917 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1
* mozilla-central: 1
* try: 1

Platform breakdown:
* osx-10-10: 2
* macosx64-nightly: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-11-12&endday=2018-11-18&tree=all

Comment 16

[Intermittent Failures Robot]

3 failures in 2387 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 2
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 17

[Intermittent Failures Robot]

3 failures in 2808 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 2
* macosx64-nightly: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-11-26&endday=2018-12-02&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 2485 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 19

[Intermittent Failures Robot]

1 failures in 2648 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-12-17&endday=2018-12-23&tree=all

Comment 20

[Intermittent Failures Robot]

3 failures in 1008 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* try: 1

Platform breakdown:
* osx-10-10: 2
* macosx64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2018-12-24&endday=2018-12-30&tree=all

Comment 21

[Intermittent Failures Robot]

4 failures in 2627 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-inbound: 1
• mozilla-central: 1

Platform breakdown:

• osx-10-10: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 22

[Intermittent Failures Robot]

2 failures in 2378 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-inbound: 1

Platform breakdown:

• osx-10-10: 1
• linux64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-01-14&endday=2019-01-20&tree=all

Comment 23

[Intermittent Failures Robot]

3 failures in 2628 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-inbound: 2

Platform breakdown:

• osx-10-10: 2
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 2658 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 25

[Intermittent Failures Robot]

1 failures in 2832 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 26

[Intermittent Failures Robot]

2 failures in 2981 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• osx-10-10: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 27

[Intermittent Failures Robot]

4 failures in 2818 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 3
• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 3
• android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 28

[Intermittent Failures Robot]

3 failures in 2771 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1
• mozilla-inbound: 1

Platform breakdown:

• android-em-7-0-x86_64: 1
• osx-10-10: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 2561 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 30

[Intermittent Failures Robot]

5 failures in 3910 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1
• mozilla-inbound: 1

Platform breakdown:

• linux64: 1
• android-em-7-0-x86_64: 2
• osx-10-10: 1
• osx-10-10-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 31

[Intermittent Failures Robot]

2 failures in 2843 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-inbound: 1

Platform breakdown:

• osx-10-10: 1
• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 32

[Intermittent Failures Robot]

3 failures in 2673 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-inbound: 1
• mozilla-central: 1

Platform breakdown:

• osx-10-10-shippable: 1
• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 33

[Intermittent Failures Robot]

4 failures in 3031 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 1
• mozilla-central: 1
• mozilla-inbound: 1

Platform breakdown:

• macosx64-devedition: 1
• osx-10-10: 1
• osx-10-10-shippable: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 34

[Intermittent Failures Robot]

5 failures in 4131 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 2
• autoland: 2
• mozilla-central: 1

Platform breakdown:

• osx-10-10: 4
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 35

[Intermittent Failures Robot]

6 failures in 4705 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-beta: 1

Platform breakdown:

• osx-10-10: 4
• android-em-4-3-armv7-api16: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 36

[Intermittent Failures Robot]

3 failures in 4658 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• osx-10-10-shippable: 2
• macosx1010-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 37

[Intermittent Failures Robot]

2 failures in 4730 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-inbound: 1

Platform breakdown:

• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 38

[Intermittent Failures Robot]

3 failures in 4337 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-inbound: 1

Platform breakdown:

• android-em-7-0-x86_64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-05-27&endday=2019-06-02&tree=all

Comment 39

[Intermittent Failures Robot]

18 failures in 5197 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• mozilla-central: 1
• mozilla-inbound: 1

Platform breakdown:

• android-em-7-0-x86_64: 17
• macosx1010-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 40

[Intermittent Failures Robot]

11 failures in 4753 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-beta: 2
• mozilla-central: 1
• try: 1
• mozilla-esr68: 1

Platform breakdown:

• android-em-7-0-x86_64: 5
• macosx1010-64-shippable: 1
• macosx1010-64: 2
• macosx1010-64-devedition: 2
• android-em-7-0-x86_64-beta: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 41

[Intermittent Failures Robot]

8 failures in 2581 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 6
• mozilla-inbound: 1

Platform breakdown:

• macosx1010-64: 3
• android-em-7-0-x86_64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-06-17&endday=2019-06-23&tree=all

Comment 42

[Intermittent Failures Robot]

9 failures in 4409 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-inbound: 1
• mozilla-central: 1
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 8
• macosx1010-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 43

[Intermittent Failures Robot]

32 failures in 3501 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 23
• mozilla-central: 5
• mozilla-inbound: 3
• mozilla-esr68: 1

Platform breakdown:

• android-em-7-0-x86_64: 30
• macosx1010-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 44

[Cosmin Sabou [:CosminS]]

Attachment: Bug 1476893 - Disable test_group_zoom.html on Android for frequent failures. r=jmaher

Comment 45

[Kartikaya Gupta (email:[e-mail])]

Note that the increase in frequency here is a result of the emulator upgrade in bug 1556058.

Comment 46

[Botond Ballo [:botond]]

I tried to identify the subtest that's timing out, but in all the instances I've checked so far, it's the very first one, and often that subtest doesn't even get very far, making me wonder if this is some sort of issue with the environment / infrastructure rather than the test itself.

Comment 47

[Kartikaya Gupta (email:[e-mail])]

The emulator log from around the failure shows EGL errors similar to those in bug 1516298. And the try push with snorp's fix from that bug also shows no failures in test_group_zoom, so likely the two failures have the same root cause, which snorp's fix addresses.

Comment 48

[Kartikaya Gupta (email:[e-mail])]

This seems to still be happening even after snorp's fix.

Comment 49

[Kartikaya Gupta (email:[e-mail])]

The screenshot in these failures shows the test harness page, whereas I would expect it to show the page with the subtest, since that's supposed to open in a new browser window/tab. So maybe it's a GV bug where new tabs don't open properly or aren't put in the foreground.

Comment 50

[Intermittent Failures Robot]

26 failures in 4710 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 21
• mozilla-central: 2
• mozilla-inbound: 2
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 51

[Intermittent Failures Robot]

17 failures in 811 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 4
• mozilla-inbound: 3
• mozilla-beta: 1

Platform breakdown:

• linux64-qr: 12
• android-em-7-0-x86_64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-07-18&endday=2019-07-18&tree=all

Comment 52

[Intermittent Failures Robot]

34 failures in 4298 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 2
• autoland: 22
• mozilla-central: 6
• mozilla-inbound: 3
• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 19
• linux64-qr: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 53

[Andreea Pavel [:apavel]]

Hi Joel, I read the conversation within the patch and wanted to ask what to do about this failure. It hasn't reach the disable-recommended queue and it looks like it won't, however the average in 30 days is ~100 failures. https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-22&endday=2019-07-22&tree=trunk&bug=1476893

Looked in OF in the last 3 months and it seems that there are higher spikes in June towards July.

There are 32 total failures in the last 7 days on:
*debug: android-em-7-0-x86_64, linux64-qr,
*opt: android-em-7-0-x86_64

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257641266&repo=autoland&lineNumber=6013

[task 2019-07-22T06:44:25.509Z] 06:44:25 INFO - 2507 INFO TEST-START | gfx/layers/apz/test/mochitest/test_group_zoom.html
[task 2019-07-22T06:44:35.718Z] 06:44:35 INFO - GECKO | "subtest":"helper_bug1280013.html | Main-thread scroll position is still at 0","status":"PASS","js_source":"TestRunner.js"}
[task 2019-07-22T06:44:45.935Z] 06:44:45 INFO - GECKO | ⰲ겿
[task 2019-07-22T06:44:56.150Z] 06:44:56 INFO - GECKO | 겿
[task 2019-07-22T06:45:06.368Z] 06:45:06 INFO - GECKO |
[task 2019-07-22T06:51:38.173Z] 06:51:38 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-07-22T06:51:38.276Z] 06:51:38 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-07-22T06:51:38.276Z] 06:51:38 INFO - TEST-INFO | started process screentopng
[task 2019-07-22T06:51:38.597Z] 06:51:38 INFO - TEST-INFO | screentopng: exit 0
[task 2019-07-22T06:51:52.950Z] 06:51:52 WARNING - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output
[task 2019-07-22T06:51:52.950Z] 06:51:52 INFO - remoteautomation.py | Application ran for: 0:08:00.338450
[task 2019-07-22T06:51:53.615Z] 06:51:53 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmp4QHyWT/481f262f-9964-eef5-6bef-a45406499930.dmp /builds/worker/workspace/build/symbols
[task 2019-07-22T06:52:10.681Z] 06:52:10 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/481f262f-9964-eef5-6bef-a45406499930.dmp
[task 2019-07-22T06:52:10.681Z] 06:52:10 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/481f262f-9964-eef5-6bef-a45406499930.extra
[task 2019-07-22T06:52:10.686Z] 06:52:10 WARNING - PROCESS-CRASH | gfx/layers/apz/test/mochitest/test_group_zoom.html | application crashed [@ libc.so + 0x8c66a]
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Crash dump filename: /tmp/tmp4QHyWT/481f262f-9964-eef5-6bef-a45406499930.dmp
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Operating system: Android
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - 0.0.0 Linux 3.10.0+ #260 SMP PREEMPT Fri May 19 12:48:14 PDT 2017 x86_64
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - CPU: amd64
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - family 6 model 6 stepping 3
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - 4 CPUs
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - GPU: UNKNOWN
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Crash reason: SIGABRT
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Crash address: 0x2c4e
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Process uptime: not available
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - Thread 0 (crashed)
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - 0 libc.so + 0x8c66a
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000010
[task 2019-07-22T06:52:10.696Z] 06:52:10 INFO - rcx = 0xffffffffffffffff rbx = 0x00000000051f6c51
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsi = 0x00007fff4e947f20 rdi = 0x000000000000001f
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rbp = 0x00000000051f6c51 rsp = 0x00007fff4e947ea8
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000008
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r10 = 0x00000000051f6c51 r11 = 0x0000000000000246
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r12 = 0x00000000051f6c51 r13 = 0x0000772e13329e60
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rip = 0x0000772e16fcd66a
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: given as instruction pointer in context
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 1 libc.so + 0x2a0e6
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947eb0 rip = 0x0000772e16f6b0e6
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 2 libutils.so + 0x184e7
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947ed0 rip = 0x0000772e13c2b4e7
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 3 system@[e-mail] + 0x50f2c8
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947ed8 rip = 0x0000000070fbe2c8
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - 4 libart.so + 0x14cd91
[task 2019-07-22T06:52:10.697Z] 06:52:10 INFO - rsp = 0x00007fff4e947ee0 rip = 0x0000772e12c58d91
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 5 boot-framework.oat + 0x1000
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f08 rip = 0x0000000072f54000
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 6 dalvik-main space (deleted) + 0x62b60
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f20 rip = 0x0000000012c62b60
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 7 boot-framework.oat + 0x1e48820
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f78 rip = 0x0000000074d9b820
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 8 libart.so + 0x14cd91
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947f80 rip = 0x0000772e12c58d91
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 9 libart.so + 0x39c37f
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947fc0 rip = 0x0000772e12ea837f
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - 10 libart.so + 0x6e3c40
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - rsp = 0x00007fff4e947fc8 rip = 0x0000772e131efc40
[task 2019-07-22T06:52:10.698Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 11 dalvik-jit-code-cache (deleted) + 0x18c7000
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e947fd8 rip = 0x0000772e00000000
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 12 libutils.so + 0x1837b
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e948020 rip = 0x0000772e13c2b37b
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 13 system@[e-mail] + 0x3f5b10
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e948048 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - 14 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.699Z] 06:52:10 INFO - rsp = 0x00007fff4e948050 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 15 libandroid_runtime.so + 0xff856
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948060 rip = 0x0000772e13db9856
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 16 system@[e-mail] + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948068 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 17 boot-framework.oat + 0x1e17220
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948080 rip = 0x0000000074d6a220
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 18 system@[e-mail] + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948098 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 19 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480a0 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 20 system@[e-mail] + 0x3f5790
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480a8 rip = 0x0000000071695790
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 21 dalvik-jit-code-cache (deleted) + 0x18c7007
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480c0 rip = 0x0000772e00000007
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 22 system@[e-mail] + 0x50f2c8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9480c8 rip = 0x0000000070fbe2c8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 23 system@[e-mail] + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948110 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 24 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948118 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 25 boot-framework.oat + 0x1e19034
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948120 rip = 0x0000000074d6c034
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 26 system@[e-mail] + 0x3f5b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948128 rip = 0x0000000071695b10
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 27 libart.so + 0x1611c0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948160 rip = 0x0000772e12c6d1c0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 28 system@[e-mail] + 0x1fd518
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948170 rip = 0x0000000070cac518
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 29 dalvik-main space (deleted) + 0x16c7f0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948190 rip = 0x0000000012d6c7f0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 30 dalvik-main space (deleted) + 0x671d0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948198 rip = 0x0000000012c671d0
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 31 dalvik-main space (deleted) + 0x204a00
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481a0 rip = 0x0000000012e04a00
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 32 system@[e-mail] + 0x3ad778
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481a8 rip = 0x000000007164d778
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 33 boot-framework.oat + 0x1e160c4
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481e0 rip = 0x0000000074d690c4
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 34 system@[e-mail] + 0x39bd40
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481e8 rip = 0x000000007163bd40
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 35 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e9481f8 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 36 dalvik-main space (deleted) + 0x62b60
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - rsp = 0x00007fff4e948200 rip = 0x0000000012c62b60
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.700Z] 06:52:10 INFO - 37 system@[e-mail] + 0x3412c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948208 rip = 0x0000000070df02c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 38 system@[e-mail] + 0x1fd518
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948210 rip = 0x0000000070cac518
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 39 system@[e-mail] + 0x206668
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948218 rip = 0x0000000070cb5668
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 40 boot-framework.oat + 0x1e12ab0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948220 rip = 0x0000000074d65ab0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 41 system@[e-mail] + 0x32c2f0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948228 rip = 0x00000000715cc2f0
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 42 dalvik-main space (deleted) + 0x95d8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948230 rip = 0x0000000012c095d8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 43 system@[e-mail] + 0x3405c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948258 rip = 0x0000000070def5c8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 44 dalvik-main space (deleted) + 0xa1840
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948270 rip = 0x0000000012ca1840
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 45 dalvik-main space (deleted) + 0x204a00
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e948280 rip = 0x0000000012e04a00
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 46 system@[e-mail] + 0x419fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482a0 rip = 0x00000000716b9fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 47 boot-framework.oat + 0x165ee53
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482b0 rip = 0x00000000745b1e53
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 48 system@[e-mail] + 0x419fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482b8 rip = 0x00000000716b9fd8
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - 49 dalvik-main space (deleted) + 0x5fe90
[task 2019-07-22T06:52:10.701Z] 06:52:10 INFO - rsp = 0x00007fff4e9482c0 rip = 0x0000000012c5fe90
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - Found by: stack scanning
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - 50 dalvik-main space (deleted) + 0x77090
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - rsp = 0x00007fff4e9482c8 rip = 0x0000000012c77090
[task 2019-07-22T06:52:10.702Z] 06:52:10 INFO - Found by: stack scanning

Comment 54

[Joel Maher ( :jmaher ) (UTC -8)]

:apavel, without hitting the limit, we need to be more conservative on disabling, this can stick around for a while.

it looks like the linux64-qr failures are different (or at least more informative):
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257120829&repo=mozilla-inbound&lineNumber=25781

:jgilbert- not sure if you are the backup for :kats; there is an assertion in the failures on linux64-qr ^, and that looks to be the root cause, any chance you know if that is a quick fix?

Comment 55

[Andreea Pavel [:apavel]]

Thank you.

Comment 56

[Kelsey Gilbert [:jgilbert]]

I'm not seeing anything obvious.
:jbonisteel, can you route this?

Comment 57

[Botond Ballo [:botond]]

It looks like the failures are all on one of two platforms: linux64-qr, and android-em-7-0-x86_64.

They seem to be two different issues, because they're failing in different parts of the test: on android, the failure happens very soon after the test starts, while on linux64-qr, it's usually a subtest somewhere in the middle that fails.

The linux64-qr failure spiked on the 18th, but seems to be back to a low frequency now; probably not worth investigating at this frequency.

That leaves the android failure; I confirmed that bug 1564846 does not help with that (I got 2/60 failures on Try both with [1] and without [2] that patch).

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1b4fbea2922a19f42a6e4ce993124b5fff4ab69
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a51059db6a7530e69801ed51e514d42e588c3d99

It looks like Kats' most recent suggestion was:

(In reply to Kartikaya Gupta (email:[e-mail]) (away until Feb-2020) from comment #49)

The screenshot in these failures shows the test harness page, whereas I would expect it to show the page with the subtest, since that's supposed to open in a new browser window/tab. So maybe it's a GV bug where new tabs don't open properly or aren't put in the foreground.

Snorp, do you have any ideas for how an issue like this can be investigated?

It would be really good not to disable this test, as it provides important test coverage for APZ.

Comment 58

[Intermittent Failures Robot]

13 failures in 4608 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 1
• mozilla-beta: 2
• mozilla-inbound: 2

Platform breakdown:

• android-em-7-0-x86_64: 12
• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 59

[Intermittent Failures Robot]

11 failures in 4439 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-beta: 3
• try: 1
• mozilla-central: 2

Platform breakdown:

• android-em-7-0-x86_64: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 60

[Intermittent Failures Robot]

8 failures in 4666 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform breakdown:

• macosx1014-64: 1
• android-em-7-0-x86_64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 61

[Intermittent Failures Robot]

21 failures in 4839 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 14
• try: 1
• mozilla-beta: 4
• mozilla-central: 2

Platform breakdown:

• android-em-7-0-x86_64: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 62

[James Willcox (:snorp) ([e-mail]) (he/him)]

Not aware of anything right now. The patch on 1564846 bounced, but like you said, that didn't seem to help anyway.

Comment 63

[Intermittent Failures Robot]

13 failures in 5073 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13

Platform breakdown:

• android-em-7-0-x86_64: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 64

[Intermittent Failures Robot]

18 failures in 5047 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-release: 1
• autoland: 14
• try: 1
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 65

[Intermittent Failures Robot]

13 failures in 4408 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-09-02&endday=2019-09-08&tree=all

Comment 66

[Intermittent Failures Robot]

10 failures in 4637 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-central: 1
• autoland: 6
• mozilla-release: 1

Platform breakdown:

• android-em-7-0-x86_64: 9
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-09-09&endday=2019-09-15&tree=all

Comment 67

[Intermittent Failures Robot]

14 failures in 4350 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 2
• mozilla-esr68: 1

Platform breakdown:

• android-em-7-0-x86_64: 12
• android-em-7-0-x86_64-beta: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 68

[Intermittent Failures Robot]

13 failures in 4530 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 12
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 69

[Intermittent Failures Robot]

17 failures in 4292 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• mozilla-esr68: 1
• mozilla-release: 1

Platform breakdown:

• android-em-7-0-x86_64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-09-30&endday=2019-10-06&tree=all

Comment 70

[Intermittent Failures Robot]

14 failures in 3801 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-beta: 2
• try: 2

Platform breakdown:

• android-em-7-0-x86_64: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 71

[Intermittent Failures Robot]

9 failures in 3964 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 2

Platform breakdown:

• android-em-7-0-x86_64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 72

[Intermittent Failures Robot]

13 failures in 4182 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-esr68: 1
• mozilla-central: 2
• mozilla-beta: 1
• autoland: 8

Platform breakdown:

• android-em-7-0-x86_64: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 73

[Andreea Pavel [:apavel]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) (on leave, back November 5th) from comment #54)

:apavel, without hitting the limit, we need to be more conservative on disabling, this can stick around for a while.

it looks like the linux64-qr failures are different (or at least more informative):
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257120829&repo=mozilla-inbound&lineNumber=25781

:jgilbert- not sure if you are the backup for :kats; there is an assertion in the failures on linux64-qr ^, and that looks to be the root cause, any chance you know if that is a quick fix?

This does not seem to go away and there is no fix since then. History looks like this: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-07-06&endday=2019-11-03&tree=trunk&bug=1476893

Recent failures are on android (last 30 days) so I made a patch for this to be disabled just on that.

Comment 74

[Andreea Pavel [:apavel]]

Attachment: Bug 1476893 - disable test_group_zoom.html on android r=jmaher

Comment 75

[Intermittent Failures Robot]

23 failures in 4314 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 5
• autoland: 15
• mozilla-central: 3

Platform breakdown:

• android-em-7-0-x86_64: 22
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-10-28&endday=2019-11-03&tree=all

Comment 76

[Intermittent Failures Robot]

20 failures in 4850 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 13
• try: 1
• mozilla-inbound: 1

Platform breakdown:

• android-em-7-0-x86_64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 77

[Intermittent Failures Robot]

18 failures in 4207 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 3
• mozilla-inbound: 1
• mozilla-beta: 1
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-11-11&endday=2019-11-17&tree=all

Comment 78

[Intermittent Failures Robot]

16 failures in 921 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 15
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-11-19&endday=2019-11-19&tree=all

Comment 79

[Dorel Luca [:dluca]]

This bug failed 44 times in the last 7 days. Occurs on android-em-7-0-x86_64 on debug and opt build types.

Log:
https://hg.mozilla.org/integration/autoland/rev/ee2997888f54ee12b6505d0af8a216076a351643

Comment 80

[Botond Ballo [:botond]]

The failure rate seems to have spiked recently -- it went from 18 failures in 7 days (comment 77) to 16 failures in a single day (comment 78).

Is it possible to get a regression window for this spike?

Comment 81

[Andreea Pavel [:apavel]]

Retrigger range: https://tinyurl.com/u99ufm3

Next shift will continue.

Comment 82

[Bogdan Tara[:bogdan_tara | bogdant]]

Continued retriggers:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitests%2Ctest-android-em-7.0-x86_64%2Fopt-geckoview-mochitest-e10s-4%2Cm%284%29&tochange=5a0c33a890beb2edfde816a966cba0605858c5d9&fromchange=f6bcd354c056b2a8cabae55a656fb0f38eb01664

Comment 83

[Andreea Pavel [:apavel]]

Retrigger range: https://tinyurl.com/u24nmcp

Hiro, hi, I saw in bug 1562757 you mentioned that this test is flaky. Is this something you can look at ?

Comment 84

[Intermittent Failures Robot]

48 failures in 5444 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 43
• mozilla-central: 3
• mozilla-release: 1
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 47
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-11-18&endday=2019-11-24&tree=all

Comment 85

[Hiroyuki Ikezoe (:hiro)]

I honestly don't think I can do better work than Botond can do in this areas, that's said, I know he's been super busy so I am happy to help him and snorp. Hope I can reproduce the failure locally!

Comment 86

[Botond Ballo [:botond]]

Thanks, Hiro, appreciated :)

Meanwhile I will work on getting test_group_touchevents.html re-enabled in bug 1591519. I think both of these tests are important for APZ test coverage on Android.

Comment 87

[Hiroyuki Ikezoe (:hiro)]

Just an update. I could reproduce the failure locally and saw a suspicious message in adb log

Choreographer: Skipped 84 frames! The application may be doing too much work on its main thread

Just before this message there are a bunch of errors in JS;

E Web Content: [JavaScript Error: "TypeError: can't access dead object"]

Comment 88

[Hiroyuki Ikezoe (:hiro)]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #87)

Just an update. I could reproduce the failure locally and saw a suspicious message in adb log

Choreographer: Skipped 84 frames! The application may be doing too much work on its main thread

Just before this message there are a bunch of errors in JS;

E Web Content: [JavaScript Error: "TypeError: can't access dead object"]

These two may or may not be related to this issue. It pretty often happens even in the case where the test passed.

As far as I can tell, there are at least two stuck points;

1. at await promiseAllPaintsDone in waitUntilApzStable
2. at yield synthesizeNativeTouchDrag(subframe, 10, 200, 0, -300) in helper_bug1280013.html

Though I have no clue for 2), 1) is fixed by bug 1564846 at least on my local Android emulator and a try in bug 1564846 comment 10.
I've already asked snorp to land the fix in bug 1564846.

Comment 89

[Intermittent Failures Robot]

42 failures in 4331 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #17 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 34
• mozilla-central: 3
• try: 5

Platform breakdown:

• android-em-7-0-x86_64: 41
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 90

[Andreea Pavel [:apavel]]

Hiroyuki I saw that James did not reply on bug 1564846, could you land the fix?

There are 39 total failures in the last 7 days on android-em-7-0-x86_64 debug and opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277968433&repo=autoland&lineNumber=3944

[task 2019-11-25T09:44:00.641Z] 09:44:00 INFO - 945 INFO TEST-START | gfx/layers/apz/test/mochitest/test_group_zoom.html
[task 2019-11-25T09:49:28.956Z] 09:49:28 INFO - Failed to get top activity, retrying, once...
[task 2019-11-25T09:50:43.825Z] 09:50:43 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-11-25T09:50:43.928Z] 09:50:43 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-11-25T09:50:43.929Z] 09:50:43 INFO - TEST-INFO | started process screentopng
[task 2019-11-25T09:50:44.222Z] 09:50:44 INFO - TEST-INFO | screentopng: exit 0
[task 2019-11-25T09:50:58.838Z] 09:50:58 WARNING - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_zoom.html | application timed out after 370 seconds with no output
[task 2019-11-25T09:50:58.838Z] 09:50:58 INFO - remoteautomation.py | Application ran for: 0:07:21.595455
[task 2019-11-25T09:50:59.502Z] 09:50:59 INFO - mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aSzGEdqwT7uVqIXqJXOCCA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-11-25T09:51:01.685Z] 09:51:01 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpoIZ9hp/5f74dab3-85aa-a649-c8af-2b28478a4d00.dmp /tmp/tmpRKSUAN
[task 2019-11-25T09:51:04.148Z] 09:51:04 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/5f74dab3-85aa-a649-c8af-2b28478a4d00.dmp
[task 2019-11-25T09:51:04.148Z] 09:51:04 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/5f74dab3-85aa-a649-c8af-2b28478a4d00.extra
[task 2019-11-25T09:51:04.151Z] 09:51:04 WARNING - PROCESS-CRASH | gfx/layers/apz/test/mochitest/test_group_zoom.html | application crashed [@ libc.so + 0x8c66a]
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - Crash dump filename: /tmp/tmpoIZ9hp/5f74dab3-85aa-a649-c8af-2b28478a4d00.dmp
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - Operating system: Android
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - 0.0.0 Linux 3.10.0+ #260 SMP PREEMPT Fri May 19 12:48:14 PDT 2017 x86_64
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - CPU: amd64
[task 2019-11-25T09:51:04.151Z] 09:51:04 INFO - family 6 model 6 stepping 3
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - 4 CPUs
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - GPU: UNKNOWN
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Crash reason: SIGABRT
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Crash address: 0x23c8
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Process uptime: not available
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - Thread 0 (crashed)
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - 0 libc.so + 0x8c66a
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000010
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rcx = 0xffffffffffffffff rbx = 0x00000000051fe175
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rsi = 0x00007fff2e2f9af0 rdi = 0x000000000000001f
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - rbp = 0x00000000051fe175 rsp = 0x00007fff2e2f9a78
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000008
[task 2019-11-25T09:51:04.152Z] 09:51:04 INFO - r10 = 0x00000000051fe175 r11 = 0x0000000000000246
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - r12 = 0x00000000051fe175 r13 = 0x00007a30c6129f40
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rip = 0x00007a30c9b5b66a
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: given as instruction pointer in context
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - 1 libc.so + 0x2a0e6
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9a80 rip = 0x00007a30c9af90e6
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - 2 libutils.so + 0x184e7
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9aa0 rip = 0x00007a30c94eb4e7
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - 3 system@[e-mail] + 0x3d5350
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ac8 rip = 0x0000000071675350
[task 2019-11-25T09:51:04.153Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 4 boot-framework.oat + 0x1000
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ad8 rip = 0x0000000072f54000
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 5 dalvik-main space (deleted) + 0x5d700
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9af0 rip = 0x0000000012c5d700
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 6 libart.so + 0x2d39f9
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b10 rip = 0x00007a30c5bdf9f9
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 7 boot-framework.oat + 0x15bc3f0
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b18 rip = 0x000000007450f3f0
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - 8 libart.so + 0x4f1be9
[task 2019-11-25T09:51:04.154Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b20 rip = 0x00007a30c5dfdbe9
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 9 system@[e-mail] + 0x2a9f48
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b38 rip = 0x0000000071549f48
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 10 system@[e-mail] + 0x3d5350
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b50 rip = 0x0000000071675350
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 11 libart.so + 0x39c37f
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b90 rip = 0x00007a30c5ca837f
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - 12 libart.so + 0x6e3c40
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9b98 rip = 0x00007a30c5fefc40
[task 2019-11-25T09:51:04.155Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 13 libutils.so + 0x1837b
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9bf0 rip = 0x00007a30c94eb37b
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 14 system@[e-mail] + 0x3f5b10
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c18 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 15 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c20 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 16 libandroid_runtime.so + 0xff856
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c30 rip = 0x00007a30c9684856
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.156Z] 09:51:04 INFO - 17 system@[e-mail] + 0x3f5b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c38 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 18 boot-framework.oat + 0x1e17220
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c50 rip = 0x0000000074d6a220
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 19 system@[e-mail] + 0x3f5b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c68 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 20 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c70 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - 21 system@[e-mail] + 0x3f5790
[task 2019-11-25T09:51:04.157Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c78 rip = 0x0000000071695790
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 22 system@[e-mail] + 0x50f2c8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9c98 rip = 0x0000000070fbe2c8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 23 system@[e-mail] + 0x3f5b10
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ce0 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 24 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9ce8 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 25 boot-framework.oat + 0x1e19034
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9cf0 rip = 0x0000000074d6c034
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.158Z] 09:51:04 INFO - 26 system@[e-mail] + 0x3f5b10
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9cf8 rip = 0x0000000071695b10
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 27 system@[e-mail] + 0x1fd518
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d40 rip = 0x0000000070cac518
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 28 dalvik-main space (deleted) + 0x181700
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d60 rip = 0x0000000012d81700
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 29 dalvik-main space (deleted) + 0x25cb00
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d68 rip = 0x0000000012e5cb00
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.159Z] 09:51:04 INFO - 30 dalvik-main space (deleted) + 0x25cb00
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d70 rip = 0x0000000012e5cb00
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 31 system@[e-mail] + 0x3ad778
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9d78 rip = 0x000000007164d778
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 32 boot-framework.oat + 0x1e160c4
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9db0 rip = 0x0000000074d690c4
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 33 system@[e-mail] + 0x39bd40
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9db8 rip = 0x000000007163bd40
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - 34 dalvik-main space (deleted) + 0x95d8
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9dc8 rip = 0x0000000012c095d8
[task 2019-11-25T09:51:04.160Z] 09:51:04 INFO - Found by: stack scanning
[task 2019-11-25T09:51:04.161Z] 09:51:04 INFO - 35 dalvik-main space (deleted) + 0x5d700
[task 2019-11-25T09:51:04.161Z] 09:51:04 INFO - rsp = 0x00007fff2e2f9dd0 rip = 0x0000000012c5d700

Comment 91

[Hiroyuki Ikezoe (:hiro)]

Landed now.

Comment 92

[Andreea Pavel [:apavel]]

Thank you!

Comment 93

[Hiroyuki Ikezoe (:hiro)]

For record, after bug 1564846 landed, this test still fails at 2) in comment 86.

From a failure log

942 INFO TEST-PASS | gfx/layers/apz/test/mochitest/test_group_zoom.html | Starting subtest helper_bug1280013.html
943 INFO TEST-PASS | gfx/layers/apz/test/mochitest/test_group_zoom.html | helper_bug1280013.html | Screen height must be at least 500 pixels for this test to work
944 INFO TEST-PASS | gfx/layers/apz/test/mochitest/test_group_zoom.html | helper_bug1280013.html | Main-thread scroll position is still at 0

Comment 94

[Andreea Pavel [:apavel]]

Botond, this should be disabled as there are 133 total failures in the last 30 days and it doesn't seem like the failures will stop.

Can you please take a look?

Comment 95

[Botond Ballo [:botond]]

According to OrangeFactor, the test has failed 8 times since Hiro's fix has landed, and all 8 failures are in the second place identified in comment 88.

Comment 96

[Botond Ballo [:botond]]

The place in question is in the subtest helper_bug1280013.html, which I note is the first subtest. I made this Try push to try to determine if the issue is specific to that subtest, or something more general that just manifests in that subtest because it is the first:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e771cac19a9b6da2d3cda77fd0fcd656b716f75e&selectedJob=279637023

Comment 97

[Andreea Pavel [:apavel]]

Great, thank you for working on this!

Comment 98

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #96)

I made this Try push to try to determine if the issue is specific to that subtest, or something more general that just manifests in that subtest because it is the first:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e771cac19a9b6da2d3cda77fd0fcd656b716f75e&selectedJob=279637023

The Try push shows that out of 120 total test runs (60 opt and 60 debug), all 4 failures (2 opt and 2 debug) occurred in the new test containing helper_bug1280013.html. This suggests that the problem is specific to helper_bug1280013.html.

So, as a starting point, we can move helper_bug1280013.html out into its own test group. This way, if we need to disable it, we can disable just that one test and leave the others running.

Comment 99

[Botond Ballo [:botond]]

Attachment: Bug 1476893 - Split helper_bug1280013.html into its own test group. r=tnikkel

Comment 100

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/15b3e36e186a
Split helper_bug1280013.html into its own test group. r=tnikkel

Comment 101

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/15b3e36e186a

Comment 102

[Botond Ballo [:botond]]

Failures of this test group seem to be down. Will continue investigation of helper_bug1280013.html in bug 1601568.

Comment 103

[Intermittent Failures Robot]

15 failures in 5144 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• try: 2

Platform breakdown:

• android-em-7-0-x86_64: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-12-02&endday=2019-12-08&tree=all

Comment 104

[Intermittent Failures Robot]

2 failures in 4867 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2

Platform breakdown:

• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 105

[Intermittent Failures Robot]

1 failures in 4582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 106

[Andreea Pavel [:apavel]]

This still shows up on beta, should the it be uplifted?

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=282409811&repo=mozilla-beta&lineNumber=3943

Comment 107

[Hiroyuki Ikezoe (:hiro)]

As you know, to fix this intermittent failure we need two changes in two different bugs, bug 1564846 and bug 1601568. Bug 1564846 is just only for test runner so that we can uplift it. Bug 1601568 changed logic a bit, it's hard to tell for me.

Anyway, the change in bug 1564846 should reduce the failure frequent. We should uplift it at least.

Comment 108

[Andreea Pavel [:apavel]]

James what do you think?

Failure rate on Beta is not high, 2 failure in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-16&endday=2019-12-23&tree=all&bug=1476893

Comment 109

[Intermittent Failures Robot]

1 failures in 1365 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 110

[Botond Ballo [:botond]]

Given the low failure rate, and the fact that the fix in bug 1601568 touched some core scrolling code (==> not a super low-risk fix), I would recommend not uplifting that.

(I'll leave it to snorp to comment on uplifting bug 1564846.)

Comment 111

[Andreea Pavel [:apavel]]

Thank you.

Comment 112

[Intermittent Failures Robot]

1 failures in 1862 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2019-12-30&endday=2020-01-05&tree=all

Comment 113

[James Willcox (:snorp) ([e-mail]) (he/him)]

Yeah, no reason to uplift IMHO.

Comment 114

[Intermittent Failures Robot]

1 failures in 3364 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• android-em-7-0-x86_64-beta: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 115

[Intermittent Failures Robot]

1 failures in 5483 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• android-em-7-0-x86_64-beta: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476893&startday=2020-03-02&endday=2020-03-08&tree=all