1433270 - Intermittent tp6_google | timeout

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: aiakab [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=158498573&repo=mozilla-inbound

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64/mozilla-inbound_ubuntu64_hw_test-tp6-e10s-bm104-tests1-linux-build12.txt.gz

Comment 1

[Intermittent Failures Robot]

20 failures in 119 pushes (0.168 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 8
* mozilla-central: 6
* mozilla-inbound: 4
* try: 2

Platform breakdown:
* linux64: 18
* osx-10-10: 1
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-01-26&endday=2018-01-26&tree=all

Comment 2

[Tiberius Oros[:tiberius_oros]]

There have been 39 failures in the last 7 days.
This was filed on January 24th
This fails on Linux x64 / opt & pgo. There are some exceptions for 	linux64-qr (2) and 	OS X 10.10.

Here is a relevant log file and a snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=158483519&lineNumber=1849

09:51:05     INFO -  PID 16806 | 17255
1848
09:51:05     INFO -  PID 16806 | ExceptionHandler::SendContinueSignalToChild sent continue signal to child
1849
09:51:05     INFO -  TEST-UNEXPECTED-ERROR | tp6_google | timeout
1850
09:51:05    ERROR -  Traceback (most recent call last):
1851
09:51:05     INFO -    File "/builds/slave/test/build/tests/talos/talos/run_tests.py", line 289, in run_tests
1852
09:51:05     INFO -      talos_results.add(mytest.runTest(browser_config, test))
1853
09:51:05     INFO -    File "/builds/slave/test/build/tests/talos/talos/ttest.py", line 62, in runTest
1854
09:51:05     INFO -      return self._runTest(browser_config, test_config, setup)
1855
09:51:05     INFO -    File "/builds/slave/test/build/tests/talos/talos/ttest.py", line 214, in _runTest
1856
09:51:05     INFO -      debugger_args=browser_config['debugger_args']
1857
09:51:05     INFO -    File "/builds/slave/test/build/tests/talos/talos/talos_process.py", line 139, in run_browser
1858
09:51:05     INFO -      raise TalosError("timeout")
1859
09:51:05     INFO -  TalosError: timeout
1860
09:51:05     INFO -  TEST-INFO took 3608353ms
1861
09:51:05     INFO -  SUITE-END | took 3608s

:rwood could you please take a look?

Comment 3

[Intermittent Failures Robot]

54 failures in 701 pushes (0.077 failures/push) were associated with this bug in the last 7 days. 

This is the #16 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 22
* mozilla-inbound: 15
* autoland: 13
* try: 4

Platform breakdown:
* linux64: 49
* linux64-qr: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-01-22&endday=2018-01-28&tree=all

Comment 4

[Intermittent Failures Robot]

22 failures in 111 pushes (0.198 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 14
* mozilla-inbound: 4
* mozilla-central: 4

Platform breakdown:
* linux64: 22

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-01-29&endday=2018-01-29&tree=all

Comment 7

[Intermittent Failures Robot]

19 failures in 148 pushes (0.128 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 15
* mozilla-inbound: 4

Platform breakdown:
* windows10-64: 9
* osx-10-10: 6
* windows7-32: 2
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-01-30&endday=2018-01-30&tree=all

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood unfortunately the new hardware didn't reduce this failure rate- might be worth investigating in the short term.

Comment 9

[Intermittent Failures Robot]

15 failures in 164 pushes (0.091 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 8
* mozilla-inbound: 6
* mozilla-central: 1

Platform breakdown:
* windows10-64: 8
* windows7-32: 5
* osx-10-10: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-01-31&endday=2018-01-31&tree=all

Comment 10

[Intermittent Failures Robot]

54 failures in 129 pushes (0.419 failures/push) were associated with this bug yesterday.    

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 41
* mozilla-central: 10
* autoland: 3

Platform breakdown:
* windows10-64: 42
* windows7-32: 7
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-01&endday=2018-02-01&tree=all

Comment 11

[Robert Wood [:rwood]]

I really have *no idea* what is causing this. It looks like tp6_google gets through several tp page cycles successfully but then out of the blue (there's nothing obvious in the logs). I'll see if I can reproduce this locally, and if not I'll get a loaner and try it there.

Comment 12

[Intermittent Failures Robot]

144 failures in 735 pushes (0.196 failures/push) were associated with this bug in the last 7 days. 

This is the #2 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 72
* autoland: 52
* mozilla-central: 20

Platform breakdown:
* windows10-64: 71
* linux64: 44
* windows7-32: 20
* osx-10-10: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-01-29&endday=2018-02-04&tree=all

Comment 13

[Intermittent Failures Robot]

21 failures in 118 pushes (0.178 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 10
* autoland: 6
* mozilla-central: 5

Platform breakdown:
* windows7-32: 7
* windows10-64: 7
* osx-10-10: 6
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-06&endday=2018-02-06&tree=all

Comment 14

[Intermittent Failures Robot]

19 failures in 154 pushes (0.123 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 8
* autoland: 8
* mozilla-central: 3

Platform breakdown:
* windows10-64: 15
* windows7-32: 3
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-07&endday=2018-02-07&tree=all

Comment 15

[Eliza Balazs [:ebalazs_]]

In the last 7 days we have 60 failures.
They occur mostly on windows10-64 (opt, pgo), Windows 7 (opt, pgo), OS X 10.10 (opt), 	Linux x64 (opt, pgo).

Failure log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=161508309&lineNumber=2459

04:23:47     INFO -  PROCESS-CRASH | tp6_google | application crashed [unknown top frame]
04:23:47     INFO -  Crash dump filename: c:\users\cltbld\appdata\local\temp\tmpfvxdld\profile\minidumps\bcef21a3-6ebd-4ee0-9885-f6858add5c38.dmp
04:23:47     INFO -  stderr from minidump_stackwalk:
04:23:47     INFO -  2018-02-10 04:23:47: minidump.cc:4359: INFO: Minidump opened minidump c:\users\cltbld\appdata\local\temp\tmpfvxdld\profile\minidumps\bcef21a3-6ebd-4ee0-9885-f6858add5c38.dmp
04:23:47     INFO -  2018-02-10 04:23:47: minidump.cc:4808: ERROR: ReadBytes: read 0/32
04:23:47     INFO -  2018-02-10 04:23:47: minidump.cc:4453: ERROR: Minidump cannot read header
04:23:47     INFO -  2018-02-10 04:23:47: stackwalk.cc:133: ERROR: Minidump c:\users\cltbld\appdata\local\temp\tmpfvxdld\profile\minidumps\bcef21a3-6ebd-4ee0-9885-f6858add5c38.dmp could not be read
04:23:47     INFO -  2018-02-10 04:23:47: minidump.cc:4331: INFO: Minidump closing minidump
04:23:47     INFO -  minidump_stackwalk exited with return code 1
04:23:47     INFO -  TEST-UNEXPECTED-ERROR | tp6_google | Found crashes after test run, terminating test

Comment 16

[Intermittent Failures Robot]

58 failures in 702 pushes (0.083 failures/push) were associated with this bug in the last 7 days. 

This is the #17 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 25
* autoland: 22
* mozilla-central: 11

Platform breakdown:
* windows10-64: 27
* windows7-32: 17
* osx-10-10: 8
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-05&endday=2018-02-11&tree=all

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

most likely a duplicate of 1378002 - hopefully resolved once we upgrade to new machines and fresh OS.

Comment 18

[Intermittent Failures Robot]

36 failures in 685 pushes (0.053 failures/push) were associated with this bug in the last 7 days. 

This is the #40 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 17
* mozilla-inbound: 10
* mozilla-central: 9

Platform breakdown:
* linux64: 14
* windows7-32: 13
* windows10-64: 7
* osx-10-10: 1
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-12&endday=2018-02-18&tree=all

Comment 19

[Stefan Hindli [:stefan_hindli]]

Over the last 7 days this bug has 30 failures. These happen on Linux x64, linux64-nightly, OS X 10.10, Windows 7 and windows10-64.

Here is the most relevant log example: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=163249283&lineNumber=1937

Here is a relevant part from that log: 

10:25:04     INFO -  PROCESS-CRASH | tp6_google | application crashed [unknown top frame]
10:25:04     INFO -  Crash dump filename: c:\users\cltbld\appdata\local\temp\tmp85ghrn\profile\minidumps\cd2a3f5d-b99c-4e74-9b5f-497955b96277.dmp
10:25:04     INFO -  stderr from minidump_stackwalk:
10:25:04     INFO -  2018-02-20 10:25:04: minidump.cc:4359: INFO: Minidump opened minidump c:\users\cltbld\appdata\local\temp\tmp85ghrn\profile\minidumps\cd2a3f5d-b99c-4e74-9b5f-497955b96277.dmp
10:25:04     INFO -  2018-02-20 10:25:04: minidump.cc:4808: ERROR: ReadBytes: read 0/32
10:25:04     INFO -  2018-02-20 10:25:04: minidump.cc:4453: ERROR: Minidump cannot read header
10:25:04     INFO -  2018-02-20 10:25:04: stackwalk.cc:133: ERROR: Minidump c:\users\cltbld\appdata\local\temp\tmp85ghrn\profile\minidumps\cd2a3f5d-b99c-4e74-9b5f-497955b96277.dmp could not be read
10:25:04     INFO -  2018-02-20 10:25:04: minidump.cc:4331: INFO: Minidump closing minidump
10:25:04     INFO -  minidump_stackwalk exited with return code 1
10:25:04     INFO -  TEST-UNEXPECTED-ERROR | tp6_google | Found crashes after test run, terminating test
10:25:04    ERROR -  Traceback (most recent call last):
10:25:04     INFO -    File "C:\slave\test\build\tests\talos\talos\run_tests.py", line 299, in run_tests
10:25:04     INFO -      talos_results.add(mytest.runTest(browser_config, test))
10:25:04     INFO -    File "C:\slave\test\build\tests\talos\talos\ttest.py", line 62, in runTest
10:25:04     INFO -      return self._runTest(browser_config, test_config, setup)
10:25:04     INFO -    File "C:\slave\test\build\tests\talos\talos\ttest.py", line 209, in _runTest
10:25:04     INFO -      test_config['name'])
10:25:04     INFO -    File "C:\slave\test\build\tests\talos\talos\ttest.py", line 46, in check_for_crashes
10:25:04     INFO -      raise TalosCrash('Found crashes after test run, terminating test')
10:25:04     INFO -  TalosCrash: Found crashes after test run, terminating test
10:25:04     INFO -  TEST-INFO took 3633885ms
10:25:04     INFO -  SUITE-END | took 3633s
WARNING | IO Completion Port failed to signal process shutdown
Parent process 5820 exited with children alive:
PIDS: 7040, 7148
Attempting to kill them, but no guarantee of success
10:28:09    ERROR - Return code: 2
10:28:09  WARNING - setting return code to 2
10:28:09    ERROR - # TBPL FAILURE #

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood- as this is not just windows- we should look into this.  Possibly this is google specific and we stop that test, or we determine that it is toolchain related.  Maybe some investigation in the short term.

Comment 21

[Intermittent Failures Robot]

23 failures in 675 pushes (0.034 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 10
* mozilla-inbound: 9
* mozilla-central: 4

Platform breakdown:
* linux64: 10
* windows7-32: 6
* windows10-64: 5
* osx-10-10: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-19&endday=2018-02-25&tree=all

Comment 22

[Intermittent Failures Robot]

16 failures in 831 pushes (0.019 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 11
* mozilla-inbound: 3
* mozilla-central: 2

Platform breakdown:
* linux64: 15
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433270&startday=2018-02-26&endday=2018-03-04&tree=all

Comment 23

[Arthur Iakab [arthur_iakab]]

:rwood do you have any updates on this Bug?

Comment 24

[Robert Wood [:rwood]]

I believe alot of these timeouts are the same tp6 issue as in Bug 1439979.

I have a couple of things running on try now to get more info / try to fix this.

https://bugzilla.mozilla.org/show_bug.cgi?id=1439979#c10

https://bugzilla.mozilla.org/show_bug.cgi?id=1439979#c11

Comment 25

[Intermittent Failures Robot]

1 failures in 822 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1433270&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 26

[Arthur Iakab [arthur_iakab]]

 :rwod do you have any updates on this bug?

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

there are 3 instances in the last 30 days, I think this bug is mostly fixed by the migration to the new hardware and taskcluster worker.

Comment 28

[Intermittent Failures Robot]

4 failures in 792 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 3
* try: 1

Platform breakdown:
* windows7-32: 2
* windows10-64: 1
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1433270&startday=2018-04-09&endday=2018-04-15&tree=trunk

Comment 29

[Robert Wood [:rwood]]

Looks good - there have been zero instances of this since April 10th.

Comment 30

[Intermittent Failures Robot]

2 failures in 782 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2

Platform breakdown:
* osx-10-10: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1433270&startday=2018-04-16&endday=2018-04-22&tree=trunk