1107462 - Trees CLOSED - Various test fail with command timed out: 1800 seconds without output, attempting to kill

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

maybe a result of the problems with ftp last night but seems a lot of test fail with 

01:44:29 INFO - retry: Calling <bound method Proxxy._download_file of <mozharness.mozilla.proxxy.Proxxy object at 0x00D4DB70>> with args: ('https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32-pgo/1417674611/firefox-37.0a1.en-US.win32.tests.zip', 'C:\\slave\\test\\build\\firefox-37.0a1.en-US.win32.tests.zip'), kwargs: {}, attempt #1 command timed out: 1800 seconds without output, attempting to kill

Comment 1

[Carsten Book [:Tomcat]]

and this is spreading so closing the integration trees and m-c

Comment 2

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-04T04:25:21
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=try&job_id=3511066
repository: try
who: emanuel[dot]hoogeveen[at]gmail[dot]com
machine: tst-linux32-spot-116
buildname: Ubuntu VM 12.04 try opt test cppunit
revision: 7c726ca5b4a6

Can't download from http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/try-builds/[e-mail]-7c726ca5b4a6/try-linux/firefox-37.0a1.en-US.linux-i686.tests.zip to /builds/slave/test/build/firefox-37.0a1.en-US.linux-i686.tests.zip!
Caught exception: timed out
Caught exception: timed out
Caught exception: timed out
command timed out: 1800 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--cppunittest-suite', 'cppunittest', '--blob-upload-branch', 'try', '--download-symbols', 'ondemand'], attempting to kill

Comment 3

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-04T04:25:35
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=try&job_id=3511076
repository: try
who: emanuel[dot]hoogeveen[at]gmail[dot]com
machine: tst-linux32-spot-1017
buildname: Ubuntu VM 12.04 try opt test mochitest-e10s-browser-chrome-3
revision: 7c726ca5b4a6

Can't download from http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/try-builds/[e-mail]-7c726ca5b4a6/try-linux/firefox-37.0a1.en-US.linux-i686.tests.zip to /builds/slave/test/build/firefox-37.0a1.en-US.linux-i686.tests.zip!
Caught exception: timed out
Caught exception: timed out
Caught exception: timed out
command timed out: 1800 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'browser-chrome-chunked', '--e10s', '--total-chunks', '3', '--this-chunk', '3', '--blob-upload-branch', 'try', '--download-symbols', 'ondemand'], attempting to kill

Comment 4

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-04T04:25:46
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=try&job_id=3511770
repository: try
who: emanuel[dot]hoogeveen[at]gmail[dot]com
machine: t-snow-r4-0154
buildname: Rev4 MacOSX Snow Leopard 10.6 try opt test cppunit
revision: 7c726ca5b4a6

command timed out: 1800 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/mac_unittest.py', '--cppunittest-suite', 'cppunittest', '--blob-upload-branch', 'try', '--download-symbols', 'ondemand'], attempting to kill

Comment 5

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-04T04:25:56
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=try&job_id=3511744
repository: try
who: emanuel[dot]hoogeveen[at]gmail[dot]com
machine: t-snow-r4-0151
buildname: Rev4 MacOSX Snow Leopard 10.6 try opt test mochitest-browser-chrome-1
revision: 7c726ca5b4a6

command timed out: 1800 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/mac_unittest.py', '--mochitest-suite', 'browser-chrome-chunked', '--total-chunks', '3', '--this-chunk', '1', '--blob-upload-branch', 'try', '--download-symbols', 'ondemand'], attempting to kill

Comment 6

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-04T04:26:17
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=try&job_id=3511766
repository: try
who: emanuel[dot]hoogeveen[at]gmail[dot]com
machine: t-snow-r4-0086
buildname: Rev4 MacOSX Snow Leopard 10.6 try opt test mochitest-e10s-browser-chrome-3
revision: 7c726ca5b4a6

command timed out: 1800 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/mac_unittest.py', '--mochitest-suite', 'browser-chrome-chunked', '--e10s', '--total-chunks', '3', '--this-chunk', '3', '--blob-upload-branch', 'try', '--download-symbols', 'ondemand'], attempting to kill

Comment 7

[Carsten Book [:Tomcat]]

Trees reopen at about 7:30am Pacific.

Hal, Jakem, do you want to update this bug before we close this as fixed ?

Comment 8

[Chris Cooper [:coop] (he/him)]

Timeouts look like carry-overs that started during the outage.

Comment 9

[Hal Wine [:hwine] use NI!]

Recapping issue & timeline from irc logs:

- releng & webops paged at 03:39 PT
- initial reports hinted at collateral from bug 1107156
- escalated to 2nd level webops @ 04:30 PT
- confirmed as zlb blocking/rate limiting issue
- 04:40 PT added SCL3 hosts to white list - did not resolve issue
- 04:50 PT backed off "general protection class settings" set yesterday as part of bug 1107156
- 05:15 PT AWS hosts added to white list - seems to resolve issues for AWS
- 05:30 PT continued scl3 issues
- log correlations imply extremely low transfer rate within scl3
- 06:12 PT escalated to catlee
- 06:50 PT confirm extremely slow transfer rate of 2kb/s
- 07:00 PT page netops
- jake discovers zlb rate limiting rules interaction not working as advertised
- 07:18 PT all rate throttling disabled for ftp.m.o VIP
- 07:19 PT netops dismissed with thanks

No more problems in scl3