1311861 - Intermittent Windows build command timed out: 10800 seconds without output running ['c:/mozilla-build/python27/python', '-u', 'scripts/scripts/fx_desktop_build.py', '--config', 'builds/releng_base_windows_64_builds.py', '--custom-build-variant-cfg', 'debu

Status: RESOLVED DUPLICATE of bug 1147271

(Release Engineering :: General, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=5408416&repo=autoland

https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win64-debug/1476984049/autoland-win64-debug-bm71-build1-build246.txt.gz

Comment 1

[Intermittent Failures Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* try: 1
* autoland: 1

Platform breakdown:
* windowsxp: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2016-11-07&endday=2016-11-13&tree=all

Comment 2

[Intermittent Failures Robot]

5 failures in 563 pushes (0.009 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 3
* autoland: 2

Platform breakdown:
* windows8-64: 4
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-01-02&endday=2017-01-08&tree=all

Comment 3

[Intermittent Failures Robot]

9 failures in 867 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 5
* mozilla-inbound: 3
* mozilla-central: 1

Platform breakdown:
* windowsxp: 8
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-04-03&endday=2017-04-09&tree=all

Comment 4

[Intermittent Failures Robot]

13 failures in 894 pushes (0.015 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 4
* mozilla-central: 2

Platform breakdown:
* windowsxp: 11
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-04-10&endday=2017-04-16&tree=all

Comment 5

[Intermittent Failures Robot]

5 failures in 817 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 3
* autoland: 2

Platform breakdown:
* windowsxp: 3
* windows8-64: 1
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-04-17&endday=2017-04-23&tree=all

Comment 6

[Intermittent Failures Robot]

16 failures in 883 pushes (0.018 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 5
* mozilla-central: 2
* try: 1

Platform breakdown:
* windowsxp: 10
* windows8-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-04-24&endday=2017-04-30&tree=all

Comment 7

[Intermittent Failures Robot]

16 failures in 156 pushes (0.103 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 8
* autoland: 7
* mozilla-beta: 1

Platform breakdown:
* windowsxp: 11
* windows8-64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-03&endday=2017-05-03&tree=all

Comment 8

[Geoff Brown [:gbrown]]

There have been infrequent failures reported here since March 24, but frequency increased significantly May 2.

Recent failures seem to consistently timeout following the browser/locales directory:

10:12:55     INFO -    adding: firefox/softokn3.chk (176 bytes security) (stored 0%)
10:12:55     INFO -    adding: firefox/softokn3.dll (176 bytes security) (deflated 46%)
10:12:55     INFO -    adding: firefox/ucrtbase.dll (176 bytes security) (deflated 52%)
10:12:55     INFO -    adding: firefox/uninstall/helper.exe (176 bytes security) (deflated 62%)
10:12:55     INFO -    adding: firefox/update-settings.ini (176 bytes security) (deflated 12%)
10:12:55     INFO -    adding: firefox/updater.exe (176 bytes security) (deflated 52%)
10:12:55     INFO -    adding: firefox/updater.ini (176 bytes security) (deflated 47%)
10:12:55     INFO -    adding: firefox/vcruntime140.dll (176 bytes security) (deflated 45%)
10:12:55     INFO -    adding: firefox/xul.dll (176 bytes security) (deflated 57%)
10:12:55     INFO -  c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/config/nsinstall.py -D ../../dist/
10:12:55     INFO -  mv -f '../../dist/l10n-stage/firefox-55.0a1.x-test.win32.zip' 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/dist/firefox-55.0a1.x-test.win32.zip'
10:12:55     INFO -  if test -f '../../dist/l10n-stage/firefox-55.0a1.x-test.win32.zip.asc'; then mv -f '../../dist/l10n-stage/firefox-55.0a1.x-test.win32.zip.asc' 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/dist/firefox-55.0a1.x-test.win32.zip.asc'; fi
10:12:55     INFO -  mozmake.EXE[7]: Leaving directory 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/browser/locales'
10:12:55     INFO -  mozmake.EXE[6]: Entering directory 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/browser/locales'
10:12:55     INFO -  repackaging done
10:12:55     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/browser/locales'
10:13:25     INFO -  mozmake.EXE[5]: Entering directory 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/browser/locales'
10:13:25     INFO -  c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/toolkit/mozapps/installer/unpack.py ../../dist/l10n-stage/firefox
10:13:25     INFO -  cd ../../dist/l10n-stage && test $(cat firefox/update.locale) = x-test
10:13:25     INFO -  mozmake.EXE[5]: Leaving directory 'c:/builds/moz2_slave/autoland-w32-00000000000000000/build/src/obj-firefox/browser/locales'
11:33:25     INFO - Automation Error: mozprocess timed out after 4800 seconds running ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v']

Comment 9

[Ryan VanderMeulen [:RyanVM]]

This is officially holding up the go to build for dot releases off multiple branches now.

Comment 10

[Phil Ringnalda (:philor)]

Coincidences happen, but this smelled like the frequency exploded right at the time of the as-yet-unexplained bug 1361603.

Comment 11

[Intermittent Failures Robot]

26 failures in 156 pushes (0.167 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 14
* mozilla-inbound: 7
* mozilla-esr52: 3
* mozilla-release: 1
* mozilla-beta: 1

Platform breakdown:
* windows8-64: 11
* windowsxp: 10
* linux64-stylo: 2
* windows8-64-devedition: 1
* osx-10-7: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-04&endday=2017-05-04&tree=all

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

well this is still failing at a very high rate, 26 failures in a single day is pretty high.  This seems to be on all platforms.

Ted, can you look into this and figure out why builds are failing so frequently>

Comment 13

[Intermittent Failures Robot]

26 failures in 150 pushes (0.173 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 16
* mozilla-esr52: 4
* mozilla-beta: 3
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows8-64: 13
* windowsxp: 12
* windowsxp-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-05&endday=2017-05-05&tree=all

Comment 14

[Intermittent Failures Robot]

113 failures in 770 pushes (0.147 failures/push) were associated with this bug in the last 7 days. 

This is the #9 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 65
* mozilla-inbound: 27
* mozilla-beta: 8
* mozilla-esr52: 7
* mozilla-central: 3
* try: 2
* mozilla-release: 1

Platform breakdown:
* windows8-64: 54
* windowsxp: 53
* linux64-stylo: 2
* windowsxp-devedition: 1
* windows8-64-devedition: 1
* osx-10-7: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-01&endday=2017-05-07&tree=all

Comment 15

[Intermittent Failures Robot]

19 failures in 129 pushes (0.147 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 15
* try: 3
* autoland: 1

Platform breakdown:
* windowsxp: 10
* windows8-64: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-08&endday=2017-05-08&tree=all

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

:ted, can you look into this?

Comment 17

[(not currently active) Ted Mielczarek]

(In reply to Joel Maher ( :jmaher) from comment #16)
> :ted, can you look into this?

I don't have cycles for this right now, sorry. :-(

Comment 18

[Intermittent Failures Robot]

34 failures in 171 pushes (0.199 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 19
* mozilla-beta: 8
* mozilla-inbound: 7

Platform breakdown:
* windowsxp: 15
* windows8-64: 15
* windowsxp-devedition: 2
* windows8-64-devedition: 1
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-09&endday=2017-05-09&tree=all

Comment 19

[Intermittent Failures Robot]

37 failures in 167 pushes (0.222 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 15
* autoland: 13
* mozilla-central: 8
* mozilla-beta: 1

Platform breakdown:
* windows8-64: 20
* windowsxp: 15
* windows8-64-devedition: 1
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-10&endday=2017-05-10&tree=all

Comment 20

[Intermittent Failures Robot]

29 failures in 186 pushes (0.156 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 12
* mozilla-inbound: 7
* mozilla-central: 7
* mozilla-esr52: 3

Platform breakdown:
* windowsxp: 18
* windows8-64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-11&endday=2017-05-11&tree=all

Comment 22

[Intermittent Failures Robot]

201 failures in 879 pushes (0.229 failures/push) were associated with this bug in the last 7 days. 

This is the #2 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 84
* autoland: 71
* mozilla-central: 20
* mozilla-beta: 12
* mozilla-esr52: 8
* try: 3
* mozilla-release: 1
* mozilla-esr45: 1
* jamun: 1

Platform breakdown:
* windows8-64: 101
* windowsxp: 93
* windows8-64-devedition: 3
* windowsxp-devedition: 2
* windows2012-32: 1
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-05-08&endday=2017-05-14&tree=all

Comment 23

[Intermittent Failures Robot]

2 failures in 892 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windowsxp: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1311861&startday=2017-06-26&endday=2017-07-02&tree=all