Frequent Windows build timeouts during make_pkg or make package pretty buildstep, often with a "--- Press `Q' to quit, or any other key to continue ---" prompt

RESOLVED FIXED in Firefox 13

Status

()

Core
General
--
critical
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: philor, Assigned: froydnj)

Tracking

({intermittent-failure})

Trunk
mozilla14
x86
Windows Server 2003
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox13 fixed)

Details

(Whiteboard: [qa-])

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
Six of one whether this is releng, or packaging. Personally, I'm betting on some Windows zip limitation.

https://tbpl.mozilla.org/php/getParsedLog.php?id=9584873&tree=Firefox
WINNT 5.2 mozilla-central pgo-build on 2012-02-23 18:00:09 PST for push 6fe899b9b372

Compressing...
cd ../../dist && (cd firefox && rm -f omni.ja components/binary.manifest && grep -h '^binary-component' components/*.manifest > binary.manifest ; for m in components/*.manifest; do sed -e 's/^binary-component/#binary-component/' $m > tmp.manifest && mv tmp.manifest $m; done; zip -r9m omni.ja chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader hyphenation update.locale  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  &&  e:/builds/moz2_slave/m-cen-w32-pgo/build/obj-firefox/dist/bin/xpcshell.exe -g "$PWD" -a "$PWD" -f /e/builds/moz2_slave/m-cen-w32-pgo/build/toolkit/mozapps/installer/precompile_cache.js -e "populate_startupcache('GreD', 'omni.ja', 'startupCache.zip');" && rm -rf jsloader && unzip startupCache.zip && rm startupCache.zip && zip -r9m omni.ja jsloader/resource/gre && d:/mozilla-build/python25/python2.5.exe /e/builds/moz2_slave/m-cen-w32-pgo/build/config/optimizejars.py --optimize /e/builds/moz2_slave/m-cen-w32-pgo/build/obj-firefox/browser/installer/../../jarlog//en-US ./ ./ && mv binary.manifest components && printf "manifest components/binary.manifest\n" > chrome.manifest) && (cd firefox && d:/mozilla-build/python25/python2.5.exe /e/builds/moz2_slave/m-cen-w32-pgo/build/config/createprecomplete.py) && zip -r9D win32/en-US/nightly-13.0a1.zip firefox
...
  adding: update.locale (188 bytes security) (stored 0%) <-- where most of them die
...
Archive:  startupCache.zip
�\Q
--- Press `Q' to quit, or any other key to continue ---
command timed out: 1200 seconds without output, attempting to kill

We had six or eight of them on inbound yesterday, so I hounded a probably-guiltless patch out of the tree, then we had two more overnight, and now this one (first PGO I remember, the previous ones were split among Win32 opt and debug and Win64 opt).

Unfortunately, we're going to have a Mexican standoff between releng and build over who should figure out what is happening, because nobody's going to want to sit watching a slave make package over and over until it happens.
(Reporter)

Comment 1

6 years ago
That was w32-ix-slave27 (and possibly the only one that was in make package pretty rather than make_pkg, or possibly make package pretty is a PGO thing).

https://tbpl.mozilla.org/php/getParsedLog.php?id=9584748&tree=Mozilla-Inbound - w32-ix-slave30

https://tbpl.mozilla.org/php/getParsedLog.php?id=9560154&tree=Mozilla-Inbound - w32-ix-slave26

https://tbpl.mozilla.org/php/getParsedLog.php?id=9549136&tree=Mozilla-Inbound - w64-ix-slave15

https://tbpl.mozilla.org/php/getParsedLog.php?id=9542348&tree=Mozilla-Inbound - w32-ix-slave40
https://tbpl.mozilla.org/php/getParsedLog.php?id=9541533&tree=Mozilla-Inbound - w32-ix-slave25

https://tbpl.mozilla.org/php/getParsedLog.php?id=9538661&tree=Mozilla-Inbound - w32-ix-slave44

so it doesn't look at all good for blame-the-slave.
Summary: Frequent Windows build timeouts during make package pretty buildstep, often with a "--- Press `Q' to quit, or any other key to continue ---" prompt → Frequent Windows build timeouts during make_pkg or make package pretty buildstep, often with a "--- Press `Q' to quit, or any other key to continue ---" prompt
(Reporter)

Comment 2

6 years ago
At first, even though I wanted to blame optimizejars.py, I believed mak that the timing didn't work. However, bug 726656 landed on m-c on the 17th, so m-i then or maybe the 18th, either way I took last weekend off, and it's purple: we ignore purple. It could easily have happened ten or twenty times between last Friday/Saturday and yesterday's explosion without anyone ever paying it any real mind.

With any "luck" whatever broke during the time it wasn't working is also broken on m-a and m-b, and we'll get confirmation by them starting to fail too, though something only intermittent on our most-coalesced builds may take time to show up on branches like those that don't do many builds.

First shot fired :)

<nthomas>: I feel a slave loan coming on
Blocks: 726656
(Reporter)

Comment 3

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9598848&tree=Mozilla-Inbound
(Reporter)

Comment 4

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9602326&tree=Firefox
(Reporter)

Comment 5

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9604335&tree=Profiling
(Reporter)

Comment 6

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9608396&tree=Fx-Team
(Reporter)

Comment 7

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9614036&tree=Mozilla-Inbound
(Reporter)

Comment 8

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9618613&tree=Mozilla-Inbound went a bit longer - the usual is stopping at the "adding: update.locale (188 bytes security) (stored 0%)" point, at the end of zipping up apparently, since this one continued on to

WARNING: CheckLinkStatus called on main thread! No check performed. Assuming link is up, status is unknown.: file e:/builds/moz2_slave/m-in-w32-dbg/build/netwerk/system/win32/nsNotifyAddrListener.cpp, line 414
resource:///components/txEXSLTRegExFunctions.js
...
resource:///modules/CSPUtils.jsm
WARNING: nsExceptionService ignoring thread destruction after shutdown: file e:/builds/moz2_slave/m-in-w32-dbg/build/xpcom/base/nsExceptionService.cpp, line 199
WARNING: NS_ENSURE_TRUE(mMainThread) failed: file e:/builds/moz2_slave/m-in-w32-dbg/build/xpcom/threads/nsThreadManager.cpp, line 292
WARNING: NS_ENSURE_TRUE(thread) failed: file e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/xpcom/build/nsThreadUtils.cpp, line 181
WARNING: NS_ENSURE_TRUE(thread) failed: file e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/xpcom/build/nsThreadUtils.cpp, line 181
WARNING: NS_ENSURE_TRUE(mMainThread) failed: file e:/builds/moz2_slave/m-in-w32-dbg/build/xpcom/threads/nsThreadManager.cpp, line 283
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0xC1F30001: file e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/xpcom/build/nsThreadUtils.cpp, line 168
nsStringStats
 => mAllocCount:          17261
 => mReallocCount:         1595
 => mFreeCount:           17261
 => mShareCount:          17419
 => mAdoptCount:           2410
 => mAdoptFreeCount:       2410
Archive:  startupCache.zip
Pú
--- Press `Q' to quit, or any other key to continue ---
command timed out: 1200 seconds without output, attempting to kill

so one zip file later.
(Reporter)

Comment 9

6 years ago
And, oddly enough, the build I started locally at about that time, on OS X, stopped at exactly that point too. I couldn't think of anything useful to do to diagnose it (other than to note that the terminal tab said I was running unzip), so I pressed the any key, and it happily continued packaging.
(Reporter)

Comment 10

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9622174&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9622445&tree=Mozilla-Inbound&full=1
(Reporter)

Comment 11

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9612084&tree=Fx-Team
(Reporter)

Comment 12

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9627748&tree=Try
(Reporter)

Comment 13

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9617953&tree=Try
(Reporter)

Comment 14

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9632812&tree=Mozilla-Inbound
(Reporter)

Comment 15

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9632219&tree=Try
(Reporter)

Comment 16

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9634539&tree=Try
(Reporter)

Comment 17

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9637948&tree=Firefox

If you cause the mozilla-central nightly to fail, your severity is something other than "normal."
Severity: normal → blocker
(Reporter)

Comment 18

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9640956&tree=Mozilla-Inbound
(Reporter)

Comment 19

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9648716&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=9655598&tree=Firefox
(Reporter)

Comment 21

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9657812&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9657814&tree=Mozilla-Inbound
(Reporter)

Comment 22

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9652092&tree=Profiling
(Reporter)

Comment 23

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9662432&tree=Mozilla-Inbound
(Reporter)

Comment 24

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9663504&tree=Mozilla-Inbound
(Reporter)

Comment 25

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9664947&tree=Mozilla-Inbound
(Reporter)

Comment 26

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9668281&tree=Mozilla-Inbound
(Reporter)

Comment 27

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9651526&tree=Services-Central
(Reporter)

Comment 28

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9664249&tree=Fx-Team
(Reporter)

Comment 29

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9653671&tree=Fx-Team
(Reporter)

Comment 30

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9681591&tree=Firefox
(Reporter)

Comment 31

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9690065&tree=Profiling
(Reporter)

Comment 32

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9681592&tree=Mozilla-Inbound
(Reporter)

Comment 33

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9698296&tree=Profiling
https://tbpl.mozilla.org/php/getParsedLog.php?id=9703210&tree=Try
(Reporter)

Comment 35

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9710146&tree=Mozilla-Inbound
(Reporter)

Comment 36

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9710151&tree=Services-Central
(Reporter)

Comment 37

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9710180&tree=Services-Central
(Reporter)

Comment 38

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9710138&tree=Services-Central
(Reporter)

Comment 39

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9707695&tree=Services-Central
(Reporter)

Comment 40

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9685426&tree=Fx-Team
(Reporter)

Comment 41

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9710110&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=9710114&tree=Firefox
(Reporter)

Comment 43

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9722110&tree=Mozilla-Inbound
(Reporter)

Comment 44

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9713489&tree=Fx-Team
(Reporter)

Comment 45

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9706790&tree=Maple
(Reporter)

Comment 46

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9729024&tree=Mozilla-Inbound
(Reporter)

Comment 47

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9737789&tree=Mozilla-Inbound
(Reporter)

Comment 48

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9739703&tree=Mozilla-Inbound
(Reporter)

Comment 49

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9745912&tree=Mozilla-Inbound
(Reporter)

Comment 50

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9745999&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9752363&tree=Mozilla-Inbound
(Reporter)

Comment 52

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9755631&tree=Mozilla-Inbound
(Reporter)

Comment 53

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9756157&tree=Mozilla-Inbound
(Reporter)

Comment 54

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9758051&tree=Mozilla-Inbound
Severity: blocker → trivial
https://tbpl.mozilla.org/php/getParsedLog.php?id=9768107&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9768466&tree=Mozilla-Inbound
(Reporter)

Comment 56

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9774549&tree=Mozilla-Inbound
(Reporter)

Comment 57

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9773205&tree=Mozilla-Inbound
(Reporter)

Comment 58

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9791891&tree=Mozilla-Inbound
(Reporter)

Comment 59

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9772185&tree=Fx-Team
Severity: trivial → major
https://tbpl.mozilla.org/php/getParsedLog.php?id=9797883&tree=Mozilla-Inbound
(Reporter)

Comment 61

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9801740&tree=Mozilla-Inbound
https://hg.mozilla.org/integration/mozilla-inbound/rev/8937b91b8435 shows up as suspect after a bunch of bisecting
Severity: major → trivial
(Reporter)

Comment 63

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9813395&tree=Mozilla-Inbound

That's a pretty crazy thing to be sending unzip into spewing bits to the terminal.
Severity: trivial → normal
(Reporter)

Comment 64

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9818421&tree=Mozilla-Inbound
(Reporter)

Comment 65

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9815570&tree=Services-Central
Severity: normal → minor
(Reporter)

Comment 66

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9812594&tree=Services-Central
(Reporter)

Comment 67

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9827758&tree=Mozilla-Inbound
(Reporter)

Comment 68

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9823556&tree=Mozilla-Inbound
(Reporter)

Comment 69

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9831320&tree=Fx-Team
(Reporter)

Comment 70

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9842533&tree=Mozilla-Inbound
(Reporter)

Comment 71

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9837144&tree=Maple
Severity: minor → critical
I can reproduce it here:

https://tbpl.mozilla.org/?tree=Try&rev=80230f9c3fc8 (based on 8937b91b8435)

but not here:

https://tbpl.mozilla.org/?tree=Try&rev=1ba37a8e8b98 (based on 8ca3fd79d50c)
Severity: critical → trivial
stupid bugzilla
Severity: trivial → critical
after more retrying, looks like 8ca3fd79d50c also fails...which shifts the blame onto https://hg.mozilla.org/integration/mozilla-inbound/rev/8ca3fd79d50c
CC' gavin as assignee of bug 729767.
(Reporter)

Comment 76

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9873943&tree=Firefox
(In reply to Chris AtLee [:catlee] from comment #74)
> after more retrying, looks like 8ca3fd79d50c also fails...which shifts the
> blame onto
> https://hg.mozilla.org/integration/mozilla-inbound/rev/8ca3fd79d50c

That would surprise me.  I think we should try backing out the omnijar ordering thing and seeing if that fixes the problem.
(Reporter)

Comment 78

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9901518&tree=Mozilla-Inbound
(Reporter)

Comment 79

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9899668&tree=Mozilla-Inbound
(Reporter)

Comment 80

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9911680&tree=Mozilla-Inbound
(Reporter)

Comment 81

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9913176&tree=Mozilla-Inbound
(Reporter)

Comment 82

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9916793&tree=Mozilla-Inbound
(Reporter)

Comment 83

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9918191&tree=Mozilla-Inbound
(Reporter)

Comment 84

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9922144&tree=Mozilla-Inbound
(Reporter)

Comment 85

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9925392&tree=Mozilla-Inbound
(Reporter)

Comment 86

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9927265&tree=Mozilla-Inbound
(Reporter)

Comment 87

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9947119&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9958931&tree=Mozilla-Inbound
(Reporter)

Comment 89

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9962350&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9962897&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9978724&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=9978895&tree=Mozilla-Inbound
(Reporter)

Comment 93

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9979892&tree=Try
(Reporter)

Comment 94

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9976609&tree=Fx-Team
(Reporter)

Comment 95

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9939889&tree=Fx-Team
(Reporter)

Comment 96

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9986863&tree=Mozilla-Inbound
(Reporter)

Comment 97

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9986017&tree=Mozilla-Inbound
(Reporter)

Comment 98

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9985168&tree=Services-Central
(Reporter)

Comment 99

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9986475&tree=Services-Central
(Reporter)

Comment 100

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9989236&tree=Firefox
(Reporter)

Comment 101

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9989763&tree=Mozilla-Inbound
* All the errors occur in the 'make installer' step, ending with
    Archive:  startupCache.zip
    <some binary data>
    --- Press `Q' to quit, or any other key to continue ---
    command timed out: 1200 seconds without output, attempting to kill
* the relevant code is 
  http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#486  (CC mhommey)
* it affects both win32 and win64 compile jobs; pgo, non-pgo and debug

Possible regressors:
* bug 726656 started optimizing jars again at https://hg.mozilla.org/mozilla-central/rev/f88a05e00f47. This was broken by renaming omni.jar to omni.ja (bug 701875) at https://hg.mozilla.org/mozilla-central/rev/3f0b94325b80, some two months earlier

* there were a five landings in startupCache/ on Feb. 22nd from bug 711297

The latter fits pretty well and is going to be much simpler to investigate, so throwing this over to froydnj to do that. If RelEng can help out with access to a windows machine you have only to ask.
Assignee: nobody → nfroyd
Blocks: 711297
(Reporter)

Comment 103

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9984297&tree=Profiling
(Reporter)

Comment 104

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9994538&tree=Services-Central
(Assignee)

Comment 105

6 years ago
(In reply to Nick Thomas [:nthomas] from comment #102)
> * All the errors occur in the 'make installer' step, ending with
>     Archive:  startupCache.zip
>     <some binary data>
>     --- Press `Q' to quit, or any other key to continue ---
>     command timed out: 1200 seconds without output, attempting to kill
>
> * there were a five landings in startupCache/ on Feb. 22nd from bug 711297

This looks like unzip is attempting to print out the binary data we stash in the zipfile comment, which is not going to end well.  Fixing that is easy enough.
(Reporter)

Comment 106

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9999641&tree=Mozilla-Inbound
(Assignee)

Comment 107

6 years ago
Created attachment 604967 [details] [diff] [review]
patch

Add -q when unzipping the startupCache.zip file so we don't print the binary data in the comment field.
Attachment #604967 - Flags: review?(ted.mielczarek)
(Assignee)

Updated

6 years ago
Whiteboard: [orange] → [orange][autoland-try:-b do -p all -u none -t none]

Updated

6 years ago
Whiteboard: [orange][autoland-try:-b do -p all -u none -t none] → [orange][autoland-in-queue]
Comment on attachment 604967 [details] [diff] [review]
patch

Review of attachment 604967 [details] [diff] [review]:
-----------------------------------------------------------------

Is there a reason this only happens intermittently? Does the data we jam into the comment field change, so it only causes failures sometimes?

::: toolkit/mozapps/installer/packager.mk
@@ +479,5 @@
>  PRECOMPILE_RESOURCE=gre
>  PRECOMPILE_GRE=$$PWD
>  endif
>  
> +# Silence the unzip step so we don't print any binary data from the comment field.

This comment would probably fit better right above the unzip line itself.
Attachment #604967 - Flags: review?(ted.mielczarek) → review+

Comment 109

6 years ago
Autoland Patchset:
	Patches: 604967
	Branch: mozilla-central => try
	Destination: http://hg.mozilla.org/try/pushloghtml?changeset=be056fdb0132
Try run started, revision be056fdb0132. To cancel or monitor the job, see: https://tbpl.mozilla.org/?tree=Try&rev=be056fdb0132
(Assignee)

Comment 110

6 years ago
(In reply to Ted Mielczarek [:ted] from comment #108)
> Is there a reason this only happens intermittently? Does the data we jam
> into the comment field change, so it only causes failures sometimes?

We're stuffing an 8-byte timestamp in there; presumably the bytes in the timestamp every so often cause heartache for the Windows shell (cmd.exe?) when it's trying to decode them.

> > +# Silence the unzip step so we don't print any binary data from the comment field.
> 
> This comment would probably fit better right above the unzip line itself.

Duly noted.  I wasn't sure that'd work, but I guess that's what try is for!
(In reply to Nathan Froyd (:froydnj) from comment #110)
> We're stuffing an 8-byte timestamp in there; presumably the bytes in the
> timestamp every so often cause heartache for the Windows shell (cmd.exe?)
> when it's trying to decode them.

Ah, that makes sense.

> > This comment would probably fit better right above the unzip line itself.
> 
> Duly noted.  I wasn't sure that'd work, but I guess that's what try is for!

The pedantically-correct way to do this is to put the # at the left margin, so it gets parsed as a Makefile comment, but I'm not 100% sure how that works inside a continuation. You might need to just have <tab># blah \ to get it to parse properly. Or it's possible that that just won't parse and you have to live with what you have.
(Assignee)

Comment 112

6 years ago
(In reply to Ted Mielczarek [:ted] from comment #111)
> The pedantically-correct way to do this is to put the # at the left margin,
> so it gets parsed as a Makefile comment, but I'm not 100% sure how that
> works inside a continuation. You might need to just have <tab># blah \ to
> get it to parse properly. Or it's possible that that just won't parse and
> you have to live with what you have.

Try (windows only) seems to indicate this doesn't work:

https://tbpl.mozilla.org/?tree=Try&rev=dbeb08eb7573

I'm just going to call the current comment-not-quite-close-to-code version good enough.
(Assignee)

Comment 113

6 years ago
Try looks good (green builds for almost all major platforms), so marking checkin-needed.
Keywords: checkin-needed
I needed to rebuild the windows builds on try 6 or 7 times to make sure it wasn't still happening. Can you do that?
(Reporter)

Comment 115

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10006602&tree=Mozilla-Inbound
(Assignee)

Comment 116

6 years ago
(In reply to Chris AtLee [:catlee] from comment #114)
> I needed to rebuild the windows builds on try 6 or 7 times to make sure it
> wasn't still happening. Can you do that?

Pushed a 3rd attempt to try:

https://tbpl.mozilla.org/?tree=Try&rev=cdab92f07ac7

which along with an earlier windows-only run:

https://tbpl.mozilla.org/?tree=Try&rev=b58312ae711d

and the autoland run is three runs, at least.  I'm pretty confident this will fix things, though.
(In reply to Nathan Froyd (:froydnj) from comment #116)
> (In reply to Chris AtLee [:catlee] from comment #114)
> > I needed to rebuild the windows builds on try 6 or 7 times to make sure it
> > wasn't still happening. Can you do that?
> 
> Pushed a 3rd attempt to try:
> 
> https://tbpl.mozilla.org/?tree=Try&rev=cdab92f07ac7

You know you can trigger rebuilds via tbpl, right? Click on the win32 opt/debug build, and on the bottom left pane there will be a blue plus icon. Click that to re-trigger it.
(Assignee)

Comment 118

6 years ago
(In reply to Chris AtLee [:catlee] from comment #117)
> You know you can trigger rebuilds via tbpl, right? Click on the win32
> opt/debug build, and on the bottom left pane there will be a blue plus icon.
> Click that to re-trigger it.

I am enlightened.  Thanks!
(Reporter)

Comment 119

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10010128&tree=Mozilla-Inbound
(Reporter)

Comment 120

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10010478&tree=Mozilla-Inbound
(Reporter)

Comment 121

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10004783&tree=Maple
(Reporter)

Comment 122

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10012785&tree=Mozilla-Inbound
(Reporter)

Comment 123

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013197&tree=Mozilla-Inbound
(Reporter)

Comment 124

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013525&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013238&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013335&tree=Firefox
(Reporter)

Comment 126

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013352&tree=Mozilla-Inbound
(Reporter)

Comment 127

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10012811&tree=Maple
(Reporter)

Comment 128

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013631&tree=Maple
(Reporter)

Comment 129

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10013656&tree=Mozilla-Inbound

Comment 130

6 years ago
Try run for be056fdb0132 is complete.
Detailed breakdown of the results available here:
	https://tbpl.mozilla.org/?tree=Try&rev=be056fdb0132
Results (out of 16 total builds):
    success: 16
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/autolanduser@mozilla.com-be056fdb0132

Updated

6 years ago
Whiteboard: [orange][autoland-in-queue] → [orange]
(Reporter)

Comment 131

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10017381&tree=Mozilla-Inbound
(Reporter)

Comment 132

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10020830&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=10032328&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=10033340&tree=Firefox
Duplicate of this bug: 735302
Duplicate of this bug: 735302
Duplicate of this bug: 735302
(Assignee)

Comment 138

6 years ago
(In reply to Chris AtLee [:catlee] from comment #114)
> I needed to rebuild the windows builds on try 6 or 7 times to make sure it
> wasn't still happening. Can you do that?

https://tbpl.mozilla.org/?tree=Try&rev=b58312ae711d

says that we've been green for about that many builds with the patch.  WDYT?
LGTM
(Reporter)

Comment 140

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10041384&tree=Mozilla-Inbound
(Reporter)

Comment 141

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10051169&tree=Mozilla-Inbound
(Reporter)

Comment 142

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10046325&tree=Mozilla-Inbound
(Reporter)

Comment 143

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10054097&tree=Mozilla-Inbound
http://hg.mozilla.org/integration/mozilla-inbound/rev/c7d9848ffff5

Updated

6 years ago
Keywords: checkin-needed
(Reporter)

Comment 145

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10074374&tree=Firefox
(Reporter)

Comment 146

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10083721&tree=Firefox
https://hg.mozilla.org/mozilla-central/rev/c7d9848ffff5
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla14
Attachment #604967 - Flags: approval-mozilla-aurora?
status-firefox13: --- → affected
Attachment #604967 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(Reporter)

Comment 148

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10155424&tree=Mozilla-Aurora
(Reporter)

Comment 149

6 years ago
https://hg.mozilla.org/releases/mozilla-aurora/rev/abf433daab89
status-firefox13: affected → fixed
Whiteboard: [orange] → [orange][qa-]
Keywords: intermittent-failure
Whiteboard: [orange][qa-] → [qa-]
You need to log in before you can comment on or make changes to this bug.