The default bug view has changed. See this FAQ.
Last Comment Bug 1026870 - Something wrong with windows build slaves ("LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt")
: Something wrong with windows build slaves ("LINK : fatal error LNK1123: failu...
Status: RESOLVED FIXED
[release-impacting]
: intermittent-failure
Product: Release Engineering
Classification: Other
Component: Buildduty (show other bugs)
: unspecified
: All Windows 7
-- normal (vote)
: ---
Assigned To: Q
: Justin Wood (:Callek)
:
Mentors:
Depends on: 1057549 1169325
Blocks: b-2008-ix-0020 b-2008-ix-0084 b-2008-ix-0116 b-2008-ix-0023 b-2008-ix-0167 1019165 b-2008-ix-0021 b-2008-ix-0022 b-2008-ix-0024 b-2008-ix-0025 b-2008-ix-0026 b-2008-ix-0027 b-2008-ix-0028 b-2008-ix-0029 b-2008-sm-0050 b-2008-ix-0140 1062877 b-2008-ix-0002 b-2008-ix-0003 b-2008-ix-0004 b-2008-ix-0005 b-2008-ix-0078 b-2008-ix-0127 b-2008-ix-0133 b-2008-ix-0008
  Show dependency treegraph
 
Reported: 2014-06-17 20:59 PDT by Mike Hommey [:glandium]
Modified: 2017-01-25 03:25 PST (History)
14 users (show)
See Also:
Crash Signature:
(edit)
Machine State: ---
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description User image Mike Hommey [:glandium] 2014-06-17 20:59:58 PDT
See https://tbpl.mozilla.org/?tree=Try&jobname=win&rev=c6fe0466209b for example. It's a try with sccache disabled (to rule it out), and exhibiting the same problem as other older builds on those slaves.

The error is:
LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt 

usually when linking ICU, but it also happens when linking something from the crash reporter on some other builds.

Apparently, only the slaves between 20 and 29 are affected, although I haven't observed failures on all of them because they don't have all recently built something, all the slaves that do fail to build are in that range.

It started happening today or yesterday.

Related information about this error code:
http://msdn.microsoft.com/en-us/library/7dz62kfh.aspx
http://support.microsoft.com/kb/2757355
Comment 1 User image Nick Thomas [:nthomas] 2014-06-17 21:46:43 PDT
b-2008-ix-0020 through b-2008-ix-0029 are disabled in slavealloc.
Comment 2 User image Jordan Lund (:jlund) 2014-06-17 22:32:29 PDT
so these slaves do seem to be using vs2010 still as you can see with lines like /c/PROGRA~2/MICROS~2.0. The slave still shows this shortname still points to:
    /c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 10.0/

so it is not using the junction set up at /c/tools/vs2013/.

I am not sure why this is failing here. On my testing slave I was able to build this builder on m-c(not try) against vs2010 while vs2013 was also installed on the machine: http://people.mozilla.org/~jlund/vs2010-mozilla-central-winxp-opt.log
Comment 3 User image Jordan Lund (:jlund) 2014-06-17 23:03:49 PDT
the fallout of burning these jobs falls on my head.

Regardless of my test slave not catching hitting this for whatever reason, it looks like this is a common issue. The resolution seems to be to install Visual Studio 2010 SP1 as mentioned in here[1] and also here[2].

pinging Q and dmajor for relops and dev perspective - Is it possible to do update our machine's VS2010 install to SP1 or should we re-image these hosts and look at coming up with another alternative?

[1] - from comment 1 - http://support.microsoft.com/kb/2757355
[2] - http://stackoverflow.com/questions/10888391/error-link-fatal-error-lnk1123-failure-during-conversion-to-coff-file-inval
Comment 4 User image Justin Wood (:Callek) 2014-06-17 23:08:25 PDT
From stack overflow: "Note that installing VS 2010 SP1 will remove the 64-bit compilers. You need to install the VS 2010 SP1 compiler pack to get them back."
Comment 5 User image Mike Hommey [:glandium] 2014-06-17 23:26:19 PDT
You should check directly on one of those problematic slaves, and check the first microsoft link in comment #0 before considering upgrading MSVC.
Comment 6 User image Q 2014-06-18 00:57:08 PDT
I will need to follow up on this one with my team as we applied the same gpo here as we did on the test slave. If sp1 does not fix this immediately we will re image the machines and then I think we should pull one and teat against it.
Comment 7 User image Mike Hommey [:glandium] 2014-06-18 01:18:17 PDT
I'd rather you do it the other way around. Installing SP1 means changing the compiler. This can have any sorts of effects, like sucking more memory to do PGO linkage, or miscompiling, and that shouldn't be done lightly.
Comment 8 User image Benjamin Smedberg [:bsmedberg] 2014-06-18 06:23:08 PDT
SP1 involves a change to the CRT for Firefox releases, and we should avoid that.
Comment 9 User image Jordan Lund (:jlund) 2014-06-18 09:10:35 PDT
OK I've asked to revert the changes I requested: https://bugzilla.mozilla.org/show_bug.cgi?id=1019165#c12

re-imaging those machines sounds like best course of action for now.
Comment 10 User image Q 2014-06-18 10:13:40 PDT
Postmortem shows this:
 VS 2010 sp1 and the compiler pack was installed in conjunction with VS 2012 for initial testing on the test machine. We were planning on rolling it with VS 2012 per releng instructions then we (Releng/Relops) made the decision to  skip 2012 and go to 2013  and that 2010 patch  step was lost since the test machine already had it. To move forward we will need 2010 sp1 and the compiler reinstalled before going to 2013.
Just in case  I will get a 2010 sp1/compiler pack installer GPO ready but I will need confirmation that we we can run with 2010 SP1 on the builders
Comment 11 User image Q 2014-06-18 10:14:04 PDT
The affected builders are being re-imaged now
Comment 12 User image Mike Hommey [:glandium] 2014-06-18 10:24:14 PDT
(In reply to Q from comment #10)
> Just in case  I will get a 2010 sp1/compiler pack installer GPO ready but I
> will need confirmation that we we can run with 2010 SP1 on the builders

See comment 7 and comment 8.
Comment 13 User image Q 2014-06-18 10:31:28 PDT
Thanks I missed comment 8
Comment 14 User image Q 2014-06-18 10:31:52 PDT
Thanks I missed comment 8
Comment 15 User image Q 2014-06-18 11:35:58 PDT
All slaves re-imaged and confirmed except 0023 which was slow due to a disk check being forced.
Comment 16 User image Q 2014-06-18 13:15:10 PDT
all slaves done.
Comment 17 User image Q 2014-06-18 13:15:47 PDT
Who is in charge of getting these back in?
Comment 18 User image Nick Thomas [:nthomas] 2014-06-18 13:26:33 PDT
That would be the lucky buildduty, aka jlund.
Comment 19 User image Jordan Lund (:jlund) 2014-06-18 23:23:47 PDT
I didn't get around to this today. I will add them first thing in the morning. Win builders are far from our worst wait_times so this should be fine.
Comment 20 User image Jordan Lund (:jlund) 2014-06-19 00:03:14 PDT
Also WRT actually installing vs2013, it looks like there are two options:

1) we have two separate pools of windows build machines. 1 for vs2010 and 1 for vs2013 where we slowly fade out the former.

2) we install VS2010 SP1 and VS2013 on all our build machines.

I would like to weigh out the pros/cons of both options in a discussion with folks more knowledgeable than myself.

Yes, the CRT will change, and we will have to measure performance in all our builders to check for diffs in things like PGO. This could be bad. But I'd like to discuss how bad, and is it worse than dividing up our win pools. Dividing up is not very optimal from Mozilla's release engineering side of things and will come with its own consequences.

Armen - I believe you have done some work on investigating vs2010 SPI before, maybe with the vs2012 work Q mentioned here: https://bugzilla.mozilla.org/show_bug.cgi?id=1026870#c10 Do you have any input?
bhearsum - I heard rumors that you used to do all the 'windows' stuff before jhopkins and armen. any thoughts?

maybe a quick group meeting with bsmedberg and/or glandium to sort this out would be best?
Comment 21 User image Ben Hearsum (:bhearsum) 2014-06-19 05:19:03 PDT
(In reply to Jordan Lund (:jlund) from comment #20)
> Also WRT actually installing vs2013, it looks like there are two options:
> 
> 1) we have two separate pools of windows build machines. 1 for vs2010 and 1
> for vs2013 where we slowly fade out the former.
> 
> 2) we install VS2010 SP1 and VS2013 on all our build machines.
> 
> I would like to weigh out the pros/cons of both options in a discussion with
> folks more knowledgeable than myself.
> 
> Yes, the CRT will change, and we will have to measure performance in all our
> builders to check for diffs in things like PGO. This could be bad. But I'd
> like to discuss how bad, and is it worse than dividing up our win pools.
> Dividing up is not very optimal from Mozilla's release engineering side of
> things and will come with its own consequences.

I'm extremely wary of changing anything in the toolchain for Beta, Release, and ESR. However, it *is* early in a Beta cycle, so we'd have lots of time to prove out the change with Beta users. This option would require RelMan sign off for sure because of the risk involved.

I haven't been following this bug until now - I'm assuming the CRT only changes for option #2. If so, can I ask why it must change? We already set PATH/LIB/etc. for our compiler - can we not pick up the old CRT by setting one of those? as far as I know, we've done all of our compilers upgrades without dividing the pool, so I'd like to understand what's special about this one. Eg: bug 563318 was the tracker for 2010. bug 563317 shows us installing it on existing machines.

The other downside to dividing up the pool is that we end up with worse machine utilization in both pools, which reduces our throughput for changes overall.
Comment 22 User image Ed Morley (Away 22nd-27th, 29th-2nd) [:emorley] 2014-06-19 05:23:09 PDT
I think dividing the pool is going to risk us missing all sorts of weirdness that we don't initially associate with a certain set of machines, and instead think is some sporadic intermittent issue.
Comment 23 User image Mike Hommey [:glandium] 2014-06-19 05:56:14 PDT
How about option 3: look into http://msdn.microsoft.com/en-us/library/7dz62kfh.aspx and see if it's not possible to *not* install SP1.
Comment 24 User image Mike Hommey [:glandium] 2014-06-19 06:01:53 PDT
(In reply to Ben Hearsum [:bhearsum] from comment #21)
> as far as I know, we've done all of our compilers upgrades
> without dividing the pool, so I'd like to understand what's special about
> this one.

AIUI, VS2012/2013 installs a CVTRES.EXE that is not compatible with VS2010, and, it looks like it puts it in some shared location, overwriting VS2010's, or in a directory that has precedence in $PATH. I hope it's the latter.

As to why that didn't happen when we switched to VS2010, in all likeliness, VS2010 wasn't installing a CRTRES.EXE that is not compatible with VS2005/VS2008.
Comment 25 User image Armen Zambrano - Back on March 27th [:armenzg] (EDT/UTC-4) 2014-06-19 06:05:12 PDT
As Ed and Ben mentions we should have divided pools unless we have to.

jlund, with regards previous knowledge, I have only one input: order matters with Windows, specially when talking about Visual Studio. Installers and uninstallers can be doing things we would not expect them too.

Another tip: do not try to install in custom places unless you have to. I had once a variable that you could set for a specific path, however, not every component of VS would install there as they had other secret environment variables.

Best of luck.

PS = Don't call Ben a Windows expert or he'll come at you :P
Comment 26 User image Benjamin Smedberg [:bsmedberg] 2014-06-19 06:41:04 PDT
With only MSVC2013 installed, cvtres.exe is installed to c:\Program Files (x86)/Microsoft Visual Studio 12.0/VC/BIN/cvtres.exe

If cvtres.exe is indeed the problem, we should check and see which version is on the PATH currently, and which version ends up on the PATH in the "bad" case, and hopefully we can just munge the PATH to get the right thing on top in each case.
Comment 27 User image Jordan Lund (:jlund) 2014-06-19 11:21:55 PDT
it sounds like bsmedberg and glandium's suggestion (option 3) is worth perusing.

I've filed to get another test machine but with vs2010 (the non sp1 version) and vs2013 installed on it: bug 1027745

I am closing this bug for now as this was a buildduty infra error bug which has been resolved. tracking getting the 10 re-imaged machines back into production will happen in the original rollout bug: Bug 1019165

overall goal of vs2013 in prod is still tracked here: Bug 1009807 - Figure out the correct path setup and mozconfigs for automation-driven MSVC2013 builds

ty you all for your aid thus far.
Comment 28 User image Ben Hearsum (:bhearsum) 2014-06-19 11:23:02 PDT
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #26)
> With only MSVC2013 installed, cvtres.exe is installed to c:\Program Files
> (x86)/Microsoft Visual Studio 12.0/VC/BIN/cvtres.exe
> 
> If cvtres.exe is indeed the problem, we should check and see which version
> is on the PATH currently, and which version ends up on the PATH in the "bad"
> case, and hopefully we can just munge the PATH to get the right thing on top
> in each case.

And even if cvtres.exe is overwritten by default, we could manually copy the older version somewhere and point PATH at it where needed, I think.
Comment 29 User image David Major [:dmajor] 2014-06-22 15:10:44 PDT
Some potential workarounds: http://social.msdn.microsoft.com/Forums/vstudio/en-US/d10adba0-e082-494a-bb16-2bfc039faa80/vs2012-rc-installation-breaks-vs2010-c-projects?forum=vssetup

It seems the root cause is a .NET DLL dependency, which people have worked around by overwriting cvtres, getting a different one onto PATH, or tinkering with the machine's .NET installation. 

(I don't see a way to link to specific comments, but you can search for "Proposed as answer")
Comment 30 User image Ed Morley (Away 22nd-27th, 29th-2nd) [:emorley] 2014-08-06 06:49:24 PDT
Tweaking summary to make the issue easier to find, particularly given we're unfortunately seeing it again not that bug 1019165 has started rolling out again.
Comment 31 User image Ed Morley (Away 22nd-27th, 29th-2nd) [:emorley] 2014-08-06 06:49:56 PDT
Adding keyword so TBPL sees this.
Comment 32 User image Treeherder Robot 2014-08-06 10:33:48 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=45345935&tree=Mozilla-Inbound
WINNT 5.2 mozilla-inbound build on 2014-08-06 09:30:21
revision: 43529d2194a0
slave: b-2008-ix-0120

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [intl/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 33 User image Treeherder Robot 2014-08-06 12:02:10 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=45354471&tree=Mozilla-Inbound
WINNT 5.2 mozilla-inbound leak test build on 2014-08-06 11:28:14
revision: 0ff2c61b9e41
slave: b-2008-ix-0120

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 34 User image Ryan VanderMeulen [:RyanVM] 2014-08-06 12:02:45 PDT
I filed bug 1049794 for b-2008-ix-0120 and disabled it in slavealloc.
Comment 35 User image Treeherder Robot 2014-08-15 10:47:09 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=46046820&tree=B2g-Inbound
WINNT 5.2 b2g-inbound non-unified on 2014-08-15 09:37:42
revision: 2fd5cd2f0272
slave: b-2008-ix-0069

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [xul.dll] Error 1123
mozmake.exe[4]: *** [toolkit/library/target] Error 2
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 36 User image Q 2014-08-15 10:49:33 PDT
This was due to a VS 2013 push to a machine with an active job. Markco can you make sure these are disabled and jobs are DONE BEFORE deploying 2013 ?
Comment 37 User image Treeherder Robot 2014-08-19 06:14:40 PDT
edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=46253937&tree=Fx-Team
WINNT 5.2 fx-team leak test build on 2014-08-19 04:58:42
revision: 836c52cc3c05
slave: b-2008-sm-0058

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 38 User image Treeherder Robot 2014-08-19 13:20:55 PDT
dkeeler
https://tbpl.mozilla.org/php/getParsedLog.php?id=46287111&tree=Try
WINNT 5.2 try build on 2014-08-19 11:58:34
revision: 45a1c8511d34
slave: b-2008-sm-0007

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 39 User image Treeherder Robot 2014-08-19 15:09:09 PDT
kats
https://tbpl.mozilla.org/php/getParsedLog.php?id=46298012&tree=Try
WINNT 5.2 try build on 2014-08-19 14:23:49
revision: dec1673e625f
slave: b-2008-sm-0025

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 40 User image Treeherder Robot 2014-08-20 09:30:38 PDT
edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=46366013&tree=B2g-Inbound
WINNT 5.2 b2g-inbound non-unified on 2014-08-20 08:30:17
revision: 72afaf22a760
slave: b-2008-sm-0050

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 41 User image Treeherder Robot 2014-08-20 12:05:15 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=46379210&tree=B2g-Inbound
b2g_b2g-inbound_win32_gecko build on 2014-08-20 11:05:16
revision: 1f79d92c1c7c
slave: b-2008-sm-0050

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [mozglue.dll] Error 1123
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [mozglue/build/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 42 User image Treeherder Robot 2014-08-20 12:59:17 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=46384503&tree=Mozilla-B2g30-v1.4
b2g_mozilla-b2g30_v1_4_win32_gecko-debug build on 2014-08-20 11:02:28
revision: 211076a8bbb6
slave: b-2008-sm-0042

command timed out: 3600 seconds without output, attempting to kill
LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
Comment 43 User image Treeherder Robot 2014-08-20 13:46:13 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=46389905&tree=Mozilla-Central
b2g_mozilla-central_win32_gecko-debug build on 2014-08-20 12:54:36
revision: ff3303c6c8c9
slave: b-2008-sm-0050

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [mozglue.dll] Error 1123
mozmake.exe[4]: *** [mozglue/build/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 44 User image Treeherder Robot 2014-08-20 16:28:28 PDT
KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46404495&tree=Mozilla-Inbound
b2g_mozilla-inbound_win32_gecko-debug build on 2014-08-20 15:51:16
revision: 477785eeb00e
slave: b-2008-sm-0038

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [b2g-bin.exe] Error 1123
mozmake.exe[4]: *** [b2g/app/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 45 User image Treeherder Robot 2014-08-20 18:09:24 PDT
KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46410008&tree=Mozilla-Central
WINNT 5.2 mozilla-central pgo-build on 2014-08-20 16:30:22
revision: e7806c9c83f3
slave: b-2008-sm-0038

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[7]: *** [all-recursive] Error 2
mozmake.exe[6]: *** [buildicu] Error 2
mozmake.exe[5]: *** [config/external/icu/target] Error 2
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [compile] Error 2
mozmake.exe[3]: *** [default] Error 2
mozmake.exe[2]: *** [realbuild] Error 2
mozmake.exe[1]: *** [profiledbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 46 User image Treeherder Robot 2014-08-20 19:17:19 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=46419218&tree=Mozilla-B2g32-v2.0
WINNT 5.2 mozilla-b2g32_v2_0 leak test build on 2014-08-20 18:07:16
revision: 677df505b707
slave: b-2008-sm-0053

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [intl/icu/compile] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 47 User image Treeherder Robot 2014-08-20 20:19:37 PDT
dmajor
https://tbpl.mozilla.org/php/getParsedLog.php?id=46424619&tree=Try
WINNT 5.2 try leak test build on 2014-08-20 19:40:49
revision: 89f8001e0a20
slave: b-2008-sm-0008

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 48 User image Treeherder Robot 2014-08-21 21:37:29 PDT
jwalden
https://tbpl.mozilla.org/php/getParsedLog.php?id=46530031&tree=Try
WINNT 5.2 try leak test build on 2014-08-21 21:01:50
revision: 4ec566a275f9
slave: b-2008-sm-0020

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 49 User image Treeherder Robot 2014-08-22 09:03:35 PDT
edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=46564418&tree=B2g-Inbound
b2g_b2g-inbound_win32_gecko-debug build on 2014-08-22 07:44:09
revision: fd0207c7e55f
slave: b-2008-sm-0051

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [mozglue.dll] Error 1123
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [mozglue/build/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 50 User image Hal Wine [:hwine] (use NI) 2014-09-02 13:32:37 PDT
reopening, as:
 a) 13 reports by sheriffs since closed
 b) lots of issues hit in bug 1057549 which point to very slow machines

(b) hit a number of release builds this cycle.
Comment 51 User image David Major [:dmajor] 2014-09-02 14:48:39 PDT
I have a suspicion: both this issue and bug 1057229 comment 5 could be explained by some builders picking up an incomplete VS2013 setup (i.e. without the cvtres fixup, or without Update 3). I don't know how that might happen, though.
Comment 52 User image Chris Cooper [:coop] 2014-09-02 16:33:44 PDT
Could the build slave become active before the GPO for VS2013 is fully applied? I assume it's a long install.
Comment 54 User image Ryan VanderMeulen [:RyanVM] 2014-12-17 06:16:38 PST
We're still seeing these failures pretty regularly, especially on the release branches.
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=72561&repo=mozilla-b2g32_v2_0
Comment 55 User image Jordan Lund (:jlund) 2014-12-17 11:27:07 PST
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #54)
> We're still seeing these failures pretty regularly, especially on the
> release branches.
> https://treeherder.mozilla.org/ui/logviewer.html#?job_id=72561&repo=mozilla-
> b2g32_v2_0

ni: q: this issue just doesn't want to go away! :) the job Ryan is pointing to was against b-2008-ix-0005 @ Tue Dec 16 15:13:49 2014. is it possible this slave recently imaged or had an incomplete gpo?
Comment 56 User image Q 2014-12-17 16:54:36 PST
HMM well we are no longer applying 2013 from GPO only in the base image so it in theory can no longer get an incomplete install. I am doing a full audit to make that is 100% true
Comment 57 User image Treeherder Robot 2014-12-22 16:26:10 PST
submit_timestamp: 2014-12-22T16:26:09
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-beta&job_id=192635
repository: mozilla-beta
who: philringnalda[at]gmail[dot]com
machine: b-2008-ix-0005
buildname: WINNT 5.2 mozilla-beta build
revision: bde0494c6bbd

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[7]: *** [all-recursive] Error 2
mozmake.exe[6]: *** [buildicu] Error 2
mozmake.exe[5]: *** [config/external/icu/target] Error 2
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [compile] Error 2
mozmake.exe[3]: *** [default] Error 2
mozmake.exe[2]: *** [realbuild] Error 2
mozmake.exe[1]: *** [profiledbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 58 User image Treeherder Robot 2014-12-22 16:51:39 PST
submit_timestamp: 2014-12-22T16:51:38
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-beta&job_id=192960
repository: mozilla-beta
who: philringnalda[at]gmail[dot]com
machine: b-2008-ix-0008
buildname: WINNT 5.2 mozilla-beta leak test build
revision: 54e086d2dcc3

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 59 User image Treeherder Robot 2014-12-22 17:36:10 PST
submit_timestamp: 2014-12-22T17:36:09
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-beta&job_id=193348
repository: mozilla-beta
who: philringnalda[at]gmail[dot]com
machine: b-2008-ix-0001
buildname: WINNT 5.2 mozilla-beta build
revision: bde0494c6bbd

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[7]: *** [all-recursive] Error 2
mozmake.exe[6]: *** [buildicu] Error 2
mozmake.exe[5]: *** [config/external/icu/target] Error 2
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [compile] Error 2
mozmake.exe[3]: *** [default] Error 2
mozmake.exe[2]: *** [realbuild] Error 2
mozmake.exe[1]: *** [profiledbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 60 User image Jordan Lund (:jlund) 2014-12-22 18:30:58 PST
(In reply to Q from comment #56)
> HMM well we are no longer applying 2013 from GPO only in the base image so
> it in theory can no longer get an incomplete install. I am doing a full
> audit to make that is 100% true

hmm, 008, 001, and 005 (twice) have recently hit this. aside from this error, sheriffs are also seeing timeouts, essentially https://bugzil.la/1055876 again. Since this bug and 1055876 were related last time, I wonder if there is something similar at play, granted this isn't part of GPO anymore.

sheriffs are now being forced disabling slaves that hit this to to (a) stress the importance of this and (b) show the scope of what slaves are 'bad'.

Q, how did the audit go? would you be able to spend some brain cycles assisting me with this? I'm free most this week. thanks for bearing with me :)
Comment 61 User image Q 2014-12-22 21:20:33 PST
Couldn't find anything obvious the gpo isn't at play anywhere.  I have set time aside tomorrow to slog through this one. Maybe we can do some vidyo time after I do some more digging in the morning?
Comment 62 User image Q 2014-12-26 13:48:10 PST
Do these errors happen to start when we started installing the new HG version?
Comment 63 User image Q 2014-12-26 13:53:03 PST
HG started rolling out on December 11th 2014
Comment 64 User image Jordan Lund (:jlund) 2014-12-31 10:40:16 PST
(In reply to Q from comment #63)
> HG started rolling out on December 11th 2014

TBH - I am not sure. Ryan made a comment on Dec 17th after a 3 month gap saying that we are still seeing this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1026870#c54

Ryan, is it possible that this only started acting up again around the 11th?
Comment 65 User image Treeherder Robot 2015-01-03 11:59:59 PST
submit_timestamp: 2015-01-03T11:59:58
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-esr31&job_id=41550
repository: mozilla-esr31
who: philringnalda[at]gmail[dot]com
machine: b-2008-ix-0140
buildname: WINNT 5.2 mozilla-esr31 leak test build
revision: d13776405921

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [intl/icu/compile] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 66 User image Treeherder Robot 2015-01-04 12:52:30 PST
submit_timestamp: 2015-01-04T12:52:29
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-beta&job_id=202508
repository: mozilla-beta
who: philringnalda[at]gmail[dot]com
machine: b-2008-ix-0008
buildname: WINNT 5.2 mozilla-beta leak test build
revision: baded7123fe8

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 67 User image Ryan VanderMeulen [:RyanVM] 2015-01-04 12:56:01 PST
(In reply to Jordan Lund (:jlund) PTO till Jan 14th from comment #64)
> Ryan, is it possible that this only started acting up again around the 11th?

Could be, can't say I remember at this point.
Comment 68 User image Treeherder Robot 2015-01-05 05:47:59 PST
submit_timestamp: 2015-01-05T05:47:59
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-beta&job_id=202975
repository: mozilla-beta
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0127
buildname: WINNT 5.2 mozilla-beta build
revision: c061a8d7d529

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[7]: *** [all-recursive] Error 2
mozmake.exe[6]: *** [buildicu] Error 2
mozmake.exe[5]: *** [config/external/icu/target] Error 2
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [compile] Error 2
mozmake.exe[3]: *** [default] Error 2
mozmake.exe[2]: *** [realbuild] Error 2
mozmake.exe[1]: *** [profiledbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 69 User image Treeherder Robot 2015-01-05 16:40:55 PST
submit_timestamp: 2015-01-05T16:40:53
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-b2g30_v1_4&job_id=99039
repository: mozilla-b2g30_v1_4
who: wkocher[at]mozilla[dot]com
machine: b-2008-ix-0116
buildname: b2g_mozilla-b2g30_v1_4_win32_gecko build
revision: bc58850e8b51

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
Comment 70 User image Treeherder Robot 2015-01-05 16:40:58 PST
submit_timestamp: 2015-01-05T16:40:57
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-b2g30_v1_4&job_id=99040
repository: mozilla-b2g30_v1_4
who: wkocher[at]mozilla[dot]com
machine: b-2008-ix-0078
buildname: b2g_mozilla-b2g30_v1_4_win32_gecko-debug build
revision: bc58850e8b51

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
Comment 71 User image Phil Ringnalda (:philor) 2015-01-05 16:45:17 PST
Unfortunately, looking a whole lot like "anything that has been reimaged in the last month or so."
Comment 72 User image Q 2015-01-06 12:37:22 PST
Per discussions it is possible that the HG install having parts of the VS redistributable included in them may be causing the problem. The fix may be the correction of the system PATH or investigating the overlap. There are cases of machines failing that have not been re-imaged
Comment 73 User image Treeherder Robot 2015-01-07 11:14:02 PST
submit_timestamp: 2015-01-07T11:14:01
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=comm-esr31&job_id=1305
repository: comm-esr31
who: arai_a[at]mac[dot]com
machine: b-2008-ix-0016
buildname: TB WINNT 5.2 comm-esr31 build
revision: 0b704a36a620

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
subprocess.CalledProcessError: Command '['C:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-c-esr31-w32-000000000000000\\build\\mozilla\\build\\pymake\\make.py', '-f', 'client.mk', 'build', 'MOZ_BUILD_DATE=20150107093202']' returned non-zero exit status 2
Comment 74 User image Treeherder Robot 2015-01-08 14:39:29 PST
submit_timestamp: 2015-01-08T14:39:29
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-beta&job_id=206873
repository: mozilla-beta
who: wkocher[at]mozilla[dot]com
machine: b-2008-ix-0016
buildname: WINNT 5.2 mozilla-beta build
revision: 0003236c17c1

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[7]: *** [all-recursive] Error 2
mozmake.exe[6]: *** [buildicu] Error 2
mozmake.exe[5]: *** [config/external/icu/target] Error 2
mozmake.exe[5]: *** Waiting for unfinished jobs....
mozmake.exe[4]: *** [compile] Error 2
mozmake.exe[3]: *** [default] Error 2
mozmake.exe[2]: *** [realbuild] Error 2
mozmake.exe[1]: *** [profiledbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 75 User image David Major [:dmajor] 2015-01-08 14:45:50 PST
In bug 1117900 comment 20, Ryan found a builder with an old VS install. Relevant?
Comment 76 User image Q 2015-01-08 14:48:46 PST
I was hoping it was however, it doesn't seem likely as we have found machines reporting the correct VS version with the issue. The machines with the wrong version have been tracked to testing machines that did not get re-imaged so far.
Comment 77 User image Q 2015-01-08 14:49:12 PST
Since it was brought up in IRC today. The three month gap is concerning in that nothing we can find changed in the build or re-image process and this seems to happen on machines that have and have not been re-imaged in the failure window. However, some GPO changes were made around the time the failures started for git and HG installs.
Comment 78 User image Chris Cooper [:coop] 2015-01-08 18:47:36 PST
I’ve started reimaging the dependent machines in batches of 5, with an hour wait between batches.
Comment 79 User image Chris Cooper [:coop] 2015-01-08 21:18:59 PST
(In reply to Chris Cooper [:coop] from comment #78)
> I’ve started reimaging the dependent machines in batches of 5, with an hour
> wait between batches.

These are all re-imaged and re-enabled now.
Comment 80 User image Treeherder Robot 2015-01-09 06:29:14 PST
submit_timestamp: 2015-01-09T06:29:13
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g32_v2_0&job_id=82351
repository: mozilla-b2g32_v2_0
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0167
buildname: b2g_mozilla-b2g32_v2_0_win32_gecko nightly
revision: d647b5114305

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [crashinject.exe] Error 1123
mozmake.exe[4]: *** [build/win32/libs] Error 2
mozmake.exe[3]: *** [libs] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 81 User image Treeherder Robot 2015-01-09 07:19:03 PST
submit_timestamp: 2015-01-09T07:19:02
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g34_v2_1&job_id=65761
repository: mozilla-b2g34_v2_1
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0140
buildname: WINNT 5.2 mozilla-b2g34_v2_1 build
revision: 2fe39f7c385f

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 82 User image Treeherder Robot 2015-01-09 07:20:33 PST
submit_timestamp: 2015-01-09T07:20:32
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g30_v1_4&job_id=100167
repository: mozilla-b2g30_v1_4
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0004
buildname: b2g_mozilla-b2g30_v1_4_win32_gecko-debug build
revision: 082dad73b723

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
Comment 83 User image Treeherder Robot 2015-01-09 07:20:44 PST
submit_timestamp: 2015-01-09T07:20:43
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-esr31&job_id=44288
repository: mozilla-esr31
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0004
buildname: WINNT 5.2 mozilla-esr31 build
revision: c34347a324e6

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [crashinject.exe] Error 1123
mozmake.exe[5]: *** [libs] Error 2
mozmake.exe[4]: *** [libs] Error 2
mozmake.exe[3]: *** [default] Error 2
mozmake.exe[2]: *** [realbuild] Error 2
mozmake.exe[1]: *** [profiledbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 84 User image Treeherder Robot 2015-01-09 07:23:50 PST
submit_timestamp: 2015-01-09T07:23:49
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g34_v2_1&job_id=65762
repository: mozilla-b2g34_v2_1
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0162
buildname: WINNT 5.2 mozilla-b2g34_v2_1 leak test build
revision: 2fe39f7c385f

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 85 User image Treeherder Robot 2015-01-09 07:40:40 PST
submit_timestamp: 2015-01-09T07:40:39
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g34_v2_1&job_id=65785
repository: mozilla-b2g34_v2_1
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0133
buildname: WINNT 5.2 mozilla-b2g34_v2_1 build
revision: 44874e713ddf

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 86 User image Treeherder Robot 2015-01-09 08:14:06 PST
submit_timestamp: 2015-01-09T08:14:06
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g32_v2_0&job_id=82433
repository: mozilla-b2g32_v2_0
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0016
buildname: b2g_mozilla-b2g32_v2_0_win32_gecko build
revision: 6589b217ba71

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [crashinject.exe] Error 1123
mozmake.exe[4]: *** [build/win32/libs] Error 2
mozmake.exe[3]: *** [libs] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 87 User image Q 2015-01-12 09:43:38 PST
I think this might be fixed with GPO updates discussed in IRC ( will transcribe here when I get to a non laptop keyboard). We haven't seen any new reports since 01/09. Is there a way to confirm the fix?
Comment 88 User image Ryan VanderMeulen [:RyanVM] 2015-01-12 09:52:03 PST
I think that the lack of new slave disablings or reports in this bug since is a positive sign :) Let's give it a week or so and resolve the bug if things look good?
Comment 89 User image Chris Cooper [:coop] 2015-01-12 14:25:52 PST
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #88)
> I think that the lack of new slave disablings or reports in this bug since
> is a positive sign :) Let's give it a week or so and resolve the bug if
> things look good?

Sounds good to me.
Comment 90 User image Jordan Lund (:jlund) 2015-01-14 16:00:28 PST
apologies, I was on PTO.

Thanks Q for implementing a likely fix.
Comment 91 User image Treeherder Robot 2015-05-22 09:34:35 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1542259
repository: mozilla-central
start_time: 2015-05-22T08:55:26
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0002
buildname: WINNT 5.2 mozilla-central build
revision: b6a1b130eb1b

CVTRES : fatal error CVT1107: 'c:\builds\moz2_slave\m-cen-w32-00000000000000000000\build\src\obj-firefox\dom\events\EventStateManager.obj' is corrupt
LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
Return code: 1
'mach build' did not run successfully. Please check log for errors.
Running post_fatal callback...
Exiting -1
# TBPL FAILURE #
# TBPL FAILURE #
Comment 92 User image Treeherder Robot 2015-05-24 19:02:26 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g34_v2_1&job_id=122694
repository: mozilla-b2g34_v2_1
start_time: 2015-05-21T19:09:44
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0132
buildname: b2g_mozilla-b2g34_v2_1_win32_gecko-debug build
revision: 23203116eac3

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [mozglue.dll] Error 1123
mozmake.exe[4]: *** [mozglue/build/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 93 User image Treeherder Robot 2015-05-26 14:21:45 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g34_v2_1&job_id=124273
repository: mozilla-b2g34_v2_1
start_time: 2015-05-26T12:30:14
who: wkocher[at]mozilla[dot]com
machine: b-2008-ix-0078
buildname: b2g_mozilla-b2g34_v2_1_win32_gecko build
revision: cc064744135f

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[5]: *** [mozglue.dll] Error 1123
mozmake.exe[4]: *** [mozglue/build/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 94 User image Treeherder Robot 2015-05-27 13:40:40 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g34_v2_1&job_id=124454
repository: mozilla-b2g34_v2_1
start_time: 2015-05-27T12:47:15
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0168
buildname: WINNT 5.2 mozilla-b2g34_v2_1 leak test build
revision: 894528758073

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [config/external/icu/target] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 95 User image Treeherder Robot 2015-05-28 18:49:16 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-esr31&job_id=71582
repository: mozilla-esr31
start_time: 2015-05-28T18:08:42
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0078
buildname: WINNT 5.2 mozilla-esr31 leak test build
revision: e95949751c06

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [intl/icu/compile] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 96 User image Treeherder Robot 2015-06-02 14:34:20 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-b2g32_v2_0&job_id=135837
repository: mozilla-b2g32_v2_0
start_time: 2015-06-02T13:55:42
who: wkocher[at]mozilla[dot]com
machine: b-2008-ix-0168
buildname: WINNT 5.2 mozilla-b2g32_v2_0 leak test build
revision: 67c38d759719

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
mozmake.exe[6]: *** [all-recursive] Error 2
mozmake.exe[5]: *** [buildicu] Error 2
mozmake.exe[4]: *** [intl/icu/compile] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [realbuild] Error 2
mozmake.exe: *** [build] Error 2
Comment 97 User image Adam Smith 2017-01-25 03:25:28 PST Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.