Closed Bug 1009807 Opened 10 years ago Closed 10 years ago

Figure out the correct path setup and mozconfigs for automation-driven MSVC2013 builds

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: benjamin, Assigned: jlund)

References

Details

Attachments

(4 files)

140501_bug_1009807_vs2013.patch
9.59 KB, patch
Details | Diff | Splinter Review
140504_bug_1009807_vs2013.patch
13.12 KB, patch
Details | Diff | Splinter Review
140819_bug_1009807_vs2013.patch
13.26 KB, patch
Details | Diff | Splinter Review
Screenshot b-2008-ix-0028 2014-08-28 at 19.27.24.png
2.25 MB, image/png
Details 
This sorta got stuck while working up bug 914523. Recording the email thread here in a bug for tracking:

> What are the next steps required to move forward here? I'm happy to file
> bugs for these if necessary.
What I'd suggest:

* (dmajor) manually create Windows 'junctions' as needed to eliminate
the need for a) paths-with-spaces and b) 8.3 short names in our
mozconfigs and keep a list of those junctions
** (RelOps) add dmajor's list of junctions to Windows GPO for future imaging
* (RelEng) set up a staging environment on dev-master1
** modify staging copy of mozilla-central mozconfigs to point at Visual
Studio 2012 paths (using the junction names above where applicable)
** perform test builds using the new compiler and work with
dmajor/developers to resolve build failures.  May need to iterate for
awhile here.
** also perform test builds against Visual Studio 2010 to make sure that
configuration hasn't been broken by the side-by-side installation of VS2013

Keep in mind that what we've installed on a builder is a release
candidate (RC) version of VS2013 so we'll need to move to the final
release (RTM) version before deployment.

Once builds are looking good and tests are green, and we're on the final
version of Visual Studio, start executing the deployment.
* When we rolled out the rev2 w64-ix-slave machines, we did so over a
long period of time and had a split pool (rev1 and rev2) for all that
time.  I think we're all agreed that we'd like to avoid a drawn-out
split pool again if at all possible because of the extra maintenance
burden.  It can be avoided provided there is a high confidence the
VS2010 install has not been compromised (hence the VS2010 testing above)
in which case we're left with deciding which branches should be using
the VS2013 compiler.
* Roll out to branches something like this:
** a single project branch
** all project branches
** (mozilla|b2g)-central and try
** ride the trains for remaining branches (aurora, beta, release, esr)

Here's my proposal for the commands:

mklink /j c:\tools\vs2010 "C:\Program Files (x86)\Microsoft Visual Studio 10.0"
mklink /j c:\tools\vs2012 "C:\Program Files (x86)\Microsoft Visual Studio 11.0"
mklink /j c:\tools\vs2013 "C:\Program Files (x86)\Microsoft Visual Studio 12.0"

A lot of caveats on this.
- This assumes c:\tools exists (it does on ix-mn-w0864-001)
- This assumes a 64-bit OS because of "Program Files (x86)" -- could make this more generic if we needed to.
- I'm not an expert on the subtleties of junctions vs symlinks vs hard links. A while back I tried them all to redirect objdirs, and for reasons I don't remember, junctions were what worked best for me.
- Junctions need NTFS and Windows Vista or higher.
- If the target does not exist (e.g. Visual Studio 11.0) then the command will still succeed, but the resulting junction won't be very useful.
- Feel free to change any of the names/paths as you see fit! (Version number instead of version year? Somewhere other than tools?)

This bug does not track the other parallel work, which is that MSVC2013 builds currently don't run on Windows XP. I'm going to file that separately.

Ni? laura for an assignee.
Flags: needinfo?(laura)
Blocks: 914523
I would like to help ensure releng does its part on this and does not block.

jhopkins is no longer with releng. So, to help get me up to speed on win64 and VS2013 state, where are we today? comment 0's email dump is from ~1month ago.

dmajor:
- I see you still have a loan out for this. Have you had any luck completing a build using vs2013 on ix-mn-w0864-001?

If so, I think we can move forward.

going forward:
dmajor - aside from GPO/imaging changes (eg: junction links), how can we reproduce a VS2013 build?
-- Is there a custom mozconfig we can point at in our staging env?
-- are there any env vars we need to set?
-- anything else that differs from our current ways?

I have pointed ix-mn-w0864-001 to my dev-master1 machine so we can trigger some staging builds on it once we know the setup required. From there I we can iterate over some trial builds until we can reproduce successful ones under both VS2010 and VS2013.

dmajor, would you have time tomorrow (I work PT hours but I'm flexible) to hook up over vidyo? I think it would be great to knowledge transfer and figure out an implementation strategy. my zimbra cal is accurate if you would like to just create a meeting.
Flags: needinfo?(laura) → needinfo?(dmajor)
Assignee: nobody → jlund
So, I don't really know how this stuff works or what needs to be done. I had been hoping to watch the machine do a triggered build (as opposed to me starting one on the console) so that I could observe how things are set up, since it's different from my development machine. Then things stalled out.

We ought to be able to point the build at the new compiler (on my machine I would do that via start-shell-etc... but it looks like the builders do this by mozconfigs?) and things should "just work". If not then those are bugs.

There was an issue about mozconfigs in that they use 8.3 naming which gets really fragile when there may or may not be more than one compiler on the machine (C:\PROGRA~1\MICROS~... gets unpredictable) I proposed to address that by adding junctions with shorter paths, but then that thread got pretty sidetracked with XP considerations.

Let me know if you still want to sync up but I'm afraid I don't have much else to add. I'll go ahead and resolve that loaner bug.
Flags: needinfo?(dmajor)
OK thank you dmajor,

So it is still unclear to me if we have successfully been able to build against VS2013 on a machine? If not, I am not sure how much releng should be involved WRT to adding this into automation.

At any rate, I'll take a stab at trying to get a build going.

I am unfamiliar with make and our build system (let alone on windows), so I will spend some time today trying to rectify that. Up till this point, I have only worked on the automation around calling client.mk.

I spoke with mshal and he gave me some build config 101. I will play around with http://mxr.mozilla.org/mozilla-central/source/build/win64/mozconfig.vs2010#2 and see if I can make a 2013 equivalent to use.

I will also connect with relops and investigate these 'junction' possibilities and the problems with 8.3 naming on windows.

Please bare with me for a day or two as stumble my way around. If there are pros out there who can assist and help speed my progress, I'd love to connect with you.
What do you mean by "a machine"? We've certainly been building on our local machines using MSVC2013. So in general the builds work.
May I ask why we hard code paths? Why not prepend path environments and just call the executables by name in the configs?
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> What do you mean by "a machine"? We've certainly been building on our local
> machines using MSVC2013. So in general the builds work.

Sorry 'a machine' wasn't clear.

OK, so the building against MSVC2013 works locally on dev machines via 'start-shell-etc' but we have not done so using mozconfigs or on a 'releng build machine'

This helps, thanks for your patients.
Flags: needinfo?(q)
oops I didn't mean to put a needinfo in for Q as I reached him on irc. We are going to touch base on how to deal with paths/env for this.
Flags: needinfo?(q)
Hi Benjamin,

I've touched base with coop and armen. There are some ideas floating around and we are going to have a mtg to discuss implementation. rather than us making assumptions, would you have time to join today? Sometime early afternoon ET? If not, no problem; I can relay anything we discuss if you are curious.
Flags: needinfo?(benjamin)
Sorry, was on PTO Friday. Happy to help if needed.
Flags: needinfo?(benjamin)
==== status update

Monday was a holiday so sorry for delay.
This week I am in Ireland (GMT time) and will be PTO on thurs, fri, and mon.

I met with coop and armen and we came up with some ideas and questions. I touched base with bsmedberg on irc today and he is happy to meet tomorrow morning to discuss such questions. I will update this bug and put up some drafts after that mtg.
Depends on: 983085
No longer depends on: 983085
met with bsmedberg before starting PTO last week and discussed questions I tracked here: https://etherpad.mozilla.org/Svlujj5KOt

since then I believe I have been able to build ff with both vs2010 and vs2013 via automation.

For vs2013, I merely just played around with: http://hg.mozilla.org/mozilla-central/file/b5b35ec88db8/build/win32/mozconfig.vs2010-win64 and set it to use vs2013 with corrected vars for things like WIN32_REDIST_DIR.

The result was a success/green build. The log can be found here: http://people.mozilla.org/~jlund/vs2013_leak_win32_build.log

now that it seems we can build with vs2010 and vs2013 in automation, the next steps are as follows:

1) verify the above pasted log is correct and what we expect as a healthy working build
   a) bsmedberg or dmajor: can you confirm this: http://people.mozilla.org/~jlund/vs2013_leak_win32_build.log looks good: 
2) figure out if we are going to stick with junction links for distinguishing between visual studio paths (what I am currently using on my testing slave: IX-MN-W0864-001) or setting env path vars like mentioned in comment 5
   a) IIUC the former would mean we need to modify our GPO scripts. The latter we would need to add to our env dicts in buildbot-configs
   b) I will talk to Q about this
3) set up mozconfigs correctly. This should be done by either:
    a) creating a 'mozconfig.vs2013-win64' based off 'mozconfig.vs2010-win64' and getting places like: http://hg.mozilla.org/releases/mozilla-beta/file/4f28715f9db6/browser/config/mozconfigs/win32/common-opt#l22 and http://hg.mozilla.org/mozilla-central/file/b5b35ec88db8/browser/config/mozconfigs/win32/debug#l12 to point to it
    b)ripping out the small differences between vs2010 and vs2013 into smaller mozconfigs and creating a generic/common 'mozconfig.msvs-common-win64' mozconfig that they share.
This would make the following changes:

- creates mozconfig.vs2013-win64
** this would be based off 'mozconfig.vs2010-win64' with only a difference to 'WIN32_REDIST_DIR' and '_VSPATH' values.


- replaces every instance of mozconfig.vs2010-win64 with mozconfig.vs2013-win64
** I could have made a 'mozconfig.vs-base-win64' and ripped out the duplicate stuff but IIUC, we want to replace 2010 completely on our 'win64 machines ff32bit builds'. This patch allows us to simply swap out 'mozconfig.vs2013-win64' with 'mozconfig.vs2010-win64' while keeping other mozconfigs like 'win32/mozconfig.vs2010' (the non win64 machine mozconfig that we need to support for things like seamonkey)

This needs the following before rolling out:

1) vs2013 installed along side vs2010 across all our win64 build machines
2) a link junction added to our gpo:
    mklink /j c:\tools\vs2013 "C:\Program Files (x86)\Microsoft Visual Studio 12.0"

once I get an f+ to go along this route, I'll file those dep bugs.

again, see comment 11 for an example of a win64 ff32 build against vs2013. I think it was successful and what we want but it be nice to get confirmation from someone who is more familiar with our builds.
Attachment #8432254 - Flags: feedback?(benjamin)
Comment on attachment 8432254 [details] [diff] [review]
140501_bug_1009807_vs2013.patch

This patch appears to only change the win64 build config, but the intent here was to change the win32 build config.

Will there be a separate patch to copy build/win32/mozconfig.vs2010 and use a new build/win32/mozconfig.vs2013 for win32 builds?
Flags: needinfo?(jlund)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #13)
> Comment on attachment 8432254 [details] [diff] [review]
> 140501_bug_1009807_vs2013.patch
> 
> This patch appears to only change the win64 build config, but the intent
> here was to change the win32 build config.
> 

hmm, so I may be getting confused but IIUC there are a few things at play. 1. what arch the OS is and 2. what arch is the app (firefox) is.

so I thought we have:
- http://hg.mozilla.org/mozilla-central/file/b5b35ec88db8/build/win32/mozconfig.vs2010-win64 (win64 ff32)
- http://hg.mozilla.org/mozilla-central/file/b5b35ec88db8/build/win32/mozconfig.vs2010 (win32 ff32)
- http://hg.mozilla.org/mozilla-central/file/b5b35ec88db8/build/win64/mozconfig.vs2010 (win64 ff64)

all our build machines are win64 and we build ff32 on them. This is why I was changing places that referenced mozconfig.vs2010-win64 first as my interpretation was that we wanted initially change ff32 builds to use vs2013.

Apologies if I am getting wires crossed.
Flags: needinfo?(jlund)
ok, then I clearly don't know which mozconfigs do what ;-)

Do we have any win32 build machines at all any more, or can we remove that configuration completely?

Otherwise, the configs that we should be modifying are the win64-ff32 and win64-ff64 configs.
> Do we have any win32 build machines at all any more, or can we remove that
> configuration completely?

IIUC we need to keep win32-ff32 support: http://hg.mozilla.org/mozilla-central/file/b5b35ec88db8/browser/config/mozconfigs/win32/debug#l14

for things like seamonkey

> 
> Otherwise, the configs that we should be modifying are the win64-ff32 and
> win64-ff64 configs.

I've been told that win64-ff64 is not actively being worked on: https://tbpl.mozilla.org/?tree=Date

So since we are not running win64-ff64 vs2010 builds, I would suspect that getting vs2013 working would be outside the scope of this bug.
> I've been told that win64-ff64 is not actively being worked on:
> https://tbpl.mozilla.org/?tree=Date

I don't know what this means. We produce ff64 nightlies every day, and close to 50% of our nightly users are actually using them. If we're switching ff32 to the new compiler, we should switch ff64 at the same time.

e.g. http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-32.0a1.en-US.win64-x86_64.installer.exe
Depends on: 1019165
> I don't know what this means. We produce ff64 nightlies every day, and close
> to 50% of our nightly users are actually using them. If we're switching ff32
> to the new compiler, we should switch ff64 at the same time.

sorry, it was clarified in #build that while it is true devs are not actively working on win64-ff64, we are still building nightlies for it. I thought that the win64-ff64 effort was on the date branch and since we had not built anything there since april 15th, I wrongfully thought we were not building it in automation.

I will modify the ff64 compiler related mozconfigs to use vs2013 as well. I'll update what my patch after I have a ff64 vs2013 and ff64 vs2010 successful compile.

I think it is fair to say at this stage that we should install vs2013 across all our window64 machines and create the junction link for /c/tools/vs2013 as mentioned in comment 12. I filed Bug 1019165 to track that effort.
Attachment #8432254 - Flags: feedback?(benjamin)
=== update

I have been able to compile what I think is a win64-ff64 vs2013 build as you can see in this debug build log: http://people.mozilla.org/~jlund/vs2013-win64-ff64.log. However there are some unknowns

1) this patch and log do seem to make use of vs2013 however I still see a couple log lines like 

'C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -D_WIN32_WINNT=0x0601 -D_AVAIL_WINVER_=0x0601 -Ic:\mozilla-build\python27\include -Ic:\builds\moz2_slave\m-cen-w64-d-000000000000000000\build\obj-firefox\_virtualenv\PC /Tcpsutil/_psutil_common.c /Fobuild\temp.win32-2.7\Release\psutil/_psutil_common.obj'

I am not sure where that is coming from 

2) This patch removes the 32bit linker override (http://hg.mozilla.org/mozilla-central/file/882826199076/build/win64/mozconfig.vs2010#l29) altogether as I thought this might have been restricted to 2010 and we would not need to do the equivalent for vs2013. Though I am also not seeing any builders in automation that do win64-ff64 PGO builds so I can't verify if this was the right thing to do or not...

I am doing a win64-ff64 nightly build as we speak to see if it works like the debug build I linked above.
hmm, so something doesn't seem right.

both win64-ff64 and win64-ff32 nightly builds fail to compile: weirdly they also fail the rm-old-packages step:
rm old pckage step: http://people.mozilla.org/~jlund/vs2013_win64-ff64_nightly_failed_old_pckge.log
compile: http://people.mozilla.org/~jlund/vs2013_win64_nightly.log (win64-ff64) but ff32 is the same

I think this issue and the 2 others specified on comment 19 are the only blockers for moving forward.

Feedback welcome. :)
The compile error is:

c:\builds\moz2_slave\m-cen-w64-ntly-000000000000000\build\memory\jemalloc\src\include\msvc_compat\stdbool.h(8) : error C2628: 'BOOL' followed by 'bool' is illegal (did you forget a ';'?)

Which I believe dmajor fixed on trunk builds recently.
Flags: needinfo?(dmajor)
Oh, and other than http://stackoverflow.com/questions/2961673/find-missing-argument-to-exec I don't really have insight into the first error: it looks like it's probably some sort of escaping issue.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #21)
> The compile error is:
> 
> c:\builds\moz2_slave\m-cen-w64-ntly-
> 000000000000000\build\memory\jemalloc\src\include\msvc_compat\stdbool.h(8) :
> error C2628: 'BOOL' followed by 'bool' is illegal (did you forget a ';'?)
> 
> Which I believe dmajor fixed on trunk builds recently.

WRT the above C2628 error, I have pulled and merged from m-c tip. Trying builds again.

WRT rm old symbols error, I didn't think it was a syntax issue as I assumed it was working in production. Turns out this step fails there too. /me files: https://bugzilla.mozilla.org/show_bug.cgi?id=1021873

WRT 'C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe' log lines, I believe this is only in the python setup and psutil stuff and we actually want it to use mvs 10. mshal found this: http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/virtualenv.py#326

still not sure if I should be removing the linker override: http://hg.mozilla.org/mozilla-central/file/882826199076/build/win64/mozconfig.vs2010#l29 but haven't found any issues with doing so.
OK nightlies are compiling.

Once bug 1019165 is resolved, I think we are good to land this[1] on a disposable branch, trigger some full builds on it with lots of eyes on it, and then land on all m-c based branches.

Unless I'm missing something, things are done from my end until 1019165 is resolved

[1] https://bugzilla.mozilla.org/attachment.cgi?id=8434728
> error C2628: 'BOOL' followed by 'bool' is illegal (did you forget a ';'?)
The fix came from glandium in bug 945582.
Flags: needinfo?(dmajor)
(In reply to Jordan Lund (:jlund) from comment #23)
> still not sure if I should be removing the linker override:
> http://hg.mozilla.org/mozilla-central/file/882826199076/build/win64/
> mozconfig.vs2010#l29 but haven't found any issues with doing so.

Yeah, that override was to work around a VS2010 bug. 2013 shouldn't need it.
Depends on: 1027745
There's actually a problem with the mozconfigs here:

+## paths: win8 sdk x86 (32-bit) tools, msvc (32-bit) build toolchain, moz tools  ##
+export PATH="/c/Program Files (x86)/Windows Kits/8.0/bin/x86:${_VSPATH}/Common7/IDE:${_VSPATH}/VC/BIN:${_VSPATH}/Common7/Tools:${_VSPATH}/VC/VCPackages:/c/mozilla-build/moztools:${PATH}"

This gives us the 32/32 VC toolchain. What we want is the 64/32 toolchain.

The correct PATH should probably be:

export PATH="/c/Program Files (x86)/Windows Kits/8.0/bin/x86:${_VSPATH}/Common7/IDE:${_VSPATH}/VC/BIN/amd64_x86:${_VSPATH}/Common7/Tools:${_VSPATH}/VC/VCPackages:/c/mozilla-build/moztools:${PATH}"

I filed an equivalent for the mozillabuild start script, bug 1027661
=== status update.

I have not looked at this recently as my focus has been on the two Q2 goals I'm attached to: loan tool[1] and mozharness builds[2]

I will be picking up vs2013 on monday.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=965691
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=858797
=== status update

I still have follow up work to do that sits higher on priority. I hope to have a day next week to look at this.
I had some time to look at this today.

Now that I have a slave(Bug 1027745) that has our production vs2010 and vs2013 installed, I could reproduce the error we saw in prod(Bug 1026870). It looks like cvstres.exe is the culprit. I was able to green the build if I copied the 2013 equivalent to the 2010:
$ cp C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cvtres.exe C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\cvtres.exe

Q: how easy would it be to to replace the 2010 cvtres with 2013's during GPO/imaging? I don't see why we would need the 2010 one so I think it be a waste trying to juggle PATHs. I am going to fire a bunch of builds on my loaner tonight to see if there are any other issues.
Flags: needinfo?(q)
all results below are run on a slave with vs2010 and vs2013 installed simultaneously

good news:

- win64-ff32 vs2010: it seems that replacing cvtres.exe solves builds. I ran a number of builds and they all came back green.
- win64-ff32 vs2013: patch https://bugzilla.mozilla.org/attachment.cgi?id=8434728 here still works and builds are successful

bad news:

- win64-ff64 vs2010: builds are broken. compile for about ~7min before crashing[1]
- win64-ff64 vs2013: untested.

I'll keep investigating win64-ff64 errors. I don't think we can not install vs2013 on our win machines if it breaks vs2010 builds. That is unless we can just switch our win64-ff64 builds over to vs2013 immediately and vs2013 works

[1]: full log - http://people.mozilla.org/~jlund/vs2013-ff64-win64_fail_with_vs2010.log
     snippet: billions of these conversion errors: "warning C4267: '=' : conversion from 'size_t' to 'uint32_t', possible loss of data"
The actual error is LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
Makefile:205: recipe for target '../lib/icuuc52.dll' failed

dmajor have you seen that before?
Flags: needinfo?(dmajor)
Oh, interesting. That sounds very similar to the win64-ff32 issue we had: https://bugzilla.mozilla.org/show_bug.cgi?id=1026870#c0 that I fixed via comment 30. I guess that fix didn't account for win64-ff64 but this smells like another fallout of havind vs2010 without SP1
Interesting, but maybe win64-ff64-vs2010 isn't an interesting target. Since we're doing nightly builds only, we could maybe switch them over to win64-ff64-vs2013 "immediately".
(In reply to Jordan Lund (:jlund) from comment #33)
> I guess that fix didn't account for win64-ff64 but this smells
> like another fallout of havind vs2010 without SP1

Well that ought to be simple enough to fix. Try doing the same replacement for the copy of cvtres.exe under VC\bin\amd64?
Flags: needinfo?(dmajor)
yes, I copied the amd64 version too and a win64-ff64 vs2010 nightly built successfully. Just running a debug and opt build equivalent now to be sure.

Once all those are done, I believe we are good to pull the trigger again on installing vs2013.

I am on PTO tomorrow but I should be able to wrap this up at the beginning of next week.
\o/
I am on buildduty so work has to be sporatic on this but I think I am still on-schedule.

as far as remaining win64-ff64 vs2010 builders, debug and opt builds ran fine too. I also ran a win64-ff64 build against my vs2013 patch. Looks like that works too so that I believe covers every variation.

I think we are ready to pull the trigger on trying another rollout of vs2013. I'll open the relops bug (bug 1019165) for that now.

Once that lands, we can apply my patch in the try server and get some eyes on vs2013.
> I think we are ready to pull the trigger on trying another rollout of
> vs2013. I'll open the relops bug (bug 1019165) for that now.
> 
> Once that lands, we can apply my patch in the try server and get some eyes
> on vs2013.

this has been resolved. we are good to apply my mozconfig patch on try and test out vs2013. uploading patch now
Flags: needinfo?(q)
hey ben, this should force our slaves to use vs2013 on any 32 or 64 win build variant. We can try this on try now.
Flags: needinfo?(benjamin)
Pushed https://tbpl.mozilla.org/?tree=Try&rev=93e879f54ae5 to try. Assuming it's not red, we should close this out and dmajor can check for test failures.
Flags: needinfo?(benjamin) → needinfo?(dmajor)
I confirmed by dumpbin that all the successful builds have linker version 12.00.

XP tests are completely orange but that's bug 1001332.
Mochi-1 is busted across the board and I suspect it's the same root cause as bug 1018402.
Win64 debug is fine, or at least as good as you can expect, that orange is always there.

Win64 opt is burning with: cl: command not found
In the opt build log, I never see any mention of VS120COMNTOOLS in the environment variables. Is the new VS only on a subset of the builders or something? As speculation I'll trigger a rebuild.
Flags: needinfo?(dmajor)
Comment on attachment 8475370 [details] [diff] [review]
140819_bug_1009807_vs2013.patch

Review of attachment 8475370 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/mozconfigs/win32/common-opt
@@ +25,2 @@
>  else
>    . $topsrcdir/build/win32/mozconfig.vs2010

It's odd that everything works without this 2010 being changed. (In other files too)
(In reply to David Major [:dmajor] from comment #43)
> Comment on attachment 8475370 [details] [diff] [review]
> 140819_bug_1009807_vs2013.patch
> 
> Review of attachment 8475370 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: browser/config/mozconfigs/win32/common-opt
> @@ +25,2 @@
> >  else
> >    . $topsrcdir/build/win32/mozconfig.vs2010
> 
> It's odd that everything works without this 2010 being changed. (In other
> files too)

IIUC - $topsrcdir/build/win32/mozconfig.vs2010 is for win32-ff32 builds. all our firefox builds use win64 now whether it's win64-ff32 or win64-ff64. that's why in this case:

win64-ff32 -> we use http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/win32/common-opt#24

instead of this else:
> >  else
> >    . $topsrcdir/build/win32/mozconfig.vs2010   # maybe used by seamonkey?
> Win64 opt is burning with: cl: command not found
> In the opt build log, I never see any mention of VS120COMNTOOLS in the
> environment variables. Is the new VS only on a subset of the builders or
> something? As speculation I'll trigger a rebuild.

The rebuild succeeded. If it means anything, the successful builds were on -ix- machines, and the failed one was a -sm-.
(In reply to David Major [:dmajor] from comment #45)
> > Win64 opt is burning with: cl: command not found
> > In the opt build log, I never see any mention of VS120COMNTOOLS in the
> > environment variables. Is the new VS only on a subset of the builders or
> > something? As speculation I'll trigger a rebuild.
> 
> The rebuild succeeded. If it means anything, the successful builds were on
> -ix- machines, and the failed one was a -sm-.

interesting, sm is seamicro. I wonder if Bug 1047621 has anything to do with that.

mshal: is this something different to what you've been seeing in 1047621?
Flags: needinfo?(mshal)
dmajor: are we in a state where we can resolve this bug as per comment 41?
Flags: needinfo?(dmajor)
Let's continue to follow up on the sm oddity but otherwise this looks good to me. Thanks for all the work on this!
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(dmajor)
Resolution: --- → FIXED
(In reply to Jordan Lund (:jlund) from comment #46)
> (In reply to David Major [:dmajor] from comment #45)
> > The rebuild succeeded. If it means anything, the successful builds were on
> > -ix- machines, and the failed one was a -sm-.
> 
> interesting, sm is seamicro. I wonder if Bug 1047621 has anything to do with
> that.
> 
> mshal: is this something different to what you've been seeing in 1047621?

I don't think that a missing cl would be related to 1047621. At least, there's nothing specific to sm in build config either before or after that bug. When was the failed build executed? According to [1], the sm machines were updated very recently. Maybe it just hadn't been imaged yet? If you build on an sm machine now, does it still fail to find cl?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1019165#c77
Flags: needinfo?(mshal)
> I don't think that a missing cl would be related to 1047621. At least,
> there's nothing specific to sm in build config either before or after that
> bug. When was the failed build executed? According to [1], the sm machines
> were updated very recently. Maybe it just hadn't been imaged yet? If you
> build on an sm machine now, does it still fail to find cl?
> 
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1019165#c77

The failed build was about 7 hours after [1]. How can I request a rebuild on a specific class of machine?
Do the builders use the 'amd64_x86' compilers for 32-bit builds? On developer builds this is set up by start-shell-msvc2013.bat, but I don't know if things are different on the build machines.

$ which cl
/c/Program Files (x86)/Microsoft Visual Studio 12.0/VC/BIN/amd64_x86/cl.exe
Flags: needinfo?(jlund)
(In reply to David Major [:dmajor] from comment #51)
> Do the builders use the 'amd64_x86' compilers for 32-bit builds? On
> developer builds this is set up by start-shell-msvc2013.bat, but I don't
> know if things are different on the build machines.
> 
> $ which cl
> /c/Program Files (x86)/Microsoft Visual Studio 12.0/VC/BIN/amd64_x86/cl.exe

I think ben discovered we were doing things wrong here: https://bugzilla.mozilla.org/show_bug.cgi?id=1009807#c27. my patch I submitted here does not include the fix for that. but we should be able to do as ben suggested by changing the equivalent of http://mxr.mozilla.org/mozilla-central/source/build/win32/mozconfig.vs2010-win64#23 but for build/win32/mozconfig.vs2013-win64 in my patch

initially when I saw that comment 27 I thought he was speaking of vs2013, but I guess this is a bug for vs2010 too. If so, maybe we should file a sep bug and fix right away?
Flags: needinfo?(jlund)
VS2010 is unaffected. The amd64_x86 toolchain is new to VS2013.

I'll try the suggested fix. Without it my PGO try pushes are burning.
(In reply to David Major [:dmajor] from comment #53)
> VS2010 is unaffected. The amd64_x86 toolchain is new to VS2013.
> 
> I'll try the suggested fix. Without it my PGO try pushes are burning.

great. /me crosses fingers

BTW - I am on PTO until wed 27th. please reach coop if you have any pressing issues.
Blocks: 1057229
dmajor: did the suggested fix work? outside of ff64, how far are we away from being able to avail of vs2013 on ff32?
Flags: needinfo?(dmajor)
The amd64_x86 fix removed one layer of problem, but then I ran into other PGO issues (bug 1057229). And most recently, I get timeouts after 3 hours with or without PGO: https://tbpl.mozilla.org/?tree=Try&rev=3204fb4f342f https://tbpl.mozilla.org/?tree=Try&rev=9c5361816db2

Any idea what's going on there? At this point it's hard to gauge how close we are since I can't tell what's hiding behind this issue.
Flags: needinfo?(jlund)
Hm, I thought I cleared this.
Flags: needinfo?(dmajor)
(In reply to David Major [:dmajor] from comment #56)
> The amd64_x86 fix removed one layer of problem, but then I ran into other
> PGO issues (bug 1057229). And most recently, I get timeouts after 3 hours
> with or without PGO: https://tbpl.mozilla.org/?tree=Try&rev=3204fb4f342f
> https://tbpl.mozilla.org/?tree=Try&rev=9c5361816db2
> 
> Any idea what's going on there? At this point it's hard to gauge how close
> we are since I can't tell what's hiding behind this issue.

are these timeouts every time? I did not hit a timeout in staging against vs2013. I should point out that we are hitting timeouts on vs2010 occasionally[1]. Something odd is happening.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1055876#c19
Flags: needinfo?(jlund)
Yes, every time, at least in the past few days. bsmedberg's first try push on the 19th was OK. I think that's the only successful one we've had. https://tbpl.mozilla.org/?tree=Try&rev=93e879f54ae5
Are you able to login to the machine with rdesktop/vnc while the build is going on? There might be an error dialog that pops up with some useful info that is preventing the build from continuing, eventually causing it to timeout.
(In reply to Michael Shal [:mshal] (no reviews 8/29 - 9/8) from comment #60)
> Are you able to login to the machine with rdesktop/vnc while the build is
> going on? There might be an error dialog that pops up with some useful info
> that is preventing the build from continuing, eventually causing it to
> timeout.

we triggered another try build that has been timing out. Then we vnc'd on the machine and discovered a dialog. dmajor thinks this was a result of the changes made in comment 53.

we also re-triggered a known good push: https://tbpl.mozilla.org/?tree=Try&rev=93e879f54ae5 thus far that push is not hung and is currently compiling as it should be.
dialog showing what is currently hanging builds
Thank you for the screenshot! The culprit was my change to use the amd64_x86 tools. Apparently it's not a complete toolset and you need the native-amd64 tools as fallback on the PATH. That's consistent with what vcvarsamd64_x86.bat does for local developer builds:

$ where cl
c:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64_x86\cl.exe
c:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\cl.exe

$ where mspdb120.dll
c:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\mspdb120.dll

With that fixed, the cl check works again: https://tbpl.mozilla.org/?tree=Try&rev=a4b429685d7b (I deliberately pushed a broken build to get faster results)
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: