Figure out if there's a way to get VC11 binaries running on XP

RESOLVED FIXED

Status

()

Firefox
General
P1
normal
RESOLVED FIXED
6 years ago
2 years ago

People

(Reporter: jimm, Unassigned)

Tracking

Trunk
x86_64
Windows XP
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(9 attachments, 1 obsolete attachment)

(Reporter)

Description

6 years ago
VC11 builds don't execute on all flavors of XP. 

https://connect.microsoft.com/VisualStudio/feedback/details/690617

Working around this can be done, but the fixes are pretty ugly, for example:

http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/

Before we start doing Win8 Metro browser builds, we'll need to solve this or split final builds up based on os version which isn't optimal from a rel eng perspective.
(Reporter)

Comment 1

6 years ago
Some notes – so the main problem here is that VC11 binaries are incompatible on operating systems lower than Vista. This is something new for MS to do, they usually keep support around until the os falls off the extended support calendar, but for whatever reason they chose to change that policy for VC11. 

The reason why binaries are incompatible is due to newer exports in system libraries that XP's system doesn't support. There are about 25 of these entry points. As a result binaries (both exe and dll) created by VC11 are marked as incompatible with XP and as such won't load on that os. We have multiple dlls we build, all of which pull in various missing exports. Xul.dll actually isn't that bad, it only pulls in one, but other dlls leverage far more, for example the sqlite lib. The number of dlls we have is around 30. 

One approach to solving this is detailed here - 

http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/

It involves linking all binaries statically to the runtime (the runtime dlls pull in these exports and as such won't load) and adding replacement functions in our code that fill the void. Providing the new functions isn't a big deal, but the static linking I believe is a no go. (cc'ing bsmedberg here for his opinion) The total increase in install size depends on the amount of use the crt gets in each dll but could be rather large.

I started looking at splitting widget code out from xul lib with the idea of statically linking VC11 compiled parts of our runtime to the crt and loading those parts dynamically. From some experimentation it look as though splitting the entire widget lib out isn’t viable. It is a dependent lib for various other libraries, is built way before xul.lib is generated (which it would depend on) and generally can’t be shimmed easily due to the number of exports it provides and the number of imports it pulls in from other areas.

So I looked at separating out just the winrt code. This looks to be is a little bit more manageable. The total number of imports currently stands at around 150 (attached). These are from various areas - base widget, various winapi classes in widget/windows that we extend, xpcomglue, xpcom, gfx, and toolkit. I think though with a widget shim compiled into xul.lib that loads a winrt dll and forwards calls, plus some static linking to things like xpcomglue, and some additional exports in xul.dll for things like gfx, plus changing up the build order so all these lib stubs are available when the winrt code is linked, we might be able to get this to work. 

This is still a sucky solution for one main reason – it requires two sets of build tools to build the browser, and our build environment is not designed to build using two different sets of tools. I’m not sure how we would get this to work. Rel eng would also have to deal with getting the tools installed. Normal devs would also have to have both sets of tools to do metro builds. 

There are some other more general downsides – for example without VC11 we can’t use any new C++ 11 features in our code base. These types of issues are more trivial.

If we don’t go with the static linking or winrt splitting option, other solutions include building two different releases, or maybe moving XP users over to ESR builds. Maybe there are other ideas I haven’t come up with. cc’ing a few people who might have them.
(Reporter)

Comment 2

6 years ago
Created attachment 615832 [details]
missing exports
So the problem is the actual generated code, not the CRT?
(Reporter)

Comment 4

6 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #3)
> So the problem is the actual generated code, not the CRT?

Generated code should be ok. Problem summary:

1) new system imports XP doesn't have that get baked in by crt code / redist crt dlls that won't load on XP (both due to imports and os/subsystem versioning)

2) winrt libraries which leverage compiler features only available in VC11

There's a possible work around to #2 that's sucky from a developer perspective, we can try to code using low level abi COM interfaces that are undocumented. We'd like to avoid that since it'd pretty much kill dev productivity.
What non-XP system-library APIs do our libraries pull in, and why?

I wonder if we could build find a way to patch the redistributable CRT DLLs to use a stub library for the missing system-library APIs.
(Reporter)

Comment 6

6 years ago
Created attachment 616076 [details]
system apis
(Reporter)

Comment 7

6 years ago
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #5)
> What non-XP system-library APIs do our libraries pull in, and why?
> 
> I wonder if we could build find a way to patch the redistributable CRT DLLs
> to use a stub library for the missing system-library APIs.

Interesting idea, I'll investigate. From reading redist.txt not sure if modifying the crt dlls is acceptable.

"For your convenience, we have provided the following folders for use when redistributing VC++ runtime files. Subject to the license terms for the software, you may redistribute the folder (unmodified) in the application local folder as a sub-folder with no change to the folder name. You may also redistribute all the files (*.dll and *.manifest) within a folder, listed below the folder for your convenience, as an entire set."
(Reporter)

Comment 8

6 years ago
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #5)
> What non-XP system-library APIs do our libraries pull in, and why?

The crt src pulls these new apis in. If you have vc11 installed you can do searches across the crt source (VC\crt\src) for a specific api to find where they are used.
Another option would be to distribute the patching tool along with unmodified libraries and run it on install, only for XP users.
(In reply to Jim Mathies [:jimm] from comment #6)
> Created attachment 616076 [details]
> system apis

I'm still curious which of these APIs get called by xul.dll, sqlite3.dll etc and why. But if we develop a tool that works on the CRT libs, I guess it could work on ours just as well.
(Reporter)

Comment 11

6 years ago
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #10)
> (In reply to Jim Mathies [:jimm] from comment #6)
> > Created attachment 616076 [details]
> > system apis
> 
> I'm still curious which of these APIs get called by xul.dll, sqlite3.dll etc
> and why. But if we develop a tool that works on the CRT libs, I guess it
> could work on ours just as well.

I haven't put together a complete breakdown, but I know xul only pulls in one - GetTickCount64. Other libs like sqlite pull in a number of routines.
(Reporter)

Comment 12

6 years ago
Created attachment 616607 [details]
typical dependency graph

It looks like standard library functions have a habit of pulling these routines in from kernel32.dll.

Comment 13

6 years ago
Writing a tool to redirect those symbols is rather easy.  But the functions that I see there are not trivial to implement, and we can't quite get away with dummy no-op impls.

I've looked into this issue extensively when I was trying to find a way to run the VC10 generated binaries on Win2K.  All of the available solutions suck, some more than others.  I don't think we can get away with not using two compilers.  Why is that difficult for releng?
http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/ is all about reimplementing those functions to work on WinXP. That blogger supposedly has code for them, but I haven't looked at it myself.
(Reporter)

Comment 15

6 years ago
I'm curious about where the dlls come from on XP. When you checkout the src from mc, the runtime dlls don't come down with the src. I'm betting this is due to licensing. For people building on Vista and up with VC11 the runtime will be present and won't require any changes. But building on XP will require modifications.

How concerned are we about people building on XP? Maybe we just tell them they have to round up these dlls and run the modification tool manually? Note the dlls will also need to be reversioned so that they can load. (All of our dlls will also need to be reversioned but we can do that during the build process.)

Comment 16

6 years ago
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) (offline April 15-18) from comment #14)
> http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-
> statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/ is all
> about reimplementing those functions to work on WinXP. That blogger
> supposedly has code for them, but I haven't looked at it myself.

Yeah, I took a look at that.  Some of those implementations are heavily broken depending on what the crt does (for example, if the crt uses fibers, the implementation of FlsAlloc and friends can seriously break things.

Comment 17

6 years ago
(In reply to Jim Mathies [:jimm] from comment #15)
> I'm curious about where the dlls come from on XP. When you checkout the src
> from mc, the runtime dlls don't come down with the src. I'm betting this is
> due to licensing. For people building on Vista and up with VC11 the runtime
> will be present and won't require any changes. But building on XP will
> require modifications.
> 
> How concerned are we about people building on XP? Maybe we just tell them
> they have to round up these dlls and run the modification tool manually?
> Note the dlls will also need to be reversioned so that they can load. (All
> of our dlls will also need to be reversioned but we can do that during the
> build process.)

I'm confused now.  Is this a problem of building *on* Windows XP or *for* Windows XP?  I was assuming the former.  We can always change our build requirements to require Vista and 7, I don't think building on XP should hold us back at all.
(Reporter)

Comment 18

6 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #17)
> (In reply to Jim Mathies [:jimm] from comment #15)
> > I'm curious about where the dlls come from on XP. When you checkout the src
> > from mc, the runtime dlls don't come down with the src. I'm betting this is
> > due to licensing. For people building on Vista and up with VC11 the runtime
> > will be present and won't require any changes. But building on XP will
> > require modifications.
> > 
> > How concerned are we about people building on XP? Maybe we just tell them
> > they have to round up these dlls and run the modification tool manually?
> > Note the dlls will also need to be reversioned so that they can load. (All
> > of our dlls will also need to be reversioned but we can do that during the
> > build process.)
> 
> I'm confused now.  Is this a problem of building *on* Windows XP or *for*
> Windows XP?  I was assuming the former.  We can always change our build
> requirements to require Vista and 7, I don't think building on XP should
> hold us back at all.

I was asking about specific issues with building on XP, and you answered my question. We can just define our build requirements to be Vista+. Might anger a few folks, but such is life. Getting VC11 builds running on XP is the main focus of this bug.
It's certainly fine to require Vista or newer for building for Win8 with VC11. The only tricky bit might be the build slaves, which IIRC are all still Win2k3.

Comment 20

6 years ago
(In reply to Ted Mielczarek [:ted] from comment #19)
> It's certainly fine to require Vista or newer for building for Win8 with
> VC11. The only tricky bit might be the build slaves, which IIRC are all
> still Win2k3.

Yes, we just need to make sure that builds work fine on Win2K3.  You should probably ask RelEng to give you access to one of those machines if you want to verify this for a fact.

If that turns out to be a non-issue, we should WFM this bug.

This all being said, I'm still 10% puzzled, since the summary of this bug is talking about whether we can run VC11 binaries on XP...
(In reply to Ehsan Akhgari from comment #17)
> We can always change our build requirements to require Vista
Vista? You'll be lucky. All the previews require at least Windows 7.
(Reporter)

Comment 22

6 years ago
(In reply to Ted Mielczarek [:ted] from comment #19)
> It's certainly fine to require Vista or newer for building for Win8 with
> VC11. The only tricky bit might be the build slaves, which IIRC are all
> still Win2k3.

VC11 build tools run on Vista and up, so any system we build on needs to be win2008 server and up. We can't build on anything older than that. I've confirmed win2008 server works ok in bug 746043. Any build machine that has an older os on it will need to be upgraded so we can install VC11 on it.

(In reply to Ehsan Akhgari [:ehsan] from comment #20)
> (In reply to Ted Mielczarek [:ted] from comment #19)
> > It's certainly fine to require Vista or newer for building for Win8 with
> > VC11. The only tricky bit might be the build slaves, which IIRC are all
> > still Win2k3.
> 
> Yes, we just need to make sure that builds work fine on Win2K3.

The won't without the work here. Builds with VC11 are compatible with operating systems Vista+ - which means if you build an app with VC11 and try to run it on XP, it'll fail.

> This all being said, I'm still 10% puzzled, since the summary of this bug is
> talking about whether we can run VC11 binaries on XP...

Right, we can't - that's the problem. :)
(Reporter)

Comment 23

6 years ago
Maybe there is some confusion - we are going to integrate metro code bits into the Firefox build/installer on mc. So basically when you build or download firefox you'll also be building/downloading the metro bits, even if you are not on win8. To do this, all firefox builds will have to happen on VC11 with the win8 sdk.

Here's the details of the plan:
https://wiki.mozilla.org/Firefox/Windows_8_Integration#A_tentative_proposal

Comment 24

6 years ago
OK, I chatted about this with Jim a bit, and he asked me to summarize my thoughts here.

* Linking statically to the crt is not a good option, for code size/memory usage reasons.  And even if we decide that we don't care about any of that, we have no evidence to suggest that we can effectively simulate the missing crt import entries with our hacky implementations on WinXP.
* Fixing our binaries up with dynamic linking at runtime is impossible -- I've already experimented with this.  There's just too many details in the Windows loader over which we have zero control.
* Fixing our binaries at build time with a tool _could_ be an option, but we need to figure out whether we're allowed to binary patch Microsoft crt DLLs legally.  And we still have to answer the same question as the first point above.
* We could potentially use two compilers in the build process, and avoid building _anything_ which needs to be loaded on XP with VC11.  I don't know much about the winrt code, so I don't know how practical the surgery is.  If that is not very hard, using two compilers might be the way to go.
* Failing that, the _only_ remaining option that I can think of is having two completely separate builds, one using VC10 for Win7 and lower, and one using VC11 for Win8 only.  We obviously don't want to bundle both into the same installer because of download size, so we're going to need a stub installer (unless the product team says that doubling our Windows download size is OK.)

FWIW, I think that moving XP users to the ESR branch is the wrong thing to do.  We just have too many users on that platform to deprive them of the latest and greatest features on the web.
How different would Windows 8 builds be compared to pre-8 builds? I'm wondering that perhaps only xul.dll would need to be replaced, although that depends on it not needing to share CRT state with the other DLLs (jemalloc helps here).
(Reporter)

Comment 26

6 years ago
(In reply to neil@parkwaycc.co.uk from comment #25)
> How different would Windows 8 builds be compared to pre-8 builds? I'm
> wondering that perhaps only xul.dll would need to be replaced, although that
> depends on it not needing to share CRT state with the other DLLs (jemalloc
> helps here).

The differences are in xul.dll and most of those differences (thus far) are in toolkit/xre and widget. There will also be two browser front ends for each environment. The xre code can dynamically switch, there's nothing compiler dependent there. Win8 widget on the other hand depends on the VC11 compiler. 

Originally I was looking at splitting widget out, or maybe just the win8 widget code and shimming widget to call one or the other. I'm not sure how well VC10 dlls interact with VC11 dlls but we could test that to see.
> we have no evidence to suggest that we can effectively simulate the missing crt
> import entries with our hacky implementations on WinXP.

Right, but we can try it and run tests on XP and see what happens. If that works out we can distribute builds for people to try. We could also instrument the functions that known-broken and see if they get called at all, including in the wild via telemetry. (If they do get called, we can try fixing those implementations.)

I still kinda like the patch-with-tool option the best --- if it works. I wouldn't patch at build time, instead I'd patch at install time so the patching only happens on XP.

Comment 28

6 years ago
(In reply to Jim Mathies [:jimm] from comment #26)
> Originally I was looking at splitting widget out, or maybe just the win8
> widget code and shimming widget to call one or the other. I'm not sure how
> well VC10 dlls interact with VC11 dlls but we could test that to see.

This should work fine.  In general, multiple DLLs can link to different CRTs unless they do stupid things (such as passing a malloc'ed pointer to free with a different CRT, for example.)  We already rely on this.  Many of the third party DLLs which we load link against different CRTs.

Comment 29

6 years ago
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) (offline April 22-25) from comment #27)
> > we have no evidence to suggest that we can effectively simulate the missing crt
> > import entries with our hacky implementations on WinXP.
> 
> Right, but we can try it and run tests on XP and see what happens. If that
> works out we can distribute builds for people to try. We could also
> instrument the functions that known-broken and see if they get called at
> all, including in the wild via telemetry. (If they do get called, we can try
> fixing those implementations.)

We can (and should) do more than just try, we should read the CRT code for usages of those APIs and figure out exactly what they're using these APIs for.  (I'm assuming that VC11 still ships the CRT source code.)

And while that is all doable, there's still the risk that ultimately we conclude that we're not able to provide a good enough replacement for these APIs on XP.  So we might take this into account when weighing different options.

> I still kinda like the patch-with-tool option the best --- if it works. I
> wouldn't patch at build time, instead I'd patch at install time so the
> patching only happens on XP.

The shim layer should still check to see if these APIs are available in kernel32 etc and forward the calls to those real functions, if they are.  So I don't think patching on non-XP platforms is going to be a huge deal (but we may not be able to do that if we're not allowed to redistribute patched binaries, which I don't think we are.)  Doing this on the client side is a little bit tricky since we need to make sure that the patching happens as part of the updates as well, but it's something which we should be able to get right given enough care.

But I'm still curious to know why you like this option the best.  It seems like it's not the lowest risk thing we can think of.
Another option would be to check why our dlls need to pull these new symbols, and avoid that if possible. Our code obviously doesn't change, it must be system defines or inlines. We could try to divert these.
(In reply to Ehsan Akhgari [:ehsan] from comment #29)
> But I'm still curious to know why you like this option the best.  It seems
> like it's not the lowest risk thing we can think of.

It seems like more of a one-time cost. Once we have it working, it won't bother us. Breaking out the Win8 widget code and using two compilers for all Windows builds (until we drop Win XP support, which is probably years away) seems like it will add significant ongoing development costs. But I could be wrong about that.
(Reporter)

Comment 32

6 years ago
(In reply to Mike Hommey [:glandium] from comment #30)
> Another option would be to check why our dlls need to pull these new
> symbols, and avoid that if possible. Our code obviously doesn't change, it
> must be system defines or inlines. We could try to divert these.

The crt dll imports everything but our code really didn't pull anything in directly except for GetTickCount64, which every one of our dlls imports. 

http://msdn.microsoft.com/en-us/library/windows/desktop/ms724411(v=vs.85).aspx
(In reply to Jim Mathies [:jimm] from comment #32)
> The crt dll imports everything but our code really didn't pull anything in
> directly except for GetTickCount64, which every one of our dlls imports. 
> 
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms724411(v=vs.85).
> aspx

So the actual problem is with the msvcrt dlls. Can we build with msvc 2011 and link against 2010 crt dlls?

Comment 34

6 years ago
(In reply to Mike Hommey [:glandium] from comment #33)
> (In reply to Jim Mathies [:jimm] from comment #32)
> > The crt dll imports everything but our code really didn't pull anything in
> > directly except for GetTickCount64, which every one of our dlls imports. 
> > 
> > http://msdn.microsoft.com/en-us/library/windows/desktop/ms724411(v=vs.85).
> > aspx
> 
> So the actual problem is with the msvcrt dlls. Can we build with msvc 2011
> and link against 2010 crt dlls?

That should technically be possible, but it might have implications as some of the Win8 specific code might use runtime library functions which don't exist in vc10's crt.
(In reply to Ehsan Akhgari [:ehsan] from comment #34)
> That should technically be possible, but it might have implications as some
> of the Win8 specific code might use runtime library functions which don't
> exist in vc10's crt.

Does the win8 specific code really use stuff from the vc11 CRT, or does it use functions from the winrt dlls or whatever they are called?

Comment 36

6 years ago
(In reply to Mike Hommey [:glandium] from comment #35)
> (In reply to Ehsan Akhgari [:ehsan] from comment #34)
> > That should technically be possible, but it might have implications as some
> > of the Win8 specific code might use runtime library functions which don't
> > exist in vc10's crt.
> 
> Does the win8 specific code really use stuff from the vc11 CRT, or does it
> use functions from the winrt dlls or whatever they are called?

I'm not sure.  That's sort of a hard question to answer in general, but someone who knows how to build our Win8 stuff may try and see if that works today.
(Reporter)

Comment 37

6 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #36)
> (In reply to Mike Hommey [:glandium] from comment #35)
> > (In reply to Ehsan Akhgari [:ehsan] from comment #34)
> > > That should technically be possible, but it might have implications as some
> > > of the Win8 specific code might use runtime library functions which don't
> > > exist in vc10's crt.
> > 
> > Does the win8 specific code really use stuff from the vc11 CRT, or does it
> > use functions from the winrt dlls or whatever they are called?
> 
> I'm not sure.  That's sort of a hard question to answer in general, but
> someone who knows how to build our Win8 stuff may try and see if that works
> today.

I'm taking a shot at this by switching out runtime dlls in vc11 install. Will report back.
(In reply to Jim Mathies [:jimm] from comment #37)
> I'm taking a shot at this by switching out runtime dlls in vc11 install.
> Will report back.

You'll need to switch the import libraries as well.
(Reporter)

Comment 39

6 years ago
(In reply to Jim Mathies [:jimm] from comment #37)
> (In reply to Ehsan Akhgari [:ehsan] from comment #36)
> > (In reply to Mike Hommey [:glandium] from comment #35)
> > > (In reply to Ehsan Akhgari [:ehsan] from comment #34)
> > > > That should technically be possible, but it might have implications as some
> > > > of the Win8 specific code might use runtime library functions which don't
> > > > exist in vc10's crt.
> > > 
> > > Does the win8 specific code really use stuff from the vc11 CRT, or does it
> > > use functions from the winrt dlls or whatever they are called?
> > 
> > I'm not sure.  That's sort of a hard question to answer in general, but
> > someone who knows how to build our Win8 stuff may try and see if that works
> > today.
> 
> I'm taking a shot at this by switching out runtime dlls in vc11 install.
> Will report back.

I'm somewhat amazed and still a little skeptical but in a simple test with Fennec XUL plus the winrt code on Elm in a release build this seems to have worked. Not sure what's the best approach to really putting this to the test, I think I'll do up a desktop build and run mochitests with it next.

The only files I've swapped out are the two core crt dlls - 

msvcp110.dll
msvcr110.dll

in VS11/VC/redist/Microsoft.VC110.CRT

with the v100 versions from vs10.

The winrt library sits there as well (vccorlib110.dll) and doesn't seem to mind the v100 crt masquerading as the 110 version either.
(Reporter)

Comment 40

6 years ago
Oops, sorry, I spoke too soon. It's still pulling in the newer 110 dlls from windows/system32. I need to try again with the right import libraries.

Comment 41

6 years ago
Wait, you actually swapped the DLLs?  That seems like a bad idea.  There's a command line argument which you can use instead.  Don't remember what it's called, but it's on the main page for the project preferences UI in VC.

I can show it to Brian in person if you can't find it.  Ping me on IRC if you need me to.

Comment 42

6 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #24)
> FWIW, I think that moving XP users to the ESR branch is the wrong thing to
> do.  We just have too many users on that platform to deprive them of the
> latest and greatest features on the web.

We are absolutely not moving XP off to ESR. We're likely supporting XP users with our latest and greatest releases for several more years, minimum.
(Reporter)

Comment 43

6 years ago
Notes thus far from looking at various possible solutions. Of the options that might work, none are really very likable.

1) patching the crt dlls on install / first run

- entry point in which to do the patch work (zip install problem)
- patch would rename kernel32.dll crt dll import table
- we provide a moz kernel32 wrapper which implicitly links to kernel32.dll and is built with vs2010.
- wrapper provides missing entry points and calls through to kernel32
- moz wrapper would need all the kernel32 exports defined, pita but possible.
- our dlls also import one missing entry point, can we bake this into our code to fix?

2) delay load the crt and patch up crt entry points in a delay load hook

- can't delay load crt or kernel32.dll, both implicitly linked

3) working directory replacement shim for kernel32.dll

- would need all kernel32 entry points
- how would the wrapper get access to kernel32.dll? circular reference problem?
- simpler than option #1

4) Link to the older crt lib stubs/dlls with new tools

- can't, linker checks lib stub and target compiler versions, fails if they don't match.

5) split the metro widget code out into a separate library

- bad: two sets of build tools to build firefox
- no new c++ language features in platform
- no assurances it will actually work

6) create vs2010 builds for xp users, vs11 builds for newer os

- releng pain
- user pain / confusion
- no new c++ language features in platform
(Reporter)

Comment 44

6 years ago
Forgot one - 

7) statically link to the crt and bake missing entry points into our code.

Actually not that bad of a solution - we have about 12 dlls and one exe. If we remove the crt dlls from the install and add back multiple copies of the crt code we use, the size differences might not be too bad once compressed.

Comment 45

6 years ago
(In reply to Jim Mathies [:jimm] from comment #43)
> Notes thus far from looking at various possible solutions. Of the options
> that might work, none are really very likable.
> 
> 1) patching the crt dlls on install / first run
> 
> - entry point in which to do the patch work (zip install problem)

I think we might wanna do this at startup for zip installations, but it requires us to make sure that the executable which runs the first time that Firefox is launched does not link against those missing functions.

Also, this will most likely break installations on network drivers which are used by both Win7 and WinXP users, for example...

> - patch would rename kernel32.dll crt dll import table
> - we provide a moz kernel32 wrapper which implicitly links to kernel32.dll
> and is built with vs2010.
> - wrapper provides missing entry points and calls through to kernel32
> - moz wrapper would need all the kernel32 exports defined, pita but possible.
> - our dlls also import one missing entry point, can we bake this into our
> code to fix?

I don't like this at all.  How about the patch tool reading the import table of kernel32, remove those, and point them to our own shim DLL, and rewrite the PE image with the new import table?  This will make sure that we won't run into problems caused by us redirecting a symbol by mistake, etc.

> 2) delay load the crt and patch up crt entry points in a delay load hook
> 
> - can't delay load crt or kernel32.dll, both implicitly linked

I have experimented with this idea, and unfortunately I could not get it to work, because of the above reason.

> 3) working directory replacement shim for kernel32.dll
> 
> - would need all kernel32 entry points
> - how would the wrapper get access to kernel32.dll? circular reference
> problem?
> - simpler than option #1

AFAIK, Windows will never attempt to load a DLL named kernel32 from the app directory.

> 4) Link to the older crt lib stubs/dlls with new tools
> 
> - can't, linker checks lib stub and target compiler versions, fails if they
> don't match.
> 
> 5) split the metro widget code out into a separate library
> 
> - bad: two sets of build tools to build firefox
> - no new c++ language features in platform
> - no assurances it will actually work
> 
> 6) create vs2010 builds for xp users, vs11 builds for newer os
> 
> - releng pain
> - user pain / confusion
> - no new c++ language features in platform

(In reply to Jim Mathies [:jimm] from comment #44)
> Forgot one - 
> 
> 7) statically link to the crt and bake missing entry points into our code.
> 
> Actually not that bad of a solution - we have about 12 dlls and one exe. If
> we remove the crt dlls from the install and add back multiple copies of the
> crt code we use, the size differences might not be too bad once compressed.

This will at least cause us take a startup hit, since we'll need to read multiple copies of the same code from disk.  It won't be 12x the amount of current code, since the linker would strip out the unused stuff, but the CRT is like a spaghetti plate, a lot of its code is entangled with every other piece.  Someone needs to do measurements here, but I would not be surprised if this adds a few megs of more I/O to do at startup.  Although if you convince bsmedberg that this is the way to go, I won't object too much.  ;-)
(Reporter)

Comment 46

6 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #45)
> > - patch would rename kernel32.dll crt dll import table
> > - we provide a moz kernel32 wrapper which implicitly links to kernel32.dll
> > and is built with vs2010.
> > - wrapper provides missing entry points and calls through to kernel32
> > - moz wrapper would need all the kernel32 exports defined, pita but possible.
> > - our dlls also import one missing entry point, can we bake this into our
> > code to fix?
> 
> I don't like this at all.  How about the patch tool reading the import table
> of kernel32, remove those, and point them to our own shim DLL, and rewrite
> the PE image with the new import table?  This will make sure that we won't
> run into problems caused by us redirecting a symbol by mistake, etc.

If you think we can do all that rewriting of the crt dlls I'm fine with that. I was looking for the simplest approach. Since the crt dlls only have one dll listed in their import table, I'm assuming you feel we can add additional dlls to that list and rewrite the whole dll without messing up all the addressing? 

Personally I've never attempted anything like this before so I'm curious if it's not as big a deal as it feels.

> This will at least cause us take a startup hit, since we'll need to read
> multiple copies of the same code from disk.  It won't be 12x the amount of
> current code, since the linker would strip out the unused stuff, but the CRT
> is like a spaghetti plate, a lot of its code is entangled with every other
> piece.  Someone needs to do measurements here, but I would not be surprised
> if this adds a few megs of more I/O to do at startup.  Although if you
> convince bsmedberg that this is the way to go, I won't object too much.  ;-)

I'm going to experiment to get some measurements. At the end of the day this is looking like the simplest, most error free solution assuming no major perf regressions.  Size of the install really shouldn't be a show stopper, assuming it's not megs of data.

Comment 47

6 years ago
(In reply to Jim Mathies [:jimm] from comment #46)
> (In reply to Ehsan Akhgari [:ehsan] from comment #45)
> > > - patch would rename kernel32.dll crt dll import table
> > > - we provide a moz kernel32 wrapper which implicitly links to kernel32.dll
> > > and is built with vs2010.
> > > - wrapper provides missing entry points and calls through to kernel32
> > > - moz wrapper would need all the kernel32 exports defined, pita but possible.
> > > - our dlls also import one missing entry point, can we bake this into our
> > > code to fix?
> > 
> > I don't like this at all.  How about the patch tool reading the import table
> > of kernel32, remove those, and point them to our own shim DLL, and rewrite
> > the PE image with the new import table?  This will make sure that we won't
> > run into problems caused by us redirecting a symbol by mistake, etc.
> 
> If you think we can do all that rewriting of the crt dlls I'm fine with
> that. I was looking for the simplest approach. Since the crt dlls only have
> one dll listed in their import table, I'm assuming you feel we can add
> additional dlls to that list and rewrite the whole dll without messing up
> all the addressing? 
> 
> Personally I've never attempted anything like this before so I'm curious if
> it's not as big a deal as it feels.

This will not be easy (we'll need to write a micro-linker!) but I believe it would be possible.  Relying on the assumption that we'll properly take care of all kernel32 import entries is way scarier IMO.

> > This will at least cause us take a startup hit, since we'll need to read
> > multiple copies of the same code from disk.  It won't be 12x the amount of
> > current code, since the linker would strip out the unused stuff, but the CRT
> > is like a spaghetti plate, a lot of its code is entangled with every other
> > piece.  Someone needs to do measurements here, but I would not be surprised
> > if this adds a few megs of more I/O to do at startup.  Although if you
> > convince bsmedberg that this is the way to go, I won't object too much.  ;-)
> 
> I'm going to experiment to get some measurements. At the end of the day this
> is looking like the simplest, most error free solution assuming no major
> perf regressions.  Size of the install really shouldn't be a show stopper,
> assuming it's not megs of data.

OK, let's see what happens here.  Also, please make sure to do PGO builds for measurements.  Thanks!
(Reporter)

Comment 48

6 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #47)
> This will not be easy (we'll need to write a micro-linker!) but I believe it
> would be possible.  Relying on the assumption that we'll properly take care
> of all kernel32 import entries is way scarier IMO.

Not sure why you feel this way. I've done call through wrappers in the past for things like opengl. It's tedious work putting it together but there aren't any dangers to doing it. I would rather not do a call through wrapper or a micro-linker to be honest - I'd probably start pushing for separate xp builds with vs2010.

I wonder what our xp install base is these days. If it's a few million out of 150+ adus pushing those users off to a separate install might not be that big of a deal.

Comment 49

6 years ago
(In reply to Jim Mathies [:jimm] from comment #48)
> (In reply to Ehsan Akhgari [:ehsan] from comment #47)
> > This will not be easy (we'll need to write a micro-linker!) but I believe it
> > would be possible.  Relying on the assumption that we'll properly take care
> > of all kernel32 import entries is way scarier IMO.
> 
> Not sure why you feel this way. I've done call through wrappers in the past
> for things like opengl. It's tedious work putting it together but there
> aren't any dangers to doing it. I would rather not do a call through wrapper
> or a micro-linker to be honest - I'd probably start pushing for separate xp
> builds with vs2010.

I just think that the micro-linker would be much easier to get right (and much less work too), but yeah, I agree that doing separate builds is probably gonna be the safest option.  :/

> I wonder what our xp install base is these days. If it's a few million out
> of 150+ adus pushing those users off to a separate install might not be that
> big of a deal.

According to <http://gs.statcounter.com/#os-ww-monthly-201104-201204>, around 30% of the Internet users are on XP.  If by a separate install you mean something like ESR, I don't think that's an option (see comment 42).
(In reply to Ehsan Akhgari [:ehsan] from comment #49)
> I just think that the micro-linker would be much easier to get right (and
> much less work too), but yeah, I agree that doing separate builds is
> probably gonna be the safest option.  :/

But it pushes the costs onto releng and (worse) our users. And our developers since we can't use post-VC2010 C++ features.

I agree with Ehsan about the micro-linker being easier, except I think calling it a "micro-linker" makes it sound bigger and scarier than it's likely to be.

(In reply to Ehsan Akhgari [:ehsan] from comment #45)
> I think we might wanna do this at startup for zip installations, but it
> requires us to make sure that the executable which runs the first time that
> Firefox is launched does not link against those missing functions.

I think we could get away with saying that people running ZIP builds on Windows XP need to run some helper tool in the ZIP before proceeding. Call it "MakeXPCompatible.exe"...

> Also, this will most likely break installations on network drivers which are
> used by both Win7 and WinXP users, for example...

We could make the installer always do the patching even if you're on Windows 7. Or, administrators could run MakeXPCompatible.exe themselves.
(Reporter)

Comment 51

6 years ago
(In reply to Jim Mathies [:jimm] from comment #46)
> I'm going to experiment to get some measurements. At the end of the day this
> is looking like the simplest, most error free solution assuming no major
> perf regressions.  Size of the install really shouldn't be a show stopper,
> assuming it's not megs of data.

As far as install size growth goes, this adds about 6.5 MB to rel zip builds (16MB -> 22.2MB). I'd consider this too much growth. 

As far as perf stats go I don't have those as I've run into problems with our js engine which doesn't seem to be working when statically linking to the crt.

So unless someone has a major objection, I think the next step would be to experiment with a local patcher per roc's comments.

Comment 52

6 years ago
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #50)
> (In reply to Ehsan Akhgari [:ehsan] from comment #49)
> > I just think that the micro-linker would be much easier to get right (and
> > much less work too), but yeah, I agree that doing separate builds is
> > probably gonna be the safest option.  :/
> 
> But it pushes the costs onto releng and (worse) our users. And our
> developers since we can't use post-VC2010 C++ features.
> 
> I agree with Ehsan about the micro-linker being easier, except I think
> calling it a "micro-linker" makes it sound bigger and scarier than it's
> likely to be.

Heh, yeah OK let's not call it a micro-linker.  :-)

The PE file format is very simple, and well documented, which is why I think this would not be too hard.

> (In reply to Ehsan Akhgari [:ehsan] from comment #45)
> > I think we might wanna do this at startup for zip installations, but it
> > requires us to make sure that the executable which runs the first time that
> > Firefox is launched does not link against those missing functions.
> 
> I think we could get away with saying that people running ZIP builds on
> Windows XP need to run some helper tool in the ZIP before proceeding. Call
> it "MakeXPCompatible.exe"...
> 
> > Also, this will most likely break installations on network drivers which are
> > used by both Win7 and WinXP users, for example...
> 
> We could make the installer always do the patching even if you're on Windows
> 7. Or, administrators could run MakeXPCompatible.exe themselves.

Hmm, another alternative would be to build this IAT-fixer as a host tool during the build process, and run it on the binaries that we produce as a last step before packaging.  This will make things much simpler I think.  There really isn't any reason why the binaries need to be different on XP.  The shim DLL which IAT-fixer redirects the missing functions can just look at kernel32.dll at runtime, and call the existing implementation if there is one.

(In reply to Jim Mathies [:jimm] from comment #51)
> (In reply to Jim Mathies [:jimm] from comment #46)
> > I'm going to experiment to get some measurements. At the end of the day this
> > is looking like the simplest, most error free solution assuming no major
> > perf regressions.  Size of the install really shouldn't be a show stopper,
> > assuming it's not megs of data.
> 
> As far as install size growth goes, this adds about 6.5 MB to rel zip builds
> (16MB -> 22.2MB). I'd consider this too much growth. 
> 
> As far as perf stats go I don't have those as I've run into problems with
> our js engine which doesn't seem to be working when statically linking to
> the crt.
> 
> So unless someone has a major objection, I think the next step would be to
> experiment with a local patcher per roc's comments.

Agreed, sounds good!
(Reporter)

Comment 53

6 years ago
Created attachment 622441 [details]
msvc 10 redist.txt

Just to confirm before we start in on this, I'd like to be sure we can't modify these files before we distribute them. The language seems to imply you can't modify the folder name, and I think also implies the files, but I'm not 100% sure. The EULA from VS2010 is posted here -

http://social.msdn.microsoft.com/Forums/en/vssetup/thread/0368d7ee-0eb3-4e3e-a143-4410969a15bb

and doesn't specifically mention that these files need to be unmodified.

Gerv, curious if you might have some idea or can help connect us to the right person in legal?
Attachment #622441 - Flags: feedback?(gerv)
(In reply to Ehsan Akhgari [:ehsan] from comment #52)
> The shim DLL which IAT-fixer redirects the missing functions can just look
> at kernel32.dll at runtime, and call the existing implementation if there is
> one.

I didn't think of that. That makes complete sense and is definitely the right way to go, if we can do it legally.
:jimm: Here's my non-lawyerly opinion: unless specific permission is given to distribute _modified_ versions (as it is for some things in that EULA, but not for 'things listed in redist.txt'), you can't distribute modified versions.

(But I thought we used to distribute a modified CRT with jemalloc in it; am I confused about that? If so, there will be legal analysis of how that's possible knocking around somewhere.)

Currently, we do redistribute unmodified msvc*.dll files on Windows - see about:license on a Windows build.

The allowability of patching each individual's on-disk copy of such a DLL (not sure if that option is still under consideration) would need to be discussed with our lawyers, but it's not clearly and obviously disallowed. The first question would be whether the right to do so is one which is reserved by copyright law.

Does that help? If not, ask more specific questions :-)

Gerv
What we really need is the Visual Studio 2011/2012 EULA. Anyone got that?
(Reporter)

Comment 57

6 years ago
Created attachment 622699 [details]
2011 beta license

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #56)
> What we really need is the Visual Studio 2011/2012 EULA. Anyone got that?

Sure, attached. Also, here's the redist:

http://msdn.microsoft.com/en-US/vstudio/hh857605.aspx
(In reply to Gervase Markham [:gerv] from comment #55)
> (But I thought we used to distribute a modified CRT with jemalloc in it; am
> I confused about that? If so, there will be legal analysis of how that's
> possible knocking around somewhere.)

We used to compile a patched CRT from source. Microsoft provides the CRT sources with VC++ for this express purpose. Unfortunately as of VC2010 they stopped providing the build scripts to compile the sources, so that option is much more difficult.
(In reply to Ted Mielczarek [:ted] from comment #58)
> (In reply to Gervase Markham [:gerv] from comment #55)
> > (But I thought we used to distribute a modified CRT with jemalloc in it; am
> > I confused about that? If so, there will be legal analysis of how that's
> > possible knocking around somewhere.)
> 
> We used to compile a patched CRT from source. Microsoft provides the CRT
> sources with VC++ for this express purpose. Unfortunately as of VC2010 they
> stopped providing the build scripts to compile the sources, so that option
> is much more difficult.

OTOH, we have the machinery to kind of patch the CRT from its binary, which we do for free().

Comment 60

6 years ago
Assuming Microsoft has actually supplied all the sources for the CRT, I really don't think it would be that hard to come up with build scripts for it, at least in the particular configuration that we care about (multithreaded DLL). The tangled mess that was the original build scripts was mainly dealing with a bunch of different configurations.
My MSVC2010 install dir has:
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\crt\src

Someone could check the corresponding VC2011 sources.
(Reporter)

Comment 62

6 years ago
According to this social post, the current source is missing key binaries -

http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/e6f222d7-8e20-4d4a-8a6f-b72ade3661ac

Ehsan might have something to say about this as he looked looked into trying to do this with 2010. I also took a stab at it using the 2008 build scripts. Unfortunately things have changed quite a bit so those are not of much help.

Comment 63

6 years ago
Yeah, IIRC it was not only the build script missing, but some object files which used to come with the CRT without the corresponding source are missing as well, so we can't really link a binary even if we simulate the build script.
(In reply to Mike Hommey [:glandium] from comment #59)
> OTOH, we have the machinery to kind of patch the CRT from its binary, which
> we do for free().

So we're already modifying the CRT we distribute?
(In reply to Mike Hommey from comment #59)
> OTOH, we have the machinery to kind of patch the CRT from its binary, which
> we do for free().
No, that's different. All modules statically link the CRT startup, which passes a pointer obtained from an internal memory allocation call to the imported free symbol. We rewrite that symbol so that it imports a dummy free function that we write so that we don't crash on shutdown.
(Reporter)

Updated

6 years ago
Attachment #622441 - Attachment is obsolete: true
Attachment #622441 - Flags: feedback?(gerv)

Updated

6 years ago
Assignee: nobody → ehsan
(Reporter)

Comment 66

6 years ago
Just and FYI, we don't have a definitive answer yet on the licensing so we've sent this over to Harvey for analysis. Will post back when I hear back from him.

Comment 67

6 years ago
I cant speak as to the licensing but I have done an analysis of the VS11 pro beta from Microsoft (available at http://www.microsoft.com/visualstudio/11/en-us/downloads ) and I can say that whilst it contains more of the CRT source code than my VS10 pro install, it does not appear to contain everything required to recompile msvcr110.dll. Not only is there not enough information on how to compile the CRT (compiler flags, which files to link into the different dlls etc) but an analysis of msvcrt.lib and msvcr110.dll shows that there is code linked into those files that is NOT included in the CRT source. (atlssup.obj being one example of a file that is included in the the CRT binaries but not in the CRT source)

Comment 68

6 years ago
Just as a status update, I've been buried with the work on background updates, and have had no time to work on this so far.
(Reporter)

Comment 69

6 years ago
The opinion of legal came back last week - generally, we can't modify these dlls either during the build or on the client side.

This doesn't leave us a whole lot of options. If we stick with a single compiler we'll be using the VC 2010 version for as long as we support XP and our win8 work will be negatively impacted. 

Rob mentioned the possible use of a stub install that brings down the right resources depending on the os. This would involve dual builds on the rel eng side. Not sure how it would affect the updater. Some investigation is needed.
 
If anyone has any additional ideas, please post.
Depends on: 756694

Comment 70

6 years ago
I haven't played with the Mozilla codebase in a while so I dont know how fesable this is but here is my idea:
1.Build a full set of release builds (FF, TB, SM etc) with VS11 in the form that would run on XP were it not for these few kernel functions.
2.Produce a list of all the imports these dlls import from the CRT DLLs
3.Then with this list, there are 3 options, option 1 is to replace that function with a functionally equivalent clone added to one of the core moz dlls (either existing-and-suitably-licensed or new-for-Mozilla), option 2 is to statically link a single copy of the function from the static CRT libraries into the aforementioned core dlls and have all the other bits of the codebase link to this one copy
4.For each function that is dealt with under point 3, it may have further dependencies on other functions that will also need to be dealt with.

There ARE ways to have the linker use <function X> from <lib Y> instead of from the CRT library, I have used them before.

There is also this:
http://www.benshoof.org/blog/minicrt/
where its clear other people have thought about ways to reduce/eliminate dependencies on the Microsoft CRT for the purpose of smaller code size (even Google has done something with it for one of their installers) although it seems to be stuck around VS2008 and doesn't cover VS10 or VS11. But it might provide some insight that could be used in this effort.

Comment 71

6 years ago
Obviously after that is done, you can then add the appropriate clones/stubs for the remaining missing-on-XP kernel functions which get linked into a shared dll and exposed to the rest of the dlls.

And you would still need a separate Metro dll only loaded on Windows 8 (or a separate "Windows 8 with Metro" build for Windows 8 users)
> Rob mentioned the possible use of a stub install that brings down the right 
> resources depending on the os. This would involve dual builds on the rel 
> eng side. Not sure how it would affect the updater. Some investigation is needed.

I think it would make the most sense to just treat Windows 8 and beyond as a separate platform with its own installer and update MAR completely.  This would also allow us to do anything win8 and beyond with a simple #ifdef XP_WIN8 check instead of loading it in dynamically.  

If we did the opposite and only left XP as its own build, I think we'll run into this same problem when MS stops supporting Vista. 

This wouldn't require a stub installer but would be nicest with one. Without the stub we would just have a bit of web work to do to redirect to the correct download depending on the OS.  

For updates if we detected that the user is using a build less than win8 but is on a higher or equal version of win8 computer then we would upgrade them to the other platform with a full MAR.
(Reporter)

Comment 73

6 years ago
(In reply to jfwfreo from comment #70)
> I haven't played with the Mozilla codebase in a while so I dont know how
> fesable this is but here is my idea:
> 1.Build a full set of release builds (FF, TB, SM etc) with VS11 in the form
> that would run on XP were it not for these few kernel functions.
> 2.Produce a list of all the imports these dlls import from the CRT DLLs
> 3.Then with this list, there are 3 options, option 1 is to replace that
> function with a functionally equivalent clone added to one of the core moz
> dlls (either existing-and-suitably-licensed or new-for-Mozilla), option 2 is
> to statically link a single copy of the function from the static CRT
> libraries into the aforementioned core dlls and have all the other bits of
> the codebase link to this one copy
> 4.For each function that is dealt with under point 3, it may have further
> dependencies on other functions that will also need to be dealt with.

Interesting idea, basically build our own crt library that statically links to the real crt and also adds the missing system exports so it can load on XP. We would also have to link the missing system exports component into our other statically linked modules, like webaprt and a couple of the glue libs.

Comment 74

6 years ago
(In reply to Jim Mathies [:jimm] from comment #73)
> (In reply to jfwfreo from comment #70)
> > I haven't played with the Mozilla codebase in a while so I dont know how
> > fesable this is but here is my idea:
> > 1.Build a full set of release builds (FF, TB, SM etc) with VS11 in the form
> > that would run on XP were it not for these few kernel functions.
> > 2.Produce a list of all the imports these dlls import from the CRT DLLs
> > 3.Then with this list, there are 3 options, option 1 is to replace that
> > function with a functionally equivalent clone added to one of the core moz
> > dlls (either existing-and-suitably-licensed or new-for-Mozilla), option 2 is
> > to statically link a single copy of the function from the static CRT
> > libraries into the aforementioned core dlls and have all the other bits of
> > the codebase link to this one copy
> > 4.For each function that is dealt with under point 3, it may have further
> > dependencies on other functions that will also need to be dealt with.
> 
> Interesting idea, basically build our own crt library that statically links
> to the real crt and also adds the missing system exports so it can load on
> XP. We would also have to link the missing system exports component into our
> other statically linked modules, like webaprt and a couple of the glue libs.

That would only work in case the static stuff that VC links into binaries do not depend on the missing exports.

Comment 75

6 years ago
You could create a clone of GetTickCount64 and export that from some linked-into-every-other-Mozilla-binary glue library which is then used by every binary in place of the kernel32.dll version of that function (GetTickCount64 is the one non-XP function imported by the static-linked code in msvcrt.lib)

Comment 76

6 years ago
Created attachment 632156 [details]
msvcr110.dll/msvcp110.dll imports used by Firefox

Further to my ideas from before, I have done some tests to identify all the imports from msvcr110.dll and msvcp110.dll that Firefox is currently using.
What I did was as follows:
1.Install Microsoft Visual Studio 11 Beta (cl.exe claims its version 17.00.50214.1)
2.Build mozilla-central following these instructions
https://developer.mozilla.org/en/Windows_8 using revision 5a4551a75db3 and the following .mozconfig:
mk_add_options MOZ_MAKE_FLAGS="-s -j4"
ac_add_options --disable-angle
3.Make a dist zip file of the resulting build
4.Unzip the zip file and run an import dumper on each exe and dll to identify anything they imported from msvcr110.dll and msvcp110.dll

The results are attached, its a list of all exports from msvcr110.dll and msvcp110.dll that are used by my test vs11 Firefox build.

I am also investigating a way to identify the symbols pulled in from the static libraries (msvcrt.lib etc) probably by generating a map file and parsing the results of that.

The way I see it, we have 3 choices of how to proceed if we want to go down this road (as opposed to having a separate vs11/metro build or whatever)
1.We continue to link to msvcrt.lib but replace the imports from msvcr110.dll and msvcp110.dll with imports from a Mozilla dll with functions that do an equivalent job (how you come up with those functions would have to be identified)

2.We go further and use my .map file idea above and then replace some parts of the static msvcrt.lib/msvcprt.lib with our new implementations in addition to replacing the msvcr110.dll/msvcp110.dll functions.

or 3.We take the approach of the minicrt/libctiny I linked to before. This would involve building a full-static (i.e. linked to libcmt.lib/libcpmt.lib) copy of FF, somehow identifying all the symbols that the FF obj files need from libcmt.lib/libcpmt.lib and then providing an implementation of each one. The implementation could be the implementation in libc*.lib, an existing implementation from elsewhere (that's under a suitable license) or a new impelementation. Some of these implementations could live in a shared Mozilla dll, others would need to be linked into each binary.
(Reporter)

Comment 77

6 years ago
(In reply to jfwfreo from comment #76)
> Created attachment 632156 [details]
> msvcr110.dll/msvcp110.dll imports used by Firefox
> 
> Further to my ideas from before, I have done some tests to identify all the
> imports from msvcr110.dll and msvcp110.dll that Firefox is currently using.
> What I did was as follows:
> 1.Install Microsoft Visual Studio 11 Beta (cl.exe claims its version
> 17.00.50214.1)
> 2.Build mozilla-central following these instructions
> https://developer.mozilla.org/en/Windows_8 using revision 5a4551a75db3 and
> the following .mozconfig:
> mk_add_options MOZ_MAKE_FLAGS="-s -j4"
> ac_add_options --disable-angle
> 3.Make a dist zip file of the resulting build
> 4.Unzip the zip file and run an import dumper on each exe and dll to
> identify anything they imported from msvcr110.dll and msvcp110.dll
> 
> The results are attached, its a list of all exports from msvcr110.dll and
> msvcp110.dll that are used by my test vs11 Firefox build.
> 
> I am also investigating a way to identify the symbols pulled in from the
> static libraries (msvcrt.lib etc) probably by generating a map file and
> parsing the results of that.
> 
> The way I see it, we have 3 choices of how to proceed if we want to go down
> this road (as opposed to having a separate vs11/metro build or whatever)
> 1.We continue to link to msvcrt.lib but replace the imports from
> msvcr110.dll and msvcp110.dll with imports from a Mozilla dll with functions
> that do an equivalent job (how you come up with those functions would have
> to be identified)

If we can get it to work I like option one. Replacing any part of the crt with 3rd party code has overhead in terms of testing and security review work. It would be simpler to use the existing static crt if we can. As far kernel32 exports we have a starting point in this work -

http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/

There's no license on this code so we may end up having to clean room our own but before we do that we can test the experiment and if we think it'll work contact the author about usage.

Comment 78

6 years ago
> * Linking statically to the crt is not a good option, for code size/memory
> usage reasons.  And even if we decide that we don't care about any of that,
> we have no evidence to suggest that we can effectively simulate the missing
> crt import entries with our hacky implementations on WinXP.
Does anyone know if its possible to link statically to the CRT right now? (forget about code size/memory usage, does FF actually work if you do that?)
If its possible, I have some experiments I plan to run and some data I plan to gather (I also need to figure out if its possible to hack the build system so that I can get .map files generated for the various binaries)
(In reply to Jim Mathies [:jimm] from comment #77)
> http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-
> statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/
> 
> There's no license on this code so we may end up having to clean room our
> own 

If you want me to contact him and sort that out, just shout.

Gerv
(Reporter)

Comment 80

6 years ago
(In reply to jfwfreo from comment #78)
> > * Linking statically to the crt is not a good option, for code size/memory
> > usage reasons.  And even if we decide that we don't care about any of that,
> > we have no evidence to suggest that we can effectively simulate the missing
> > crt import entries with our hacky implementations on WinXP.
> Does anyone know if its possible to link statically to the CRT right now?
> (forget about code size/memory usage, does FF actually work if you do that?)
> If its possible, I have some experiments I plan to run and some data I plan
> to gather (I also need to figure out if its possible to hack the build
> system so that I can get .map files generated for the various binaries)

We experimented with that, see comment 51. The overall growth of the install was pretty big, and the js lib had some issues. I'm sure we could work around the js problems but bloating the install by 6-7 megs compressed renders full static link for every component a non-option.
(Reporter)

Comment 81

6 years ago
(In reply to Gervase Markham [:gerv] from comment #79)
> (In reply to Jim Mathies [:jimm] from comment #77)
> > http://tedwvc.wordpress.com/2012/03/11/how-to-get-visual-c-2012-vc-11-beta-
> > statically-linked-crt-and-mfc-applications-to-run-on-windows-xp/
> > 
> > There's no license on this code so we may end up having to clean room our
> > own 
> 
> If you want me to contact him and sort that out, just shout.
> 
> Gerv

That would be great Gerv. No assurance we'll end up using it in our final approach but it would be good to know whether we would be able to use his work at all.

Comment 82

6 years ago
(In reply to Jim Mathies [:jimm] from comment #77)
> (In reply to jfwfreo from comment #76)
> > The way I see it, we have 3 choices of how to proceed if we want to go down
> > this road (as opposed to having a separate vs11/metro build or whatever)
> > 1.We continue to link to msvcrt.lib but replace the imports from
> > msvcr110.dll and msvcp110.dll with imports from a Mozilla dll with functions
> > that do an equivalent job (how you come up with those functions would have
> > to be identified)
> 
> If we can get it to work I like option one.

I think I'm missing something.  Can someone please explain this option in more detail?  How are we going to replace the imports from those two DLLs?

Comment 83

6 years ago
Basically the idea I had was that we build a list of all msvcr110.dll/msvcp110.dll imports we are using (which I posted before from my test VS11 FF build). 
Then we create functionally-equivalent clones of each one (by linking in the version of the function in libcmt.lib, by using code from a 3rd party that we can use under a suitable license or by writing our own)
These clones would go in a shared moz dll and be exported from it. We then pass the import library for this shared dll (either an existing dll that all the binaries link into like NSPR or a new dll just for the CRT) to the linker in a way that symbols in it override the symbols in msvcr110.dll/msvcp110.dll

Comment 84

6 years ago
OK, yep, so like I said, we need to make sure that the object files linked in to our binaries will not add any imports to those functions non-existing on XP.

One way to verify this would be to build Firefox using VC11, then take all of the binaries produced by the build system to Dependency Walker on Windows XP and making sure that none of them have such imports.

Of course this would be a tad fragile, especially if we start using C++11 features which pull in new problematic code from the object files linked into our binaries under our nose.
(In reply to Ehsan Akhgari from comment #84)
> Of course this would be a tad fragile, especially if we start using C++11
> features which pull in new problematic code from the object files linked
> into our binaries under our nose.
If we disable the real CRT then the link will fail, until someone fixes the "miniCRT" to export the new code, won't it?

Comment 86

6 years ago
(In reply to neil@parkwaycc.co.uk from comment #85)
> (In reply to Ehsan Akhgari from comment #84)
> > Of course this would be a tad fragile, especially if we start using C++11
> > features which pull in new problematic code from the object files linked
> > into our binaries under our nose.
> If we disable the real CRT then the link will fail, until someone fixes the
> "miniCRT" to export the new code, won't it?

Not necessarily.  For some of the language features, the compiler links in prebuilt object files that ship with it into our binaries when we link.  There's nothing which would cause that to fail to link.

Comment 87

6 years ago
We know the kernel32.dll functions that are missing on XP. And we can find out (from the shipped CRT source and the various libraries and dlls) which symbols directly call those functions. So we can create functionally-equivalent copies of either the kernel32.dll function or the CRT function and that will deal with the problems for there.

Then we just need to make sure that when building VS11-for-XP, we run a check on the resulting binaries somehow looking for references to msvcr110.dll or msvcp110.dll and report linking to either dll as a bug (so someone knows to do something about that function)

Comment 88

6 years ago
(In reply to jfwfreo from comment #87)
> We know the kernel32.dll functions that are missing on XP. And we can find
> out (from the shipped CRT source and the various libraries and dlls) which
> symbols directly call those functions. So we can create
> functionally-equivalent copies of either the kernel32.dll function or the
> CRT function and that will deal with the problems for there.

I'm specifically worried about the object files which are not shipped with their source.  FWIW, I think there's a very small chance that what I'm worrying actually happens in practice.  :-)

> Then we just need to make sure that when building VS11-for-XP, we run a
> check on the resulting binaries somehow looking for references to
> msvcr110.dll or msvcp110.dll and report linking to either dll as a bug (so
> someone knows to do something about that function)

We should be more severe than that, we should *fail* the builds if any of the resulting binaries start referencing one of those functions!

Comment 89

6 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #88)
> I'm specifically worried about the object files which are not shipped with
> their source.  FWIW, I think there's a very small chance that what I'm
> worrying actually happens in practice.  :-)
We can easily identify every file that links to the handful of missing kernel functions and if a function we need to deal with happens to be closed-source, we can cross that bridge when we come to it.
> We should be more severe than that, we should *fail* the builds if any of
> the resulting binaries start referencing one of those functions!
Yeah it should be possible to add some stuff to the build system that uses microsoft dumpbin.exe /imports on all our binaries and flags a warning if we import anything we are not supposed to import (i.e. if the output from dumpbin.exe references msvcr110.dll or msvcp110.dll, we can throw an error listing the function(s) that are imported from those dlls that should not be)
We could also do the same thing and have a list in tree listing all kernel32.dll functions exported by kernel32.dll on XP and check if we import anything not on the list and flag an error.

But the first step is to identify all the symbols that the ff binaries pull in from the various CRT libraries (I am identifying that now) and figuring out the best solution for dealing with each of em. (i.e. do we use the MS implementation from msvcrt.lib, do we use an existing tested suitably-licensed 3rd party implementation or do we do something new)
Microsoft is apparently going to come out with an official solution to this problem "post-RTM", but the timescale is unknown:
http://supportxp.com/2012/05/24/its-official-no-xp-targeting-support-in-visual-c-11-2012-rtm-version/

Negotiations with the author of the existing code are progressing well; he hopes to have proper licensing sorted by the end of the week.

Gerv

Comment 91

6 years ago
(In reply to Gervase Markham [:gerv] from comment #90)
> Microsoft is apparently going to come out with an official solution to this
> problem "post-RTM", but the timescale is unknown:
> http://supportxp.com/2012/05/24/its-official-no-xp-targeting-support-in-
> visual-c-11-2012-rtm-version/

If this is our only source about this, I wouldn't hold my breath for it.  :-)

BTW, comment 89 makes perfect sense.  Thanks for working on this, jfwfreo!

Comment 92

6 years ago
Created attachment 633047 [details]
symbols in CRT libraries referenced by Firefox code

I have used the /VERBOSE option to link.exe along with a bunch of manual processing of the output and have produced a list containing all the symbols in the CRT libraries that are referenced directly by code other than the CRT libraries (i.e. not one CRT library module referencing another CRT library module)

So this is the list of symbols any "mini CRT" would need to provide unless code is changed to not depend on these symbols.

Comment 93

6 years ago
Can we get legal clarification (e.g. contact authors or whatever else) about use of the code in http://code.google.com/p/omaha/source/browse/trunk/third_party/minicrt/ and http://www.benshoof.org/blog/minicrt/
There is code in there that may be of value for us to use if we can.
(In reply to jfwfreo from comment #93)
> Can we get legal clarification (e.g. contact authors or whatever else) about
> use of the code in
> http://code.google.com/p/omaha/source/browse/trunk/third_party/minicrt/ 

This is Apache-licensed and shipped by Google, who therefore must have checked it out. It's OK to use, but if you want to copy Apache-licensed code into MPLed files, let us know as we have to get the boilerplate right.

> and http://www.benshoof.org/blog/minicrt/

Reading that page, this is a licensing quagmire. He's copied code from all sorts of places, including the Microsoft CRT source and random places around the net. I'm not saying we can't use anything from there, but we'd have to look at bits of code on a case-by-case basis, and we certainly can't take the thing wholesale.

Gerv
(In reply to Gervase Markham [:gerv] from comment #94)
> (In reply to jfwfreo from comment #93)
> > Can we get legal clarification (e.g. contact authors or whatever else) about
> > use of the code in
> > http://code.google.com/p/omaha/source/browse/trunk/third_party/minicrt/ 
> 
> This is Apache-licensed and shipped by Google, who therefore must have
> checked it out. It's OK to use, but if you want to copy Apache-licensed code
> into MPLed files, let us know as we have to get the boilerplate right.

Note that there are plenty of object files without corresponding source. These are most likely *not* Apache licensed.
You are quite right :-| Lots of files have Microsoft copyright notices as well, with no permissions statement... I guess I could go and ask Google about that. But first, jfwfreo: can you tell us more about which code you think would be useful? Was your plan to use the entire project as-is, or did you just mean "there's a couple of functions here we could use"?

Gerv

Comment 97

6 years ago
It would appear that both the Google project and the other link contain a mixture of code from 4 sources:
1.Code from the original libctiny written by Matt
2.Code from the Microsoft CRT sources
3.obj files from the Microsoft CRT
and 4.Original code written by Google and by the other guy

I dont think we can legally use #2 or #3 and the license on #1 and #2 is unclear.
As for the reason to use it is that it might contain implementations of things we can use, most of it we probably dont need but some of it might help.
We can always worry about licensing once we discover that there is something there we want to use.

My next step is going to be to look at my list from earlier and see what we might have in-tree that could potentially be a suitable replacement (e.g. if we have anything in NSPR)
jfwfreo: I'm sure better hackers than I can jump in and tell me whether I've got the wrong end of the stuck, but it seems to me that the idea that we should dump the MS CRT in favour of our own reimplemented one (that is what you are suggesting?), made from code pulled in from various sources, would be a big change at the bottom of Mozilla, and have potential impacts on performance, correctness, and more. Such a library, particularly one containing new code, would need a lot of QA.

Is it worth getting wider buy-in from core Mozilla hackers before going off down that route?

Gerv
(Reporter)

Comment 99

6 years ago
(In reply to jfwfreo from comment #97)
> My next step is going to be to look at my list from earlier and see what we
> might have in-tree that could potentially be a suitable replacement (e.g. if
> we have anything in NSPR)

Wouldn't it be easier to statically link to the Win runtime lib and have this dll act as a pass through for most everything?

Comment 100

6 years ago
The #1 issue with simply linking to the static CRT and exporting functions is that there are differences between the static and dynamic CRT. There are things that msvcr110.dll exports (and that msvcrt.lib imports) that are not present in libcmt.lib or that are present but function differently.
(Reporter)

Comment 101

6 years ago
(In reply to jfwfreo from comment #100)
> The #1 issue with simply linking to the static CRT and exporting functions
> is that there are differences between the static and dynamic CRT. There are
> things that msvcr110.dll exports (and that msvcrt.lib imports) that are not
> present in libcmt.lib or that are present but function differently.

Can you give an example?

We can build xul.dll by either linking to msvcrt.lib or libcmt.lib, both work. I'd be surprised if functionality between the two were different.

At the end of the day we are interested in exporting the main crt entry points the mozilla code base uses from a new dll we control. If we can build our dlls with /nodefaultlibs and link to a lib stub that points to a dll that provides these apis with pass through to the functionality in the ms static libs, we should be ok. (I think!)

Comment 102

6 years ago
(In reply to Jim Mathies [:jimm] from comment #101)
> (In reply to jfwfreo from comment #100)
> > The #1 issue with simply linking to the static CRT and exporting functions
> > is that there are differences between the static and dynamic CRT. There are
> > things that msvcr110.dll exports (and that msvcrt.lib imports) that are not
> > present in libcmt.lib or that are present but function differently.
> 
> Can you give an example?
> 
> We can build xul.dll by either linking to msvcrt.lib or libcmt.lib, both
> work. I'd be surprised if functionality between the two were different.

IIRC the C runtime makes assumptions about whether the static version is used (in which case some things are kept track of per module) or the dynamic version is used (in which case some things can be kept track of in the same place.  But linking in the CRT in one module statically and then exporting the symbols dynamically might work, since the CRT already knows how to handle multiple copies being used in the same process (i.e., you can mix modules linking statically and dynamically to the CRT -- perhaps different versions of it -- in the same process.)

> At the end of the day we are interested in exporting the main crt entry
> points the mozilla code base uses from a new dll we control. If we can build
> our dlls with /nodefaultlibs and link to a lib stub that points to a dll
> that provides these apis with pass through to the functionality in the ms
> static libs, we should be ok. (I think!)

Yeah.  Just to be clear, I don't think we want to use the minicrt library at all, because of technical and legal reasons.  To the best of my knowledge, the only big project which uses it is Omaha, which *only* uses it in the code for the stub executable, which I'm sure uses a very small version of the CRT.  Replacing the CRT in a large application such as Firefox takes way too much effort to ensure compatibility, correctness and performance, and I don't think we have the resources to dedicate to that effort.

Comment 103

6 years ago
Just as one example, the symbol __setusermatherr is pulled from msvcr110.dll by my test FF build. That symbol does not exist anywhere in libcmt.lib.
Some debugging shows that __setusermatherr comes from a file called matherr.c (which for reasons known only to Microsoft is not included in the CRT source code)

Comment 104

6 years ago
And I do concur that totally rewriting the CRT is a bad idea.
But there are definatly cases where we will have to come up with replacements for specific symbols.

Comment 105

6 years ago
(In reply to jfwfreo from comment #103)
> Just as one example, the symbol __setusermatherr is pulled from msvcr110.dll
> by my test FF build. That symbol does not exist anywhere in libcmt.lib.
> Some debugging shows that __setusermatherr comes from a file called
> matherr.c (which for reasons known only to Microsoft is not included in the
> CRT source code)

Hmm, I do have __libm_setusermatherr in libcmt.lib.
(Reporter)

Comment 106

6 years ago
I don't think we need to worry about entry points like this. My guess is that if fx modules were linked with nodefaultlibs to the wrapper it would go away. Our code base doesn't pull this entry point in on it's own. The code for setusermatherr (and the apis that leverage it) would be embedded in the wrapper dll.
Just for the record, Google's solution to this problem was to build their Chrome Metro builds with Visual C++ 2010 by lightly patching the Windows 8 SDK headers.

Comment 108

6 years ago
Not linking our main binaries (all the dlls etc) to msvcrt.lib or libcmt.lib (via /NODEFAULTLIB or whatever), and only linking them to a hypothetical mozcrt.dll will not work. There is code in msvcrt.lib and in libcmt.lib which must be linked into the exe or dll being built (after all, if the code in msvcrt.lib didn't need to be in msvcrt.lib, it would be in msvcr110.dll)

I have also found evidence that many CRT functions are different in libcmt.lib and msvcr110.dll and that the libcmt.lib versions of some of these functions need to live in the same module as the code mentioned in my other point above. (search for CRTDLL in the CRT source code to see some examples, there are also likely instances in parts of the CRT that Microsoft has chosen not to open source in VS11)

Simply creating a "mozcrt.dll" and exporting the functions from libcmt.lib wont work (based on my analysis thus far)

Comment 109

6 years ago
(In reply to Jim Mathies [:jimm] from comment #80)
> (In reply to jfwfreo from comment #78)
> > > * Linking statically to the crt is not a good option, for code size/memory
> > > usage reasons.  And even if we decide that we don't care about any of that,
> > > we have no evidence to suggest that we can effectively simulate the missing
> > > crt import entries with our hacky implementations on WinXP.
> > Does anyone know if its possible to link statically to the CRT right now?
> > (forget about code size/memory usage, does FF actually work if you do that?)
> > If its possible, I have some experiments I plan to run and some data I plan
> > to gather (I also need to figure out if its possible to hack the build
> > system so that I can get .map files generated for the various binaries)
> 
> We experimented with that, see comment 51. The overall growth of the install
> was pretty big, and the js lib had some issues. I'm sure we could work
> around the js problems but bloating the install by 6-7 megs compressed
> renders full static link for every component a non-option.
Ok, so can someone come up with patches or other stuff to do this (make all the FF dlls link statically to the CRT that is)?

The fact that it breaks the JS library doesn't matter to me because the experiments I am doing dont require a binary that I can actually run and the fact that it bloats the binary size doesn't matter to me because I wont be distributing these binaries, I just want to do some experiments on a static CRT build (including figuring out exactly which not-on-XP kernel functions we would be using in a statically linked CRT build and what causes us to pull those functions in)

Comment 110

6 years ago
Also can anyone tell me whether we can (legally or technically) extract specific .obj files from libcmt.lib and link only to those obj files and not to the rest of the library (I ask because if the idea I am currently leaning towards is possible, linking only to the .obj files we need to and not to the rest will ensure we get link errors on anything we haven't specifically marked as "ok to use")
(In reply to jfwfreo from comment #110)
> Also can anyone tell me whether we can (legally or technically) extract
> specific .obj files from libcmt.lib and link only to those obj files and not
> to the rest of the library (I ask because if the idea I am currently leaning
> towards is possible, linking only to the .obj files we need to and not to
> the rest will ensure we get link errors on anything we haven't specifically
> marked as "ok to use")

legally, I don't know, but technically, that's what we already do with msvcrt.lib/msvcprt.lib.
(Reporter)

Comment 112

6 years ago
(In reply to jfwfreo from comment #109)
> Ok, so can someone come up with patches or other stuff to do this (make all
> the FF dlls link statically to the CRT that is)?

I don't have those patches anymore but they aren't hard to put together. You have to add |USE_STATIC_LIBS = 1| to the make files in places where we build binaries and make sure to link to the right glue lib - 

http://mxr.mozilla.org/mozilla-central/search?string=XPCOM_STANDALONE_STATICRUNTIME_GLUE_LDOPTS

Comment 113

6 years ago
Here is my current plan and idea:
1.Build a static build of FF
2.Use this to identify exactly which obj files from the static CRT libraries (libcmy.lib etc) are required to make that happen
3.Having identified these obj files, we then split them up into 3 groups:
A.obj files we need to link into every binary for whatever reason
B.obj files we can safely put into our new mozcrt.dll and export/use (hopefully most of the obj files for things like strings, stdio, iostreams etc will be safe to use in mozcrt.dll)
and C.obj files we need to replace with code of our own (probably only a handful will apply here)

By only linking to the subset of objs we need to, if a code change means we pull in some new function, it doesn't get pulled in until we figure out whether it needs to be linked into every binary or just into mozcrt.dll

Comment 114

6 years ago
(In reply to Jim Mathies [:jimm] from comment #112)
> (In reply to jfwfreo from comment #109)
> > Ok, so can someone come up with patches or other stuff to do this (make all
> > the FF dlls link statically to the CRT that is)?
> 
> I don't have those patches anymore but they aren't hard to put together. You
> have to add |USE_STATIC_LIBS = 1| to the make files in places where we build
> binaries and make sure to link to the right glue lib - 
> 
> http://mxr.mozilla.org/mozilla-central/
> search?string=XPCOM_STANDALONE_STATICRUNTIME_GLUE_LDOPTS
I haven't messed with the Mozilla build system since the days when the full-suite was called Mozilla and some poor schmuck was trying to port the Windows version of Mozilla to the free-as-in-no-cost MingW compiler (remember this was before Microsoft started making its compiler available for free), can someone help me find the right bits of magic to use to make that complicated mess of configure scripts, makefiles and other stuff do what I want? :)
Just an FYI for anyone following along:
http://blogs.msdn.com/b/vcblog/archive/2012/06/15/10320645.aspx

"Later this fall, Microsoft will provide an update to Visual Studio 2012 that will enable C++ applications to target Windows XP"

I don't know how soon later this Fall is, but the fact that they said Fall seems promising.

Comment 116

6 years ago
Do we want to wait for an as-yet-unspecified solution at an as-yet-unspecified time or do we want to continue with this work?
We've contacted those-to-be-contacted to get clarification and since this could be the better solution it is likely best to hold off until we know more.

Comment 118

6 years ago
I have experimented with trying to identify which obj files we are using and not using (in a static build) and got lost somewhere in JS as it spat out 100s of "missing symbol blah" errors including large amounts of floating point stuff.
It doesn't help that large swathes of the floating point conversion code and transcendentals code and the exception handling and related code (setjmp etc) has never had its source released by Microsoft.

So I think the only sure-fire way to get this to happen is to wait for the Microsoft fix and do one of these in the mean time:
1.Keep building builds with VS2010 and not support Metro in official builds for now until the fix is available
2.Keep building builds with VS2010 and support Metro via what Google is doing or via a seperate metro-only DLL built with VS2012 (then migrate the whole project to VS2012 when the fix is available
or 3.Ship dual builds (Metro and XP) for now until the fix is available

Updated

6 years ago
Assignee: ehsan → nobody
Target Milestone: --- → Firefox 17
Version: Trunk → 16 Branch

Updated

6 years ago
Target Milestone: Firefox 17 → ---
Version: 16 Branch → Trunk
(Reporter)

Updated

6 years ago
Depends on: 794983

Comment 120

5 years ago
As another warning, VS2012 defaults to targeting SSE2 processors. Disable using /arch:IA32 if needed.
(Reporter)

Comment 121

5 years ago
No longer blocking this work, since we're back porting to vs2010.

MS recently released kludgy xp support for 12. Long term we might want to investigate this for release builds - 

http://blogs.msdn.com/b/vcblog/archive/2012/10/08/10357555.aspx

Until we find a solution here we're stuck using vs2010 for official releases.
No longer blocks: 737975

Comment 122

5 years ago
What exactly makes using VS 2012 difficult now that they have the XP support?
I dont see too much that's kludgy except possibly the different SDKs (one that supports Vista+ and one that supports XP+ but doesn't have any of the new Windows 8 APIs)

Comment 123

5 years ago
(In reply to comment #122)
> What exactly makes using VS 2012 difficult now that they have the XP support?
> I dont see too much that's kludgy except possibly the different SDKs (one that
> supports Vista+ and one that supports XP+ but doesn't have any of the new
> Windows 8 APIs)

Lack of Windows XP support in the dynamic runtime DLLs means that we have to statically link those runtime libraries, which is a non-starter for code size reasons.

Comment 124

5 years ago
From the blog post:
The Visual C++ 2012 Redistributables in “Microsoft Visual Studio 11.0\VC\redist\1033” have not been updated to include Windows XP support. For this preview release, please use static linking when targeting Windows XP or deploy the C++ runtime DLLs from “Microsoft Visual Studio 11.0\VC\redist\<architecture>” inside your executable’s installation folder."

Comment 125

5 years ago
The impression I get from the blog post is that once the final official SP1 release is launched, all the CRT DLLs (and presumably matching CRT redist installers) will be updated with the XP support.

Comment 126

5 years ago
(In reply to comment #125)
> The impression I get from the blog post is that once the final official SP1
> release is launched, all the CRT DLLs (and presumably matching CRT redist
> installers) will be updated with the XP support.

The blog post did not mention that as far as I can tell.

Comment 127

5 years ago
Created attachment 678736 [details] [diff] [review]
Build system edit to build for XP

I've successfully built Firefox Nightly (m-c) 19a1 with these edits to have it run on XP. In tandem, the only other edit was a change to the batch file to start the mozilla-build environment to force the SDK to 7.*, I'll upload that too.

Comment 128

5 years ago
Created attachment 678737 [details]
Changed batchfile (added MOZ_MAXWINSDK)

Comment 129

5 years ago
Oops, forgot to mention, this is done with VS2012 with the Update1 CTP4 installed. I expect that the final will have little if any differences regarding the VC++ compiler and CRT, probably more polishing the IDE changes that are also part of the update.

Comment 130

5 years ago
Hmm, what mozconfig did you use?  Are you linking against the runtime libraries dynamically?

Comment 131

5 years ago
I used one of my own, not a default suppl(In reply to Ehsan Akhgari [:ehsan] from comment #130)
> Hmm, what mozconfig did you use?  Are you linking against the runtime
> libraries dynamically?

I used one of my own, not a default supplied one. But yes, linking dynamically. .mozconfig options used:

ac_add_options --enable-update-packaging
mk_add_options MOZ_CO_PROJECT=browser
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-O2 -GFLs -GS-"
ac_add_options --enable-jemalloc
ac_add_options --disable-debug
ac_add_options --disable-tests
ac_add_options --disable-mochitests
ac_add_options --enable-strip
ac_add_options --disable-crashreporter
ac_add_options --disable-accessibility
ac_add_options --disable-parental-controls
ac_add_options --disable-maintenance-service
ac_add_options --disable-windows-mobile-components
ac_add_options --disable-metro
ac_add_options --disable-activex
ac_add_options --disable-activex-scripting
ac_add_options --disable-logging
ac_add_options --disable-necko-wifi

Comment 132

5 years ago
(In reply to jfwfreo from comment #125)
> The impression I get from the blog post is that once the final official SP1
> release is launched, all the CRT DLLs (and presumably matching CRT redist
> installers) will be updated with the XP support.

That happened now: http://www.microsoft.com/en-us/download/details.aspx?id=30679
"Supported operating systems: …, Windows XP"

Comment 133

5 years ago
(In reply to comment #132)
> (In reply to jfwfreo from comment #125)
> > The impression I get from the blog post is that once the final official SP1
> > release is launched, all the CRT DLLs (and presumably matching CRT redist
> > installers) will be updated with the XP support.
> 
> That happened now:
> http://www.microsoft.com/en-us/download/details.aspx?id=30679
> "Supported operating systems: …, Windows XP"

It would be great if someone can build with this and test it on Windows XP.

Comment 134

5 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #133)
> (In reply to comment #132)
> > (In reply to jfwfreo from comment #125)
> > > The impression I get from the blog post is that once the final official SP1
> > > release is launched, all the CRT DLLs (and presumably matching CRT redist
> > > installers) will be updated with the XP support.
> > 
> > That happened now:
> > http://www.microsoft.com/en-us/download/details.aspx?id=30679
> > "Supported operating systems: …, Windows XP"
> 
> It would be great if someone can build with this and test it on Windows XP.

Have tested it.

And we also need to add -SUBSYSTEM:WINDOWS,5.01

Comment 135

5 years ago
Great!

Comment 136

5 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #135)
> Great!

Unfortunately, VC2012 noPGO build is slight faster than VC2010 build, but VC2012 PGO build is much slower than VC2010 build by 10%-20% (Sunspider/V8/Kraken/Peacekeeper/...)
(In reply to xunxun from comment #136)
> (In reply to Ehsan Akhgari [:ehsan] from comment #135)
> > Great!
> 
> Unfortunately, VC2012 noPGO build is slight faster than VC2010 build, but
> VC2012 PGO build is much slower than VC2010 build by 10%-20%
> (Sunspider/V8/Kraken/Peacekeeper/...)

How did you compare? Were the VC2010 PGO and VC2012 PGO builds done on the same (class of) hardware?

Comment 138

5 years ago
(In reply to Mike Hommey [:glandium] from comment #137)
> (In reply to xunxun from comment #136)
> > (In reply to Ehsan Akhgari [:ehsan] from comment #135)
> > > Great!
> > 
> > Unfortunately, VC2012 noPGO build is slight faster than VC2010 build, but
> > VC2012 PGO build is much slower than VC2010 build by 10%-20%
> > (Sunspider/V8/Kraken/Peacekeeper/...)
> 
> How did you compare? Were the VC2010 PGO and VC2012 PGO builds done on the
> same (class of) hardware?

Of course the same hardware/OS. My Y450 notebook.

Comment 139

5 years ago
(In reply to Mike Hommey [:glandium] from comment #137)
> (In reply to xunxun from comment #136)
> > (In reply to Ehsan Akhgari [:ehsan] from comment #135)
> > > Great!
> > 
> > Unfortunately, VC2012 noPGO build is slight faster than VC2010 build, but
> > VC2012 PGO build is much slower than VC2010 build by 10%-20%
> > (Sunspider/V8/Kraken/Peacekeeper/...)
> 
> How did you compare? Were the VC2010 PGO and VC2012 PGO builds done on the
> same (class of) hardware?

You can see the Microsoft bug: https://connect.microsoft.com/VisualStudio/feedback/details/735695/pgo-is-too-conservative-in-vc11-beta

Comment 140

5 years ago
(In reply to xunxun from comment #139)
> You can see the Microsoft bug:

According to http://social.msdn.microsoft.com/Forums/en-US/vstsprofiler/thread/cf71f45e-1d07-4b57-aede-0f1b6c0c987b#8f8c0529-608e-40b5-9844-5efbe93ce9ab the bug report you linked to is a "feedback for VS2010" and not VS12 and has therefore been closed.

Comment 141

5 years ago
(In reply to [Baboo] from comment #140)
> (In reply to xunxun from comment #139)
> > You can see the Microsoft bug:
> 
> According to
> http://social.msdn.microsoft.com/Forums/en-US/vstsprofiler/thread/cf71f45e-
> 1d07-4b57-aede-0f1b6c0c987b#8f8c0529-608e-40b5-9844-5efbe93ce9ab the bug
> report you linked to is a "feedback for VS2010" and not VS12 and has
> therefore been closed.

No, it's for VC11 Beta. I will open a new bug on Microsoft when I have time.

Comment 142

5 years ago
(In reply to xunxun from comment #141)
> No, it's for VC11 Beta. I will open a new bug on Microsoft when I have time.
Any update?

BTW, I wonder how the VC11 builds now compare to mozilla-inbound since the PGO has been disabled there for some components because of bug 832992. Maybe a less aggressive PGO approach has also its upsides and that might have been why MS changed it.

Comment 143

5 years ago
Personally, I found out that you don't even need the older SDK with VS2012, you can use 8.0; meaning with VS2012 with KB2707250 (Update1) installed, you'd only need -SUBSYSTEM:WINDOWS,5.01 in the linker command line to build for XP compatibility, everything else just works. I'm still using the DirectX SDK Jun2010 aside from the built-in SDKs, but that's all.

Building with VS2012 seems to link much faster than 2010, and I haven't run into any resource issues with PGO.
(Reporter)

Comment 144

5 years ago
Created attachment 705834 [details] [diff] [review]
xp compat build

Comment 145

5 years ago
(In reply to comment #144)
> Created attachment 705834 [details] [diff] [review]
>   --> https://bugzilla.mozilla.org/attachment.cgi?id=705834&action=edit
> xp compat build

Nice!  Can we take this patch?  I don't think that it should negatively impact the VS2010 builds...
Comment on attachment 705834 [details] [diff] [review]
xp compat build

Review of attachment 705834 [details] [diff] [review]:
-----------------------------------------------------------------

The nspr and nss changes will have to be separated.

::: configure.in
@@ +2135,5 @@
>          MKSHLIB='$(LD) -NOLOGO -DLL -OUT:$@ -PDB:$(LINK_PDBFILE) $(DSO_LDOPTS)'
>          MKCSHLIB='$(LD) -NOLOGO -DLL -OUT:$@ -PDB:$(LINK_PDBFILE) $(DSO_LDOPTS)'
>          MKSHLIB_FORCE_ALL=
>          MKSHLIB_UNFORCE_ALL=
> +        DSO_LDOPTS=

Do you really mean to remove the flag from link.exe calls? (the config.mk change doesn't quite replace it)
(Reporter)

Comment 147

5 years ago
(In reply to Mike Hommey [:glandium] from comment #146)
> Comment on attachment 705834 [details] [diff] [review]
> xp compat build
> 
> Review of attachment 705834 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> The nspr and nss changes will have to be separated.
> 
> ::: configure.in
> @@ +2135,5 @@
> >          MKSHLIB='$(LD) -NOLOGO -DLL -OUT:$@ -PDB:$(LINK_PDBFILE) $(DSO_LDOPTS)'
> >          MKCSHLIB='$(LD) -NOLOGO -DLL -OUT:$@ -PDB:$(LINK_PDBFILE) $(DSO_LDOPTS)'
> >          MKSHLIB_FORCE_ALL=
> >          MKSHLIB_UNFORCE_ALL=
> > +        DSO_LDOPTS=
> 
> Do you really mean to remove the flag from link.exe calls? (the config.mk
> change doesn't quite replace it)

I'm sure there are different ways to do it. If you leave the subsystem directive in DSO_LDOPTS and skip setting LDFLAGS in config.mk some exes don't get the subsystem switch, and dlls get subsystem windows. The dll thing probably isn't a big deal, but the default should really be console for those. Test exes though would need to run on xp, so they have to have the subsystem directive. Using LDFLAGS vs DSO_LDOPTS seemed to get everything right.

Comment 148

5 years ago
vs2012 update 1 has added support for targeting xp with c++ compiler
is it necessary to continue working on this bug given that metro builds
of firefox will need to move to vs2012 ?

Comment 149

5 years ago
Yes it is necessary to continue with this, this bug is about making the changes to the build system so that when you compile Firefox (and other Mozilla projects) with Visual Studio 2012 Update 1 (with the XP support), the binaries that are produced will run on Windows XP.

Comment 150

5 years ago
I think this bug has served its purpose.  We now know all we need to know.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Comment hidden (spam)

Comment 152

2 years ago
Maybe this could help http://www.gadgetsupersite.com/0xc00007b-error-fix/
Comment hidden (spam)
Comment hidden (spam)
Comment hidden (spam)
You need to log in before you can comment on or make changes to this bug.