1274659 - (win64-migration) Migrate 32-bit Firefox (WOW64) users to 64-bit Firefox (Win64)

Status: RESOLVED FIXED

(Firefox :: General, defect, P3)

Description

[Eric Rahm [:erahm]]

There was a rather lengthy discussion [1] on whether or not it's a good idea to update users of 32-bit Firefox a 64-bit version of Windows to a 64-bit version of Firefox.

The general benefits proposed are:
  - Performance: Guaranteed SSE2 support
  - Stability: No more OOMs due to address space fragmentation (this is a huge win)

Concerns voiced:

*Memory usage* 64-bit builds use more physical memory.

While this is true (probably about 20% looking at AWSY measurements), eliminating all OOMs related to address space fragmentation and exhaustion is a much higher priority than the potentially degraded performance from swapping.

Note: these are OOMs that happen even if there is plenty of physical memory left. The memshrink group has looked at this issue extensively and has concluded that the possible increase in physical memory usage is acceptable and amount of RAM a user has should not be taken into consideration.

*NPAPI plug-ins* If a user has 32-bit plug-ins we may not want to upgrade them.

- Flash and Silverlight both provide 64-bit plug-ins
- Once we drop support for NPAPI this becomes a non-issue

There were additional concerns that there are issues with our ability to sandbox the 64-bit version of flash. We can discuss that further in the comments.

[1] https://groups.google.com/forum/#!topic/mozilla.dev.platform/ANiu9qdUIwQ

Comment 1

[Robert Strong (they/them - no direct email)]

From the app update technical side (not advocating for or against) of this:
1. We won't be able to change the install directory due to not being able to update shortcuts for other users on the same system along with a few other reasons.
2. Minimal testing of running Firefox x64 in a directory under C:\Program Files (x86)\ was performed a few years ago without any issues being found.
3. To accomplish this, releng would need to configure balrog to serve the update. No app update client changes should be required.

Comment 2

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #1)
> From the app update technical side (not advocating for or against) of this:
> 1. We won't be able to change the install directory due to not being able to
> update shortcuts for other users on the same system along with a few other
> reasons.
Could we install in C:\Program Files\ and put a stub in c:\Program Files (x86)\firefox.exe that just launches C:\Program Files\firefox.exe to catch those shortcuts?

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #2)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #1)
> > From the app update technical side (not advocating for or against) of this:
> > 1. We won't be able to change the install directory due to not being able to
> > update shortcuts for other users on the same system along with a few other
> > reasons.
> Could we install in C:\Program Files\ and put a stub in c:\Program Files
> (x86)\firefox.exe that just launches C:\Program Files\firefox.exe to catch
> those shortcuts?

Starting with Vista, we could even do a symbolic link.

Comment 4

[Robert Strong (they/them - no direct email)]

The updater intentionally does not touch anything on the filesystem outside of the install directory to mitigate exploits.

We could possibly do this in a synchronously in the post update code. Considering that this is just a cosmetic issue I really don't think we should add this type of complexity.

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Eric Rahm [:erahm] from comment #0)
> *NPAPI plug-ins* If a user has 32-bit plug-ins we may not want to upgrade
> them.
> 
> - Flash and Silverlight both provide 64-bit plug-ins
> - Once we drop support for NPAPI this becomes a non-issue

Bug 1269807 is the meta-bug tracking the removal of NPAPI plugins other than Flash.

Comment 6

[Benjamin Smedberg]

This is not a scheduled part of the win64 deployment plan, so I'm on the fence about whether to mark this P5 or actually mark it INCOMPLETE. The staged rollout of win64 will focus first on new users via the installer, before we attempt to update existing users.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #6)
> This is not a scheduled part of the win64 deployment plan, so I'm on the
> fence about whether to mark this P5 or actually mark it INCOMPLETE. The
> staged rollout of win64 will focus first on new users via the installer,
> before we attempt to update existing users.

To be clear: you explicitly do NOT want to move any users currently running the 32-bit Firefox on a 64-bit Windows to a 64-bit Firefox?

Comment 8

[Benjamin Smedberg]

That is correct. This is effectively WONTFIX for a while.

Comment 9

[Eric Rahm [:erahm]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #8)
> That is correct. This is effectively WONTFIX for a while.

By *far* the top crasher on 46.0.1 is OOM small, virtually all of these are on 32-bit builds running on Windows. Including "OOM | unknown" from the top 15 we're looking at 10% of all crashes.

If we're serious about quality and project uptime this should be a much higher priority.

Comment 10

[Chris Peterson [:cpeterson]]

rstrong recommends auto-upgrading in place ("pave over" install) in the 32-bit "C:\Program Files (x86)" directory instead of moving files to the 64-bit "C:\Program Files" directory. This would avoid breaking any user shortcuts that point to the original "C:\Program Files (x86)" directory.

The Win64-aware stub installer (bug 797208) should place new 64-bit installs in the 64-bit "C:\Program Files" directory, allowing 64-bit side-by-side with any existing 32-bit installs in "C:\Program Files (x86)".

Comment 11

[Masatoshi Kimura [:emk]]

Don't forget using 32-bit Firefox is a workaround for bug 1271398.

Comment 12

[Will]

If we have the opportunity to eliminate those top crashers for 'free', it would be crazy to pass that up. We cannot afford to pass up that opportunity, it would be a mistake. Why would all that work and resources be put into making 64bit for Windows if there was no intention of moving people over to it?

And let's not forget about the security benefits, including this (link below), which for some reason never gets brought up http://news.softpedia.com/news/windows-legacy-layer-used-to-bypass-emet-security-measures-495691.shtml

Comment 13

[Chris Peterson [:cpeterson]]

mbest recommends that we migrate 32-bit Firefox users to 64-bit before making Flash click-to-play.

Comment 14

[mirh]

(In reply to Eric Rahm [:erahm] from comment #0)
> *Memory usage* 64-bit builds use more physical memory.
> 
> While this is true (probably about 20% looking at AWSY measurements),
> eliminating all OOMs related to address space fragmentation and exhaustion
> is a much higher priority than the potentially degraded performance from
> swapping.

Putting aside I can generally agree 32-64 bit memory difference is around 20%, at least in bug 1282322 we are talking of a minimum delta of 40%.

Comment 15

[Will]

Google started moving eligible machines to 64-bit Chrome today and there is already an issue https://www.reddit.com/r/chrome/comments/68x0zd/help_aww_snap_after_upgrading_to_580302996/

Comment 16

[Chris Peterson [:cpeterson]]

(In reply to Will from comment #15)
> Google started moving eligible machines to 64-bit Chrome today and there is
> already an issue
> https://www.reddit.com/r/chrome/comments/68x0zd/
> help_aww_snap_after_upgrading_to_580302996/

Thanks, Will. That is very interesting! I will follow up with our Firefox enterprise users about testing Citrix XenApp.

Comment 17

[Masatoshi Kimura [:emk]]

(In reply to Chris Peterson [:cpeterson] from comment #16)
> Thanks, Will. That is very interesting! I will follow up with our Firefox
> enterprise users about testing Citrix XenApp.

Is it still a problem despite that we dropped support for NPAPI except Flash?

Comment 18

[Chris Peterson [:cpeterson]]

(In reply to Masatoshi Kimura [:emk] from comment #17)
> Is it still a problem despite that we dropped support for NPAPI except Flash?

AFAIK, this is not a plugin problem. According to the following Chrome KB article, 64-bit Chrome crashes when run as a Citrix remote desktop application due to some Citrix hooks injected into the Chrome process:

https://support.google.com/chrome/a/answer/7380899

Comment 19

[Marco Castelluccio [:marco]]

Perhaps we can blocklist Citrix.

Comment 20

[Will]

I don't know what the relationship with Google Chrome team is like but would they give any advice or recommendations so Firefox can avoid any issues when updating 32-bit machines to 64-bit Firefox? Maybe someone can reach out to them?

Comment 21

[Chris Peterson [:cpeterson]]

(In reply to Will from comment #20)
> I don't know what the relationship with Google Chrome team is like but would
> they give any advice or recommendations so Firefox can avoid any issues when
> updating 32-bit machines to 64-bit Firefox? Maybe someone can reach out to
> them?

We are closely watching Google's rollout of 64-bit Chrome to their 32-bit users. For example, that is how we found the cause because Citrix crash bug 1359443. :)

Comment 22

[Will]

Yes, I notice quite a few issues on reddit too, such as Chrome not even starting up (this is different from the citrix problem) and re-installing the 32-bit version fixed it. 
I don't know if this is already being considered but whenever 64-bit is deemed ready (version 55?) there should be a bit of a push via blog posts, the firefox twitter account etc. to get technical users to opt-in to 64-bit to make sure there are no unexpected issus. Is there an easy way to get users to opt-in to 64-bit? Such as via the update window?

Comment 23

[Will]

Attachment: 64bitoptin.png

I just made this terrible thing very quickly. Something like this would be ideal I think

Comment 24

[Chris Peterson [:cpeterson]]

(In reply to Will from comment #22)
> I don't know if this is already being considered but whenever 64-bit is
> deemed ready (version 55?) there should be a bit of a push via blog posts,
> the firefox twitter account etc. to get technical users to opt-in to 64-bit
> to make sure there are no unexpected issues. Is there an easy way to get
> users to opt-in to 64-bit? Such as via the update window?

Yeah, we plan to publish some blog posts promoting 64-bit Firefox 55. With the Firefox 56 release, we plan to automatically migrate existing eligible 32-bit users to 64-bit. Until the automatic migration, users need to manually re-run the Firefox installer to get 64-bit.

In fact, 79% of Nightly channel users and 7% of Release channel users on Windows are already running 64-bit Firefox, so we have pretty good test coverage. :)

We try to keep our release schedule on this wiki page up to date:

https://wiki.mozilla.org/Firefox/Win64#Schedule

Comment 25

[Will]

I'm not concerned that there's any issues with 64-bit Firefox itself, it's the act of upgrading 32-bit to 64-bit that I think there could be issues, such as Firefox not even starting up (like what's been happening to Chrome). Maybe in that blog post asking more technical users to opt-in and report any issues could be helpful. Google didn't do this and it caused some problems.

Comment 26

[mirh]

(In reply to Chris Peterson [:cpeterson] from bug 1366860, comment #1)
> For now, we don't need a minimum memory check in the 64-bit full installer.
> If a user went out of their way to download the 64-bit full installer, we
> should probably give it to them.

A really, really, relevant check to add for 64-bit _migration_ would be instead memory consumption average/Xpercentile. 
Or perhaps more specifically if this multiplied 1.5x is less than 50% of installed RAM. 

Ie: my use-case is that I have 4GB installed, and FF on *average* uses 2. 
(Unless bug 1125557 lands anytime soon) I think it's pretty glaring that I'd be screwed with x64. 

Your call on where to put the boundary then.

Comment 28

[Worcester12345]

It seems to be that new updates are migrating to 64 bit without asking. Is this bug done, then?

Comment 29

[Chris Peterson [:cpeterson]]

(In reply to Worcester12345 from comment #28)
> It seems to be that new updates are migrating to 64 bit without asking. Is
> this bug done, then?

I'd like to keep this meta bug open until we have unthrottled the migration for 100% of eligible users.

We started migrating 1% of eligible 32-bit Firefox 56.0 users to 64-bit Firefox 56.0.1 on October 9. We unthrottled migration to 5% on October 12. If all goes well, we should be fully unthrottled in a couple weeks. :)

Comment 30

[Emanuel Hoogeveen [:ehoogeveen]]

Sounds like you didn't mean to close this yet?

Comment 31

[Worcester12345]

(In reply to Chris Peterson [:cpeterson] from comment #29)
> (In reply to Worcester12345 from comment #28)
> > It seems to be that new updates are migrating to 64 bit without asking. Is
> > this bug done, then?
> 
> I'd like to keep this meta bug open until we have unthrottled the migration
> for 100% of eligible users.
> 
> We started migrating 1% of eligible 32-bit Firefox 56.0 users to 64-bit
> Firefox 56.0.1 on October 9. We unthrottled migration to 5% on October 12.
> If all goes well, we should be fully unthrottled in a couple weeks. :)

A couple weeks have gone by. Close now?

Also, do you have a link to the stats?

Thanks.

Comment 32

[sjw]

(In reply to Worcester12345 from comment #31)
> Also, do you have a link to the stats?

The Hardware Report has some:
https://hardware.metrics.mozilla.com/#detail-browser-share-32-64

Comment 33

[Romain Testard [:RT]]

The hardware report graph shows the split across all platforms, Windows only data is accessible here: https://sql.telemetry.mozilla.org/dashboard/win64-release-criteria---release

Status:
- 66.6% of Win7+ release users are now on 64 bit Firefox
- 19.2% of Win7+ users are on 32 bit OS, they won't ever get migrated
- 14.2% of Win7+ release users are on 64 bit OS with 32 bit Firefox
--- 5.2% of these users have 2048 Gb RAM or less, therefore we can say that 13.5% of Win7+ release users are left to be migrated; their migration will although now be slow (we're addressing the long tail) and we hope to get most users migrated by end of the year.

Comment 34

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:Dolske, maybe it's time to close this bug?

Comment 35

[Chris Peterson [:cpeterson]]

This migration is complete. We enabled migration for 100% eligible 32-bit Firefox 56.0 users to 64-bit 56.0.1 on 2018-10-23.

https://wiki.mozilla.org/Firefox/Win64#History

Comment 36

[Worcester12345]

(In reply to Eric Rahm [:erahm] from comment #0)

There was a rather lengthy discussion [1] on whether or not it's a good idea
to update users of 32-bit Firefox a 64-bit version of Windows to a 64-bit
version of Firefox.

The general benefits proposed are:
...

• Stability: No more OOMs due to address space fragmentation (this is a
  huge win)
  ...

1. Not so sure these "OOMs" stopped.

2. Need a corresponding bug to this one for Thunderbird.

Comment 37

[Andrew McCreight [:mccr8]]

(In reply to Worcester12345 from comment #36)

1. Not so sure these "OOMs" stopped.

Anecdotally, most of the OOM crashes I've seen lately have been due to small available page files, not address fragmentation.

Comment 38

[Worcester12345]

(In reply to Worcester12345 from comment #36)

2. Need a corresponding bug to this one for Thunderbird.

Someone want to open a new bug and move it forward?