Closed Bug 1274659 (win64-migration) Opened 8 years ago Closed 6 years ago

Migrate 32-bit Firefox (WOW64) users to 64-bit Firefox (Win64)

Categories

(Firefox :: General, defect, P3)

x86_64
Windows
defect

Tracking

()

RESOLVED FIXED
Tracking Status
firefox-esr52 --- unaffected
firefox56 --- fixed
firefox57 --- unaffected
firefox58 --- unaffected

People

(Reporter: erahm, Unassigned)

References

(Depends on 2 open bugs)

Details

(Keywords: 64bit)

Attachments

(1 file)

There was a rather lengthy discussion [1] on whether or not it's a good idea to update users of 32-bit Firefox a 64-bit version of Windows to a 64-bit version of Firefox.

The general benefits proposed are:
  - Performance: Guaranteed SSE2 support
  - Stability: No more OOMs due to address space fragmentation (this is a huge win)

Concerns voiced:

*Memory usage* 64-bit builds use more physical memory.

While this is true (probably about 20% looking at AWSY measurements), eliminating all OOMs related to address space fragmentation and exhaustion is a much higher priority than the potentially degraded performance from swapping.

Note: these are OOMs that happen even if there is plenty of physical memory left. The memshrink group has looked at this issue extensively and has concluded that the possible increase in physical memory usage is acceptable and amount of RAM a user has should not be taken into consideration.

*NPAPI plug-ins* If a user has 32-bit plug-ins we may not want to upgrade them.

- Flash and Silverlight both provide 64-bit plug-ins
- Once we drop support for NPAPI this becomes a non-issue

There were additional concerns that there are issues with our ability to sandbox the 64-bit version of flash. We can discuss that further in the comments.

[1] https://groups.google.com/forum/#!topic/mozilla.dev.platform/ANiu9qdUIwQ
From the app update technical side (not advocating for or against) of this:
1. We won't be able to change the install directory due to not being able to update shortcuts for other users on the same system along with a few other reasons.
2. Minimal testing of running Firefox x64 in a directory under C:\Program Files (x86)\ was performed a few years ago without any issues being found.
3. To accomplish this, releng would need to configure balrog to serve the update. No app update client changes should be required.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #1)
> From the app update technical side (not advocating for or against) of this:
> 1. We won't be able to change the install directory due to not being able to
> update shortcuts for other users on the same system along with a few other
> reasons.
Could we install in C:\Program Files\ and put a stub in c:\Program Files (x86)\firefox.exe that just launches C:\Program Files\firefox.exe to catch those shortcuts?
(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #2)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #1)
> > From the app update technical side (not advocating for or against) of this:
> > 1. We won't be able to change the install directory due to not being able to
> > update shortcuts for other users on the same system along with a few other
> > reasons.
> Could we install in C:\Program Files\ and put a stub in c:\Program Files
> (x86)\firefox.exe that just launches C:\Program Files\firefox.exe to catch
> those shortcuts?

Starting with Vista, we could even do a symbolic link.
The updater intentionally does not touch anything on the filesystem outside of the install directory to mitigate exploits.

We could possibly do this in a synchronously in the post update code. Considering that this is just a cosmetic issue I really don't think we should add this type of complexity.
(In reply to Eric Rahm [:erahm] from comment #0)
> *NPAPI plug-ins* If a user has 32-bit plug-ins we may not want to upgrade
> them.
> 
> - Flash and Silverlight both provide 64-bit plug-ins
> - Once we drop support for NPAPI this becomes a non-issue

Bug 1269807 is the meta-bug tracking the removal of NPAPI plugins other than Flash.
Depends on: npapi-eol
This is not a scheduled part of the win64 deployment plan, so I'm on the fence about whether to mark this P5 or actually mark it INCOMPLETE. The staged rollout of win64 will focus first on new users via the installer, before we attempt to update existing users.
Priority: -- → P5
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #6)
> This is not a scheduled part of the win64 deployment plan, so I'm on the
> fence about whether to mark this P5 or actually mark it INCOMPLETE. The
> staged rollout of win64 will focus first on new users via the installer,
> before we attempt to update existing users.

To be clear: you explicitly do NOT want to move any users currently running the 32-bit Firefox on a 64-bit Windows to a 64-bit Firefox?
Flags: needinfo?(benjamin)
That is correct. This is effectively WONTFIX for a while.
Flags: needinfo?(benjamin)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #8)
> That is correct. This is effectively WONTFIX for a while.

By *far* the top crasher on 46.0.1 is OOM small, virtually all of these are on 32-bit builds running on Windows. Including "OOM | unknown" from the top 15 we're looking at 10% of all crashes.

If we're serious about quality and project uptime this should be a much higher priority.
rstrong recommends auto-upgrading in place ("pave over" install) in the 32-bit "C:\Program Files (x86)" directory instead of moving files to the 64-bit "C:\Program Files" directory. This would avoid breaking any user shortcuts that point to the original "C:\Program Files (x86)" directory.

The Win64-aware stub installer (bug 797208) should place new 64-bit installs in the 64-bit "C:\Program Files" directory, allowing 64-bit side-by-side with any existing 32-bit installs in "C:\Program Files (x86)".
Don't forget using 32-bit Firefox is a workaround for bug 1271398.
Depends on: 1271398
No longer blocks: support-win64
If we have the opportunity to eliminate those top crashers for 'free', it would be crazy to pass that up. We cannot afford to pass up that opportunity, it would be a mistake. Why would all that work and resources be put into making 64bit for Windows if there was no intention of moving people over to it?

And let's not forget about the security benefits, including this (link below), which for some reason never gets brought up http://news.softpedia.com/news/windows-legacy-layer-used-to-bypass-emet-security-measures-495691.shtml
Keywords: 64bit
Summary: Update all WOW64 users to 64-bit Firefox → Update 32-bit Firefox (WOW64) users to 64-bit Firefox
mbest recommends that we migrate 32-bit Firefox users to 64-bit before making Flash click-to-play.
Priority: P5 → P3
Summary: Update 32-bit Firefox (WOW64) users to 64-bit Firefox → Migrate 32-bit Firefox (WOW64) users to 64-bit Firefox (Win64)
Depends on: 1318039
No longer depends on: 1318039
Depends on: 1324617
Depends on: 1324866
No longer depends on: 1324617
Alias: wow64
Blocks: Quantum
Depends on: 1323750
Depends on: 1345649
(In reply to Eric Rahm [:erahm] from comment #0)
> *Memory usage* 64-bit builds use more physical memory.
> 
> While this is true (probably about 20% looking at AWSY measurements),
> eliminating all OOMs related to address space fragmentation and exhaustion
> is a much higher priority than the potentially degraded performance from
> swapping.

Putting aside I can generally agree 32-64 bit memory difference is around 20%, at least in bug 1282322 we are talking of a minimum delta of 40%.
Google started moving eligible machines to 64-bit Chrome today and there is already an issue https://www.reddit.com/r/chrome/comments/68x0zd/help_aww_snap_after_upgrading_to_580302996/
(In reply to Will from comment #15)
> Google started moving eligible machines to 64-bit Chrome today and there is
> already an issue
> https://www.reddit.com/r/chrome/comments/68x0zd/
> help_aww_snap_after_upgrading_to_580302996/

Thanks, Will. That is very interesting! I will follow up with our Firefox enterprise users about testing Citrix XenApp.
(In reply to Chris Peterson [:cpeterson] from comment #16)
> Thanks, Will. That is very interesting! I will follow up with our Firefox
> enterprise users about testing Citrix XenApp.

Is it still a problem despite that we dropped support for NPAPI except Flash?
(In reply to Masatoshi Kimura [:emk] from comment #17)
> Is it still a problem despite that we dropped support for NPAPI except Flash?

AFAIK, this is not a plugin problem. According to the following Chrome KB article, 64-bit Chrome crashes when run as a Citrix remote desktop application due to some Citrix hooks injected into the Chrome process:

https://support.google.com/chrome/a/answer/7380899
Perhaps we can blocklist Citrix.
I don't know what the relationship with Google Chrome team is like but would they give any advice or recommendations so Firefox can avoid any issues when updating 32-bit machines to 64-bit Firefox? Maybe someone can reach out to them?
(In reply to Will from comment #20)
> I don't know what the relationship with Google Chrome team is like but would
> they give any advice or recommendations so Firefox can avoid any issues when
> updating 32-bit machines to 64-bit Firefox? Maybe someone can reach out to
> them?

We are closely watching Google's rollout of 64-bit Chrome to their 32-bit users. For example, that is how we found the cause because Citrix crash bug 1359443. :)
Alias: wow64 → wow64-migration
Depends on: 1359443
Yes, I notice quite a few issues on reddit too, such as Chrome not even starting up (this is different from the citrix problem) and re-installing the 32-bit version fixed it. 
I don't know if this is already being considered but whenever 64-bit is deemed ready (version 55?) there should be a bit of a push via blog posts, the firefox twitter account etc. to get technical users to opt-in to 64-bit to make sure there are no unexpected issus. Is there an easy way to get users to opt-in to 64-bit? Such as via the update window?
Attached image 64bitoptin.png
I just made this terrible thing very quickly. Something like this would be ideal I think
(In reply to Will from comment #22)
> I don't know if this is already being considered but whenever 64-bit is
> deemed ready (version 55?) there should be a bit of a push via blog posts,
> the firefox twitter account etc. to get technical users to opt-in to 64-bit
> to make sure there are no unexpected issues. Is there an easy way to get
> users to opt-in to 64-bit? Such as via the update window?

Yeah, we plan to publish some blog posts promoting 64-bit Firefox 55. With the Firefox 56 release, we plan to automatically migrate existing eligible 32-bit users to 64-bit. Until the automatic migration, users need to manually re-run the Firefox installer to get 64-bit.

In fact, 79% of Nightly channel users and 7% of Release channel users on Windows are already running 64-bit Firefox, so we have pretty good test coverage. :)

We try to keep our release schedule on this wiki page up to date:

https://wiki.mozilla.org/Firefox/Win64#Schedule
Alias: wow64-migration → win64-migration
I'm not concerned that there's any issues with 64-bit Firefox itself, it's the act of upgrading 32-bit to 64-bit that I think there could be issues, such as Firefox not even starting up (like what's been happening to Chrome). Maybe in that blog post asking more technical users to opt-in and report any issues could be helpful. Google didn't do this and it caused some problems.
(In reply to Chris Peterson [:cpeterson] from bug 1366860, comment #1)
> For now, we don't need a minimum memory check in the 64-bit full installer.
> If a user went out of their way to download the 64-bit full installer, we
> should probably give it to them.

A really, really, relevant check to add for 64-bit _migration_ would be instead memory consumption average/Xpercentile. 
Or perhaps more specifically if this multiplied 1.5x is less than 50% of installed RAM. 

Ie: my use-case is that I have 4GB installed, and FF on *average* uses 2. 
(Unless bug 1125557 lands anytime soon) I think it's pretty glaring that I'd be screwed with x64. 

Your call on where to put the boundary then.
Depends on: 1366917
Depends on: 1386176
Depends on: 1367054
Depends on: 1393447
Depends on: 1347867
Depends on: 1268470
Depends on: 1369361
Depends on: 1372823
Depends on: 1397301
Depends on: 1396728
Depends on: 1403353
No longer depends on: 1397750
Depends on: 1405681
Depends on: 1407015
Depends on: 1387466
Depends on: 1407147
Depends on: 1407337
No longer depends on: 1407015
Depends on: 1407712
Depends on: 1407736
No longer depends on: 1407337
It seems to be that new updates are migrating to 64 bit without asking. Is this bug done, then?
(In reply to Worcester12345 from comment #28)
> It seems to be that new updates are migrating to 64 bit without asking. Is
> this bug done, then?

I'd like to keep this meta bug open until we have unthrottled the migration for 100% of eligible users.

We started migrating 1% of eligible 32-bit Firefox 56.0 users to 64-bit Firefox 56.0.1 on October 9. We unthrottled migration to 5% on October 12. If all goes well, we should be fully unthrottled in a couple weeks. :)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Sounds like you didn't mean to close this yet?
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Depends on: 1409155
Depends on: 1345988
Depends on: 1409275
Depends on: 1397339
Depends on: 1411428
Depends on: 1413428
Depends on: 1413809
(In reply to Chris Peterson [:cpeterson] from comment #29)
> (In reply to Worcester12345 from comment #28)
> > It seems to be that new updates are migrating to 64 bit without asking. Is
> > this bug done, then?
> 
> I'd like to keep this meta bug open until we have unthrottled the migration
> for 100% of eligible users.
> 
> We started migrating 1% of eligible 32-bit Firefox 56.0 users to 64-bit
> Firefox 56.0.1 on October 9. We unthrottled migration to 5% on October 12.
> If all goes well, we should be fully unthrottled in a couple weeks. :)

A couple weeks have gone by. Close now?

Also, do you have a link to the stats?

Thanks.
(In reply to Worcester12345 from comment #31)
> Also, do you have a link to the stats?

The Hardware Report has some:
https://hardware.metrics.mozilla.com/#detail-browser-share-32-64
The hardware report graph shows the split across all platforms, Windows only data is accessible here: https://sql.telemetry.mozilla.org/dashboard/win64-release-criteria---release

Status:
- 66.6% of Win7+ release users are now on 64 bit Firefox
- 19.2% of Win7+ users are on 32 bit OS, they won't ever get migrated
- 14.2% of Win7+ release users are on 64 bit OS with 32 bit Firefox
--- 5.2% of these users have 2048 Gb RAM or less, therefore we can say that 13.5% of Win7+ release users are left to be migrated; their migration will although now be slow (we're addressing the long tail) and we hope to get most users migrated by end of the year.
Depends on: amdbug
Depends on: 1420251
Depends on: 1423735
Depends on: 1424488
Depends on: 1432978
The leave-open keyword is there and there is no activity for 6 months.
:Dolske, maybe it's time to close this bug?
Flags: needinfo?(dolske)
This migration is complete. We enabled migration for 100% eligible 32-bit Firefox 56.0 users to 64-bit 56.0.1 on 2018-10-23.

https://wiki.mozilla.org/Firefox/Win64#History
Status: REOPENED → RESOLVED
Closed: 7 years ago6 years ago
Flags: needinfo?(dolske)
Keywords: leave-open
Resolution: --- → FIXED

(In reply to Eric Rahm [:erahm] from comment #0)

There was a rather lengthy discussion [1] on whether or not it's a good idea
to update users of 32-bit Firefox a 64-bit version of Windows to a 64-bit
version of Firefox.

The general benefits proposed are:
...

  • Stability: No more OOMs due to address space fragmentation (this is a
    huge win)
    ...
  1. Not so sure these "OOMs" stopped.

  2. Need a corresponding bug to this one for Thunderbird.

(In reply to Worcester12345 from comment #36)

  1. Not so sure these "OOMs" stopped.

Anecdotally, most of the OOM crashes I've seen lately have been due to small available page files, not address fragmentation.

(In reply to Worcester12345 from comment #36)

  1. Need a corresponding bug to this one for Thunderbird.

Someone want to open a new bug and move it forward?

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: