Closed Bug 1567506 Opened 5 years ago Closed 4 years ago

Evaluate if wsl2 can be used for local Windows builds

Categories

(Firefox Build System :: General, task)

Product:
Firefox Build System
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kmoir, Assigned: rstewart)

References

Details

Evaluate it can be used to be replace MozillaBuild which is currently not maintained and very out of date.

https://www.phoronix.com/scan.php?page=article&item=windows-10-wsl2&num=1

I talked to dmajor about this work. He outlined the steps as something like

  • Install the current Windows 1903 release, install WSL and a distro, set up a build as if you were on Linux (hg clone, mach bootstrap, etc.)
  • Time some clobber builds, some "touch a single cpp" rebuilds, and some no-op rebuilds
  • Repeat the above with a Windows Insider build and WSL2
  • Bonus: use the Windows WSL utilities to downgrade the WSL2 to WSL1, and repeat the above to see if the Windows Insider WSL1 is any different from what currently ships in 1903
  • Bonus: compare the time against MSVC builds on the same machine

I'm thinking that might take 2-3 days (which includes a lot of waiting, so you could be doing other things if you have another computer).
n a Windows 10 machine, you can go into the Settings app and click the "Update & Security" tile, and there's a section near the bottom-left for "Windows Insider Program". For an offline installer you can also get .iso images from https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewadvanced.

dmajor is happy to help out and answer questions

It's implied, but just to call it out explicitly: for this first look we'd want to let WSL do a Linux-target build (this is the default, since WSL believes it's Linux). Cross-compiling to produce a Windows binary would be a much larger project, and we'd need the results from this initial investigation to see whether that's even worth attempting.

Assignee: nobody → rstewart

for anyone wanting to test wsl 1 on infrastructure, i've set up a windows server 2019 worker pool for this purpose.

here's an example task that can be run in the task creator:

provisionerId: gecko-1
workerType: win2019-gamma
retries: 5
created: '2019-10-04T05:21:19.325Z'
deadline: '2019-10-05T05:21:19.325Z'
expires: '2019-11-01T05:21:19.325Z'
scopes: []
payload:
  maxRunTime: 3600
  artifacts: []
  command:
    # display the windows build number
    - powershell -NoProfile -InputFormat None -ExecutionPolicy RemoteSigned -Command \"systeminfo | Select-String '^OS Name','^OS Version'\"
    # download wsl edition of ubuntu
    - 'curl -L https://aka.ms/wsl-ubuntu-1804 -o wsl-ubuntu-1804.zip'
    # extract ubuntu archive to the task user folder
    - 7z x -owsl-ubuntu-1804 wsl-ubuntu-1804.zip
    # install ubuntu to the windows subsystem for linux (wsl) using root as the default user account
    - wsl-ubuntu-1804\ubuntu1804.exe install --root
    # set ubuntu as the default wsl distro (ignoring exit code)
    - wsl --setdefault Ubuntu-18.04 & exit 0
    # do stuff with wsl ubuntu...
    - wsl add-apt-repository ppa:ubuntu-toolchain-r/ppa
    - wsl apt-get update
    - wsl apt-get install -y git python3.7
    - wsl python3.7 --version
    # unregister ubuntu from wsl (ignoring exit code)
    - wslconfig /u Ubuntu-18.04 & exit 0
metadata:
  owner: somebody@mozilla.com
  source: 'https://bugzilla.mozilla.org/show_bug.cgi?id=1567506'
  description: install ubuntu 1804 to windows subsystem for linux
  name: wsl1-sandbox
tags:
  os: windows
  worker-implementation: generic-worker
extra: {}

(In reply to Rob Thijssen [:grenade (EET/UTC+0300)] from comment #3)

for anyone wanting to test wsl 2 on infrastructure, i've set up a windows server 2019 worker pool for this purpose.

Isn't that WSL1 though? WSL2 is currently only available on Insider builds. (And presumably the command line would have to request version 2, or set version 2 as default)

(In reply to :dmajor from comment #4)

(In reply to Rob Thijssen [:grenade (EET/UTC+0300)] from comment #3)

for anyone wanting to test wsl 2 on infrastructure, i've set up a windows server 2019 worker pool for this purpose.

Isn't that WSL1 though? WSL2 is currently only available on Insider builds. (And presumably the command line would have to request version 2, or set version 2 as default)

yes, you're correct. i managed to confuse my versioning. i'll edit the comment above, until i can get hold of an insider build. these instances are on build 18362.

the setup steps are the same for wsl 2 but require build 18945 or higher and an extra command step to set the version: wsl --set-default-version 2.

https://www.thomasmaurer.ch/2019/08/how-to-install-wsl-2-on-windows-server/

Hey all -- I've begun looking at this and while I haven't spent that long profiling WSL2 builds, it is immediately obvious that latency is not only not better when compared to WSL1, but in fact it's significantly worse (to a factor of 3-4x as slow). Public Microsoft documentation on WSL2 doesn't provide any interesting advice (the only advice it has about I/O latency is to "ensure Linux files are on the Linux root file system", which is true in my case as far as I can tell). This seems really wrong and I'm wondering if there's something fundamental I'm missing or any advice anyone has about how to debug/tune performance here before I write WSL2 off completely.

Wow, that's surprising. I'm happy to help troubleshoot but it would go better over a more realtime communication medium. Feel free to ping me on IRC or slack when you have some time.

Just curious how you are measuring the latency. I can run the same on my WSL2 setup to see if there are any differences and we are sure it's not a local issue.

Flags: needinfo?(rstewart)

I'm just using the unix "time" command to measure latency, nothing special (although the compile process does spit out timestamps as it goes along). My benchmark has been "./mach clobber && time ./mach build" -- I've found this took ~50m on Windows 1903 w/ WSL1, and nearly 200m with Windows Insider and WSL2. I'm still investigating and trying to come up with a simple repro though.

Flags: needinfo?(rstewart)

I get:
real 33m16s
user 299m8s
sys 8m33s
on my laptop. This is around the same I get for a windows build.

I did limit wsl memory use to 10GB as it used up my entire memory and did not give back (known bug).

Thanks for the tip on limiting WSL memory. I tried that and build performance immediately substantially improved for me.

For reference, https://github.com/microsoft/WSL/issues/4166

(In reply to Kris Taeleman (:kris/kris) from comment #10)

I did limit wsl memory use to 10GB as it used up my entire memory and did not give back (known bug).

Kris, what steps did you take to limit the memory use for wsl, and where is the link to the latter "known" bug, if any? Will that be fixed soon, presumably by Microsoft?

Flags: needinfo?(ktaeleman)

I linked to that known bug in comment 11.

My %UserProfile%.wslconfig looks like this:

[wsl2]
memory=10GB
swap=0
localhostForwarding=true

(I haven't done any profiling or anything to determine that 10GB is the "right" amount of memory to cap usage at, but this is the workaround according to commenters on that bug.)

Flags: needinfo?(ktaeleman)

Threw together a doc with some initial findings.

I'm throwing the needinfo back to kmoir pending discussion about whether these results are "good enough" to be used for local Windows builds or whether more information is needed.

Flags: needinfo?(kmoir)

Do I have to uninstall VMware Workstation to build Mozilla? It will make it impossible to test on down-level systems for me.

(In reply to Masatoshi Kimura [:emk] from comment #15)

Do I have to uninstall VMware Workstation to build Mozilla? It will make it impossible to test on down-level systems for me.

This is a good point, thank you for raising it.

My understanding is, you wouldn't need to completely uninstall VMware, but you would need to change a system setting and reboot, each time you want to switch. That's still a bad experience.

I believe WSL2 can coexist with Hyper-V VMs or newer VirtualBox, so if you're willing to redo your VM setup, that could be a solution. It's a one-time pain but still not a good experience.

As a third workaround, you could use WSL1 and take the performance hit. According to Ricky's findings it's not too horrible, I was expecting worse. WSL1 still does better than MozillaBuild today.

I know that none of these workarounds are perfect, but I think we'll have to live with them; I can't imagine us supporting MozillaBuild in the super long term. Ideally a future version of VMware might coexist with WSL2, but we can't count on it.

Note that with WSL1, you can't properly interact with files on the Linux file system. So, you'd need to compare the time when building on drvfs (/mnt/c) in order for this to be practical, assuming that hasn't been done already.

Ricky, Mike and Chris discussed a path forward in the triage meeting so clearing the ni.

Flags: needinfo?(kmoir)

(In reply to :dmajor from comment #16)

Ideally a future version of VMware might coexist with WSL2, but we can't count on it.

MS announced last August that they have a pre-release version of VMware Workstation with early support for Hyper-V so I'd expect that to happen (it wasn't released in 15.5 (Sep 2019) unfortunately).

Also note that VirtualBox (since version 6/Dec 2018) and the Google Android Emulator are already compatible with Hyper-V.

(In reply to Kim Moir [:kmoir] ET from comment #18)

Ricky, Mike and Chris discussed a path forward in the triage meeting so clearing the ni.

I'm interested to hear the outcome of that meeting. Is there any plan to work on this this quarter, Kim? Working on Windows is so painful.

Flags: needinfo?(kmoir)
Depends on: 1444914
Flags: needinfo?(kmoir) → needinfo?(rstewart)

I completed this investigation a while ago. This is basically the state of the world right now:

  1. There are no apparent blockers to getting Windows builds working on WSL2, although that proof of concept hasn't been 100% completed yet.
  2. Building under WSL2 is evidently noticeably faster than performing the "same" build under MozillaBuild.
  3. There are some usability concerns for WSL2. The most obvious is that WSL2 mandates all your source code live inside a VHD -- i.e., not as actual files directly on your machine. I imagine that there are many Windows developers for whom this wouldn't be a serious concern, but it would likely break a lot of people's workflows. For example, if you use a Windows-native IDE for development, while your source code lives in the VHD it's going to be substantially slower to access it unless you're e.g. using the WSL2 plugin for VSCode which is specifically meant to help you work around this inefficiency. Given the massively distributed nature of Windows development on Firefox we felt uncomfortable committing to WSL2 and possibly disrupting a lot of people, and it wasn't clear to any of us exactly what might even be at risk of breaking here. Meanwhile, committing the resources to make this work without the promise of allowing us to e.g. deprecate MozillaBuild seemed like a nonstarter.
  4. Windows cross-compilation from Linux in automation is on the build roadmap for H1 2020. Our current feeling is that this is going to pay off regardless of which way we go with WSL2 and once Linux cross-compilation is in a good state, it should be a relatively short hop to get builds from WSL2 working. At that point the biggest challenge here is feeling comfortable effectively mandating the use of WSL2, with all the consequences that implies.
Flags: needinfo?(rstewart)

VMware update 15.5.5 now supports coexisting with Hyper-V/WSL2: https://docs.vmware.com/en/VMware-Workstation-Pro/15.5/rn/VMware-Workstation-1555-Pro-Release-Notes.html

Building under WSL2 is evidently noticeably faster than performing the "same" build under MozillaBuild.

Was the local clone in the linux drive? AFAIK accessing files in the linux drive from Windows apps (e.g. VS Code) is notably slower and I worry that it might actually degrade my experience.

For example, if you use a Windows-native IDE for development, while your source code lives in the VHD it's going to be substantially slower to access it unless you're e.g. using the WSL2 plugin for VSCode which is specifically meant to help you work around this inefficiency.

Ah, this. Sorry for making a duplicated comment :/

(In reply to Ricky Stewart from comment #21)

  1. Building under WSL2 is evidently noticeably faster than performing the "same" build under MozillaBuild.

FWIW, "noticeably", in my experience, would be between 10 and 15%. Faster, but not /that/ much faster all things considered.

Yes, those figures match up with what I saw as well. It's not an improvement by an order of magnitude or anything, but it is noticeable.

As per comment 21, this has been evaluated.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.