Closed
Bug 1244750
Opened 8 years ago
Closed 8 years ago
Port 2015 TaskCluster Windows build POC to current mozilla-central default head
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(firefox50 fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox50 | --- | fixed |
People
(Reporter: grenade, Assigned: grenade)
References
Details
Attachments
(1 file, 38 obsolete files)
2.49 KB,
patch
|
garndt
:
review+
|
Details | Diff | Splinter Review |
- adapt POC patch to work with current m-c - output to include binary builds: - 32 bit opt - 64 bit opt - 32 bit debug - 64 bit debug - determine if a new platform is needed (whether try jobs can run TC/BB builds independently)
Assignee | ||
Comment 1•8 years ago
|
||
Assignee | ||
Comment 2•8 years ago
|
||
> grenade@quadbrat ~/hg/hg.mozilla.org/mozilla-central hg:default:282529 ?+$ hg add . > adding testing/taskcluster/tasks/builds/firefox_linux_base.yml > adding testing/taskcluster/tasks/builds/firefox_win32_debug.yml > adding testing/taskcluster/tasks/builds/firefox_win32_opt.yml > adding testing/taskcluster/tasks/builds/firefox_win64_debug.yml > adding testing/taskcluster/tasks/builds/firefox_win64_opt.yml > adding testing/taskcluster/tasks/builds/firefox_windows_base.yml > adding testing/taskcluster/tasks/linux_build.yml > adding testing/taskcluster/tasks/windows_build.yml > grenade@quadbrat ~/hg/hg.mozilla.org/mozilla-central hg:default:282529 +$ hg commit -m "try: -b o -p win32,win64 -u none -t none" > grenade@quadbrat ~/hg/hg.mozilla.org/mozilla-central hg:default:282530 $ hg push -f ssh://hg.mozilla.org/try > pushing to ssh://hg.mozilla.org/try > searching for changes > remote: adding changesets > remote: adding manifests > remote: adding file changes > remote: added 1 changesets with 20 changes to 20 files (+1 heads) > remote: recorded push in pushlog > remote: > remote: View your change here: > remote: https://hg.mozilla.org/try/rev/1d543c583e70 > remote: > remote: Follow the progress of your build on Treeherder: > remote: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1d543c583e70 > remote: recorded changegroup in replication log in 0.056s
Assignee | ||
Comment 3•8 years ago
|
||
Attachment #8714745 -
Attachment is obsolete: true
Assignee | ||
Comment 4•8 years ago
|
||
revert generic-worker version to 1.0.11
Attachment #8716890 -
Attachment is obsolete: true
Assignee | ||
Comment 5•8 years ago
|
||
use new worker type "y-2012"
Attachment #8719741 -
Attachment is obsolete: true
Assignee | ||
Comment 6•8 years ago
|
||
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284334 $ hg import --no-commit https://bug1244750.bmoattachments.org/attachment.cgi?id=8719743
> applying https://bug1244750.bmoattachments.org/attachment.cgi?id=8719743
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284334 +$ hg add .
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284334 +$ hg commit -m "try: -b do -p win32,win64 -u none -t none"
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284335 $ hg push -f ssh://hg.mozilla.org/try
> pushing to ssh://hg.mozilla.org/try
> searching for changes
> remote: adding changesets
> remote: adding manifests
> remote: adding file changes
> remote: added 5 changesets with 22 changes to 68 files (+1 heads)
> remote: recorded push in pushlog
> remote:
> remote: View your changes here:
> remote: https://hg.mozilla.org/try/rev/f5df9f4596dc
> remote: https://hg.mozilla.org/try/rev/03fc467e1002
> remote: https://hg.mozilla.org/try/rev/53029769feef
> remote: https://hg.mozilla.org/try/rev/6ea654cad929
> remote: https://hg.mozilla.org/try/rev/c2a12e796c42
> remote:
> remote: Follow the progress of your build on Treeherder:
> remote: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2a12e796c42
> remote: recorded changegroup in replication log in 0.065s
>
> [taskcluster:error] Error calling 'stopped' for extendTaskGraph : Error encountered when attempting to extend task graph. Graph server error while extending task graph id bExcSj11R9GH-z_w2FmFjQ : You didn't give the task-graph scopes allowing it define tasks on the queue., [{"message":"You do not have sufficient scopes. This request requires you\nto have one of the following sets of scopes:\n[\n [\n \"queue:define-task:aws-provisioner-v1/y-2012\"\n ],\n [\n \"queue:create-task:aws-provi
Assignee | ||
Comment 7•8 years ago
|
||
Attachment #8719743 -
Attachment is obsolete: true
Assignee | ||
Comment 8•8 years ago
|
||
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 ?+$ hg pull pulling from ssh://hg.mozilla.org/mozilla-central searching for changes no changes found grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 ?+$ hg update -r default -C 13 files updated, 0 files merged, 0 files removed, 0 files unresolved grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 ?$ hg purge grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 $ hg import --no-commit https://bug1244750.bmoattachments.org/attachment.cgi?id=8722460 applying https://bug1244750.bmoattachments.org/attachment.cgi?id=8722460 grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 +$ hg commit -m "try: -b do -p win32,win64 -u none -t none" grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285026 $ hg push -f ssh://hg.mozilla.org/try pushing to ssh://hg.mozilla.org/try searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 2 changes to 2 files (+1 heads) remote: recorded push in pushlog remote: remote: View your change here: remote: https://hg.mozilla.org/try/rev/9011535fbd44 remote: remote: Follow the progress of your build on Treeherder: remote: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9011535fbd44 remote: recorded changegroup in replication log in 0.070s
Assignee | ||
Comment 9•8 years ago
|
||
Attachment #8722460 -
Attachment is obsolete: true
Assignee | ||
Comment 10•8 years ago
|
||
Attachment #8724019 -
Attachment is obsolete: true
Assignee | ||
Comment 11•8 years ago
|
||
Attachment #8724021 -
Attachment is obsolete: true
Assignee | ||
Comment 12•8 years ago
|
||
Attachment #8736654 -
Attachment is obsolete: true
Assignee | ||
Comment 13•8 years ago
|
||
Assignee | ||
Updated•8 years ago
|
Attachment #8736753 -
Attachment is obsolete: true
Assignee | ||
Comment 14•8 years ago
|
||
Attachment #8736761 -
Attachment is obsolete: true
Assignee | ||
Comment 15•8 years ago
|
||
Attachment #8736769 -
Attachment is obsolete: true
Assignee | ||
Comment 16•8 years ago
|
||
Attachment #8736830 -
Attachment is obsolete: true
Assignee | ||
Comment 17•8 years ago
|
||
Attachment #8737731 -
Attachment is obsolete: true
Assignee | ||
Comment 18•8 years ago
|
||
https://hg.mozilla.org/try/rev/ddbf29b9692e https://treeherder.mozilla.org/#/jobs?repo=try&revision=ddbf29b9692e
Assignee | ||
Comment 19•8 years ago
|
||
Attachment #8737772 -
Attachment is obsolete: true
Assignee | ||
Comment 20•8 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5767f4eb2711
Comment 21•8 years ago
|
||
Hey Ted -- could you have a look at this latest patch and the try output? :pmoore, could you have a look too? I see this in the log output at https://public-artifacts.taskcluster.net/bi9kSwbcQF6fq66PG8q_5A/0/public/logs/command_000003.log: 17:03:29 INFO - Creating Python environment 17:03:30 INFO - New python executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python2.7.exe 17:03:30 INFO - Also creating executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python.exe 17:03:32 INFO - Installing setuptools, pip, wheel...done. 17:03:33 INFO - running build_ext 17:03:33 INFO - building 'psutil._psutil_windows' extension 17:03:33 INFO - error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27 17:03:33 INFO - Error processing command. Ignoring because optional. (optional:setup.py:python/psutil:build_ext:--inplace) 17:03:33 INFO - C:\home\worker\workspace\build\src\python\mozbuild\mozbuild\virtualenv.py:361: UserWarning: Hacking environment to allow binary Python extensions to build. You can make this warning go away by installing Visual Studio 2008. You can download the Express Edition installer from http://go.microsoft.com/?linkid=7729279 17:03:33 INFO - warnings.warn('Hacking environment to allow binary Python ' 17:03:33 INFO - Reexecuting in the virtualenv 17:03:33 INFO - Creating Python environment 17:03:33 INFO - Please use the *system* python to run this script 17:03:33 INFO - Installing setuptools, pip, wheel... 17:03:33 INFO - Error [Error 5] Access is denied while executing command c:\home\worker\works...uild\src\obj-firefox - setuptools pip wheel 17:03:33 INFO - ...Installing setuptools, pip, wheel...done. 17:03:33 INFO - Traceback (most recent call last): 17:03:33 INFO - File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 2316, in <module> 17:03:33 INFO - main() 17:03:33 INFO - File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 708, in main 17:03:33 INFO - symlink=options.symlink) 17:03:33 INFO - File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 941, in create_environment 17:03:33 INFO - download=download, 17:03:33 INFO - File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 897, in install_wheel 17:03:33 INFO - call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT) 17:03:33 INFO - File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 748, in call_subprocess 17:03:33 INFO - cwd=cwd, env=env) 17:03:33 INFO - File "c:\Python27\Lib\subprocess.py", line 710, in __init__ 17:03:33 INFO - errread, errwrite) 17:03:33 INFO - File "c:\Python27\Lib\subprocess.py", line 958, in _execute_child 17:03:33 INFO - startupinfo) 17:03:33 INFO - WindowsError: [Error 5] Access is denied 17:03:33 INFO - Traceback (most recent call last): 17:03:33 INFO - File "C:/home/worker/workspace/build/src/configure.py", line 94, in <module> 17:03:33 INFO - sys.exit(main(sys.argv)) 17:03:33 INFO - File "C:/home/worker/workspace/build/src/configure.py", line 22, in main 17:03:33 INFO - sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure')) 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 172, in run 17:03:33 INFO - self.exec_file(path) 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 165, in exec_file 17:03:33 INFO - exec(code, self) 17:03:33 INFO - File "C:/home/worker/workspace/build/src/moz.configure", line 7, in <module> 17:03:33 INFO - include('build/moz.configure/init.configure') 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 354, in include_impl 17:03:33 INFO - self.exec_file(what) 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 165, in exec_file 17:03:33 INFO - exec(code, self) 17:03:33 INFO - File "C:/home/worker/workspace/build/src/build/moz.configure/init.configure", line 165, in <module> 17:03:33 INFO - @advanced 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 337, in decorator 17:03:33 INFO - self._results[func] = func(*resolved_args) 17:03:33 INFO - File "C:/home/worker/workspace/build/src/build/moz.configure/init.configure", line 215, in virtualenv_python 17:03:33 INFO - manager.build(python) 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\virtualenv.py", line 424, in build 17:03:33 INFO - self.create(python) 17:03:33 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\virtualenv.py", line 188, in create 17:03:33 INFO - 'Failed to create virtualenv: %s' % self.virtualenv_root) 17:03:33 INFO - Exception: Failed to create virtualenv: c:/home/worker/workspace/build/src/obj-firefox\_virtualenv 17:03:33 INFO - *** Fix above errors and then restart with\ 17:03:33 INFO - "C:/home/worker/workspace/build/src/mozmake.EXE -f client.mk build" 17:03:33 INFO - C:/home/worker/workspace/build/src/client.mk:382: recipe for target 'configure' failed 17:03:33 INFO - mozmake.EXE[2]: *** [configure] Error 1 17:03:33 INFO - mozmake.EXE[2]: Leaving directory 'C:/home/worker/workspace/build/src' 17:03:33 INFO - C:/home/worker/workspace/build/src/client.mk:396: recipe for target 'C:/home/worker/workspace/build/src/obj-firefox/Makefile' failed 17:03:33 INFO - mozmake.EXE[1]: *** [C:/home/worker/workspace/build/src/obj-firefox/Makefile] Error 2 17:03:33 INFO - mozmake.EXE[1]: Leaving directory 'C:/home/worker/workspace/build/src' 17:03:33 INFO - client.mk:181: recipe for target 'build' failed 17:03:33 INFO - mozmake.EXE: *** [build] Error 2 17:03:33 INFO - 0 compiler warnings present. 17:03:34 INFO - 2 17:03:34 ERROR - Return code: 1 17:03:34 WARNING - setting return code to 2 17:03:34 FATAL - 'mach build' did not run successfully. Please check log for errors.
Flags: needinfo?(ted)
Comment 22•8 years ago
|
||
I think gps is better equipped to help diagnose this try error, but if he doesn't have time for it or can't get it sorted I will take a look. (We're both in TRIBE all day tomorrow.)
Flags: needinfo?(gps)
Assignee | ||
Comment 23•8 years ago
|
||
leave buildbot intact...
Attachment #8738610 -
Attachment is obsolete: true
Assignee | ||
Updated•8 years ago
|
Attachment #8738925 -
Attachment is patch: true
Assignee | ||
Comment 24•8 years ago
|
||
My debug efforts for error shown in comment 21: - Got rid of the "Microsoft Visual C++ 9.0" warning by setting all the lib/include/path environment variables that point to directories under the vs2015 install. This didn't affect the virtualenv call (or stop it failing). - Modified line 188 of src/python/mozbuild/mozbuild/virtualenv.py to read "'Failed to create virtualenv: %s (%s)' % (self.virtualenv_root, args))" so that i could see what call was failing. - The failing call turned out to be (fails with same error when run manually): c:\\home\\worker\\workspace\\build\\src\\obj-firefox\\_virtualenv\\Scripts\\python.exe C:/home/worker/workspace/build/src\\python\\virtualenv\\virtualenv.py --no-download c:/home/worker/workspace/build/src/obj-firefox\\_virtualenv - When run with system python manually (c:\Python27\python.exe C:/home/worker/workspace/build/src\\python\\virtualenv\\virtualenv.py --no-download c:/home/worker/workspace/build/src/obj-firefox\\_virtualenv), call succeeds. - When virtualenv.py is modified to call system python instead, it still fails.
Assignee | ||
Comment 25•8 years ago
|
||
vs 2015 lib/include/path env vars: https://hg.mozilla.org/try/rev/f8ae5c527fc56349ed8d3789b3435c949f98ef0b
Assignee | ||
Comment 26•8 years ago
|
||
Attachment #8738925 -
Attachment is obsolete: true
Assignee | ||
Comment 27•8 years ago
|
||
Assignee | ||
Updated•8 years ago
|
Attachment #8739049 -
Attachment is obsolete: true
Assignee | ||
Comment 28•8 years ago
|
||
For anyone who wants to help debug this, you can create a windows builder from a vanilla amazon ec2 windows 2012 r2 ami by running the following two lines of powershell: > Set-ExecutionPolicy RemoteSigned -f > Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/MozRelOps/OpenCloudConfig/master/userdata/win2012-vs2015.ps1') Then run this script in a cmd prompt: https://gist.github.com/grenade/1e63fcb938ff4fc22924da23127e1d81#file-build-firefox-cmd
Comment 29•8 years ago
|
||
(In reply to Rob Thijssen (:grenade - GMT) from comment #25) > vs 2015 lib/include/path env vars: > https://hg.mozilla.org/try/rev/f8ae5c527fc56349ed8d3789b3435c949f98ef0b I'm not a fan of this patch because it is reimplementing logic in the in-tree mozconfigs. The order of directories for the MSVC toolchain environment variables is pretty sensitive. I'd prefer we not have this logic implemented in multiple places.
Comment 30•8 years ago
|
||
The configure rewrite recently changed the way virtualenvs are installed. The error in comment #21 possibly points to a regression from that refactor. What's alarming about comment #21 is "Creating Python environment" is printed twice. Specifically, the wonkiness is: 16:47:24 INFO - Calling ['C:\\Python27\\python.exe', 'mach', '--log-no-times', 'build', '-v'] with output_timeout 4800 17:03:04 INFO - C:/home/worker/workspace/build/src/configure 17:03:29 INFO - Creating Python environment 17:03:30 INFO - New python executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python2.7.exe 17:03:30 INFO - Also creating executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python.exe 17:03:32 INFO - Installing setuptools, pip, wheel...done. 17:03:33 INFO - Reexecuting in the virtualenv # This is where things get weird 17:03:33 INFO - Creating Python environment 17:03:33 INFO - Please use the *system* python to run this script `configure` is supposed to create the virtualenv then re-execute itself from the newly-created virtualenv. However, when it re-executes itself, it attempts to install the virtualenv over the existing virtualenv (it is somehow not detecting it is running from the just-created virtualenv). This leads to a permission denied error because Windows won't let you modify/replace an executable that is running. glandium: since this is likely fallout from configure rewrite, can you take a look?
Flags: needinfo?(gps) → needinfo?(mh+mozilla)
Assignee | ||
Comment 31•8 years ago
|
||
Greg: re comment 29, it'd be great to have a chat when you have time about the changes introduced by having vs2015 downloaded via tooltool. In response to that change, I dropped the installation of any Visual Studio version to the taskcluster worker type. We had previously been installing either 2013 or 2015 community edition. The patch to set lib/include paths was more because of my misunderstanding of how that all works now, and motivated by wanting to get rid of environment variables that reference `C:\Program Files (x86)\Visual Studio 14.0` which no longer exists. If we don't need them because they're already in mozconfig, i'll just rip'em back out. I'd also like to understand if the builders/workertypes still need installations of windows sdks or if that requirement is deprecated too.
Flags: needinfo?(gps)
Comment 32•8 years ago
|
||
The vs2015 archive in tooltool contains *everything* Microsoft needed to build. No Visual Studio or SDKs need to be installed at the system level. It is possible to build Firefox with just a MozillaBuild install and the vs2015 tooltool archive. (Although there is a bug in MozillaBuild where a vcruntimeXXX.dll file is missing and I /think/ yasm.exe fails to run, so watch out for that.) The in-tree mozconfigs (e.g. build/win64/mozconfig.vs2015) set all the environment variables needed to find the MSVC and SDK files from the tooltool archive.
Flags: needinfo?(gps)
Comment 33•8 years ago
|
||
Bug 1255585 sounds like a candidate.
Flags: needinfo?(mh+mozilla) → needinfo?(cmanchester)
Comment 34•8 years ago
|
||
The symptom looks related (the virtualenv is always considered invalid). I'll push to try with some diagnostic logging to see why that might be.
Updated•8 years ago
|
Flags: needinfo?(ted)
Flags: needinfo?(cmanchester)
Comment 35•8 years ago
|
||
The difference between the TC builds and buildbot builds causing this failure is that TC is calling os.path.getsize on the Python interpreter at c:\Python27\python2.7.exe and finding its size is 0, while buildbot is calling os.path.getsize on the interpreter at c:\mozilla-build\python27\python2.7.exe and finding an actual size. Why this is, I'm not sure. We compare this size to the size of the Python interpreter in the objdir virtualenv to attempt to detect incompatible Python versions, and consider the virtualenv to be invalid when they don't match.
Assignee | ||
Comment 36•8 years ago
|
||
Thanks Chris! That solves it for me. The TC builders had a symlink at c:\Python27\python2.7.exe pointing to c:\Python27\python.exe. That's now patched to be a copy of the executable instead.
Assignee | ||
Comment 37•8 years ago
|
||
Attachment #8739053 -
Attachment is obsolete: true
Assignee | ||
Comment 38•8 years ago
|
||
Attachment #8739905 -
Attachment is obsolete: true
Comment 39•8 years ago
|
||
Comment on attachment 8739921 [details] [diff] [review] patch rewrite for mozharness builds Review of attachment 8739921 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mozharness/configs/builds/releng_base_windows_32_builds.py @@ +30,4 @@ > ], > "buildbot": [ > sys.executable, > + os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build', 'buildbotve', 'scripts', 'buildbot') Do we need the reference to os.path.sep here, or would this also work? If it can be removed, note there are some more references below that should also be updated. os.path.join(os.getenv('SystemDrive', 'C:'), 'mozilla-build', 'buildbotve', 'scripts', 'buildbot') @@ +38,4 @@ > ], > 'virtualenv': [ > sys.executable, > + os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build', 'buildbotve', 'virtualenv.py') What is the reason to use os.path.join rather than a literal backslash? Isn't this config only useful for a Windows machine, which always uses a backslash? ::: testing/taskcluster/scripts/builder/build-windows.cmd @@ +13,5 @@ > +::if not defined MH_CUSTOM_BUILD_VARIANT_CFG set MH_CUSTOM_BUILD_VARIANT_CFG= > +if not defined MH_BRANCH set MH_BRANCH=mozilla-central > +if not defined MH_BUILD_POOL set MH_BUILD_POOL=staging > + > +if not defined WORKSPACE set WORKSPACE=%SystemDrive%\home\worker\workspace What is WORKSPACE for? Should this be in the task folder created by the worker? @@ +43,5 @@ > + > +set custom_build_variant_cfg_flag= > +:: if defined MH_CUSTOM_BUILD_VARIANT_CFG set custom_build_variant_cfg_flag=--custom-build-variant-cfg=%MH_CUSTOM_BUILD_VARIANT_CFG% > + > +::if "%CHECKOUT_GAIA%" == "true" ( Do we still need this? ::: testing/taskcluster/tasks/windows_build.yml @@ +8,5 @@ > + routes: > + - 'index.buildbot.branches.{{project}}.{{build_name}}' > + - 'index.buildbot.revisions.{{head_rev}}.{{project}}.{{build_name}}' > + > + #routes: did you mean to comment this out? @@ +22,5 @@ > + # All builds share a common artifact directory for ease of uploading. > + #artifacts: > + #'public/build': > + #type: directory > + #path: 'C:\\home\\worker\\artifacts\\' Should this commented section be here? If it should be here, shouldn't it use the Task directory, not C:\home\worker\artifacts? ::: testing/windows/desktop-build/checkout-sources.cmd @@ +49,5 @@ > +if exist %GECKO_DIR% rmdir %GECKO_DIR% /s /q || echo ERROR && exit /b > +mkdir %GECKO_DIR% || echo ERROR && exit /b > +hg clone -U %GECKO_BASE_REPOSITORY% %GECKO_DIR% || echo ERROR && exit /b > +hg pull -u -R %GECKO_DIR% --rev %GECKO_HEAD_REV% %GECKO_HEAD_REPOSITORY% || echo ERROR && exit /b > +hg update -R %GECKO_DIR% --rev %GECKO_HEAD_REV% || echo ERROR && exit /b Won't this script always clone the repositories, rather than using a vcs cache?
Assignee | ||
Comment 40•8 years ago
|
||
Attachment #8739921 -
Attachment is obsolete: true
Assignee | ||
Comment 41•8 years ago
|
||
Attachment #8741364 -
Attachment is obsolete: true
Assignee | ||
Comment 42•8 years ago
|
||
Latest build failure: 11:00:34 INFO - js\src\ctypes\libffi> ******************************************************** 11:00:34 INFO - js\src\ctypes\libffi> * WARNING: Don't know the best CFLAGS for this system * 11:00:34 INFO - js\src\ctypes\libffi> * Use ./configure CFLAGS=... to specify your own flags * 11:00:34 INFO - js\src\ctypes\libffi> * (otherwise, a default of CFLAGS=-O3 will be used) * 11:00:34 INFO - js\src\ctypes\libffi> ******************************************************** 11:00:34 INFO - js\src\ctypes\libffi> 11:00:34 INFO - js\src\ctypes\libffi> checking whether C compiler accepts -O3... yes 11:00:34 INFO - js\src\ctypes\libffi> checking CFLAGS for maximum warnings... -Wall 11:00:34 INFO - js\src\ctypes\libffi> checking whether to enable maintainer-specific portions of Makefiles... no 11:00:34 INFO - js\src\ctypes\libffi> checking sys/mman.h usability... no 11:00:34 INFO - js\src\ctypes\libffi> checking sys/mman.h presence... no 11:00:34 INFO - js\src\ctypes\libffi> checking for sys/mman.h... no 11:00:34 INFO - js\src\ctypes\libffi> checking for mmap... no 11:00:34 INFO - js\src\ctypes\libffi> checking for sys/mman.h... (cached) no 11:00:34 INFO - js\src\ctypes\libffi> checking for mmap... (cached) no 11:00:34 INFO - js\src\ctypes\libffi> checking for ANSI C header files... (cached) yes 11:00:34 INFO - js\src\ctypes\libffi> checking for memcpy... no 11:00:34 INFO - js\src\ctypes\libffi> checking for size_t... yes 11:00:34 INFO - js\src\ctypes\libffi> checking for working alloca.h... no 11:00:34 INFO - js\src\ctypes\libffi> checking for alloca... yes 11:00:34 INFO - js\src\ctypes\libffi> checking size of double... 8 11:00:34 INFO - js\src\ctypes\libffi> checking size of long double... 8 11:00:34 INFO - js\src\ctypes\libffi> checking whether byte ordering is bigendian... no 11:00:34 INFO - js\src\ctypes\libffi> checking assembler .cfi pseudo-op support... no 11:00:34 INFO - js\src\ctypes\libffi> checking for _ prefix in compiled symbols... no 11:00:34 INFO - js\src\ctypes\libffi> configure: updating cache c:/home/worker/workspace/build/src/obj-firefox/js/src/ctypes/libffi/config.cache 11:00:34 INFO - js\src\ctypes\libffi> checking that generated files are newer than configure... done 11:00:34 INFO - js\src\ctypes\libffi> configure: creating ./config.status 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating include/Makefile 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating include/ffi.h 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating Makefile 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating testsuite/Makefile 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating man/Makefile 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating libffi.pc 11:00:34 INFO - js\src\ctypes\libffi> config.status: creating fficonfig.h 11:00:34 INFO - js\src\ctypes\libffi> config.status: linking C:/home/worker/workspace/build/src/js/src/ctypes/libffi/src/x86/ffitarget.h to include/ffitarget.h 11:00:34 INFO - js\src\ctypes\libffi> config.status: executing buildir commands 11:00:34 INFO - js\src\ctypes\libffi> config.status: skipping top_srcdir/Makefile - not created 11:00:34 INFO - js\src\ctypes\libffi> config.status: executing depfiles commands 11:00:34 INFO - js\src\ctypes\libffi> config.status: executing libtool commands 11:00:34 INFO - js\src\ctypes\libffi> config.status: executing include commands 11:00:34 INFO - js\src\ctypes\libffi> config.status: executing src commands 11:00:36 INFO - Reticulating splines... 11:00:36 INFO - Traceback (most recent call last): 11:00:36 INFO - File "config.status", line 992, in <module> 11:00:36 INFO - config_status(**args) 11:00:36 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\config_status.py", line 156, in config_status 11:00:36 INFO - definitions = list(definitions) 11:00:36 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\frontend\emitter.py", line 165, in emit 11:00:36 INFO - for out in output: 11:00:36 INFO - File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\frontend\reader.py", line 1066, in read_mozbuild 11:00:36 INFO - raise bre 11:00:36 INFO - mozbuild.frontend.reader.BuildReaderError: 11:00:36 INFO - ============================== 11:00:36 INFO - ERROR PROCESSING MOZBUILD FILE 11:00:36 INFO - ============================== 11:00:36 INFO - The error occurred while processing the following file: 11:00:36 INFO - C:/home/worker/workspace/build/src/moz.build 11:00:36 INFO - The underlying problem is an illegal file access. This is likely due to trying to access a file outside of the top source directory. 11:00:36 INFO - The path whose access was denied is: 11:00:36 INFO - C:/home/worker/workspace/build/src/js/src/moz.build 11:00:36 INFO - Modify the script to not access this file and try again. 11:00:36 INFO - Creating config.status 11:00:36 INFO - *** Fix above errors and then restart with\ 11:00:36 INFO - "C:/home/worker/workspace/build/src/mozmake.EXE -f client.mk build" 11:00:36 INFO - C:/home/worker/workspace/build/src/client.mk:370: recipe for target 'configure' failed 11:00:36 INFO - mozmake.EXE[2]: *** [configure] Error 1 11:00:36 INFO - mozmake.EXE[2]: Leaving directory 'C:/home/worker/workspace/build/src' 11:00:36 INFO - C:/home/worker/workspace/build/src/client.mk:384: recipe for target 'C:/home/worker/workspace/build/src/obj-firefox/Makefile' failed 11:00:36 INFO - mozmake.EXE[1]: *** [C:/home/worker/workspace/build/src/obj-firefox/Makefile] Error 2 11:00:36 INFO - mozmake.EXE[1]: Leaving directory 'C:/home/worker/workspace/build/src' 11:00:36 INFO - client.mk:168: recipe for target 'build' failed 11:00:36 INFO - mozmake.EXE: *** [build] Error 2 11:00:36 INFO - 1 compiler warnings present. 11:00:36 INFO - 2 11:00:36 ERROR - Return code: 1 11:00:36 WARNING - setting return code to 2 11:00:36 FATAL - 'mach build' did not run successfully. Please check log for errors.
Assignee | ||
Comment 43•8 years ago
|
||
above is from: https://tools.taskcluster.net/task-inspector/#DwIueGF9QuiyVyrIa9mgRg/0
Comment 44•8 years ago
|
||
I created a diff between the broken TC env and the working BB env: http://people.mozilla.org/~pmoore/Screen%20Shot%202016-04-19%20at%2013.53.15.png I'll upload source files too...
Comment 45•8 years ago
|
||
Env for failing TC build
Comment 46•8 years ago
|
||
Env for passing BB build
Comment 47•8 years ago
|
||
I just remembered, I had this error too and solved it a different way! It was a horrid mach bug. Long story short, if you execute mach with an absolute path, you need to specify a lowercase drive letter. I had a problem because I was executing: C:\\mozilla-build\\msys\\bin\\bash --login %CD%\mach.exe Instead I now specify a relative path to avoid the drive being included in the path. The reason for this is when mach tries to work out the topsrcdir variable, it takes inspiration from `$0`, i.e. the command used to call mach. When it compares the location of a mozconfig relative to this location, it expects to find it underneath. However, by that point it has used python function os.getcwd() which returns a lower case drive letter, or at least it does when called from msys. Since you're calling mozharness, I'm guessing the location of mach is going to be in a mozharness config file. Probably this should be specified in msys format (/c/....) with a lower case drive letter. Note, to get it working for me, where I call mach directly, I changed the mach call to be: C:\\mozilla-build\\msys\\bin\\bash --login -c "cd '%CD%'; ./mach build" Nasty little bug. :)
Comment 48•8 years ago
|
||
Pushed a new commit to get more debug, to confirm whether this is exactly the same cause that I had, or a subtly different reason: * https://treeherder.mozilla.org/#/jobs?repo=try&revision=912139a9c662
Comment 49•8 years ago
|
||
This is similar to bug 1177634 and should be fixed the same way, in fact, in the same function.
Comment 50•8 years ago
|
||
Yep, that was it: 0:01.22 CAUSE OF FAILURE: C:/home/worker/workspace/build/src/js/src/moz.build falls outside of c:/home/worker/workspace/build/src https://public-artifacts.taskcluster.net/eDAjycrnQtW9qQDbdbIWUA/0/public/logs/command_000003.log
Comment 51•8 years ago
|
||
This is how it is invoked in the failing TC build: Running command: ['C:\\mozilla-build\\python\\python.exe', 'mach', '--log-no-times', 'build', '-v'] in C:\home\worker\workspace\build\src This is how it is invoked in the succeeding BB build: Running command: ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v'] in c:\builds\moz2_slave\try-w64-0000000000000000000000\build\src Notice the `C:` vs `c:`.
Assignee | ||
Comment 52•8 years ago
|
||
Attachment #8742758 -
Attachment is obsolete: true
Assignee | ||
Comment 53•8 years ago
|
||
So in recent try pushes win64 builds don't complete (win32 is a different issue, related to DirectXSDK). The logs are never uploaded to treeherder/taskinspector because the build just doesn't finish. An excerpt of the raw log (taken from the worker instance) showing the looping follows: Build configuration changed. Regenerating backend. c:/Users/Task_1461100687/build/src/obj-firefox/_virtualenv/Scripts/python.exe config.status =============================== Visual Studio Support Available You are building Firefox on Windows. Please help us test the experimental Visual Studio project files (yes, IntelliSense works) by running the following: mach build-backend --backend=VisualStudio =============================== Reticulating splines... Finished reading 2990 moz.build files in 7.75s Processed into 9663 build config descriptors in 4.85s RecursiveMake backend executed in 24.08s 2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile FasterMake backend executed in 0.58s 10 total backend files; 0 created; 0 updated; 10 unchanged; 0 deleted Total wall time: 62.56s; CPU time: 62.55s; Efficiency: 100%; Untracked: 25.29s Build configuration changed. Regenerating backend. c:/Users/Task_1461100687/build/src/obj-firefox/_virtualenv/Scripts/python.exe config.status =============================== Visual Studio Support Available You are building Firefox on Windows. Please help us test the experimental Visual Studio project files (yes, IntelliSense works) by running the following: mach build-backend --backend=VisualStudio =============================== Reticulating splines... Finished reading 2990 moz.build files in 7.69s Processed into 9663 build config descriptors in 4.80s RecursiveMake backend executed in 23.94s 2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile FasterMake backend executed in 0.57s 10 total backend files; 0 created; 0 updated; 10 unchanged; 0 deleted Total wall time: 62.71s; CPU time: 62.71s; Efficiency: 100%; Untracked: 25.71s Build configuration changed. Regenerating backend. c:/Users/Task_1461100687/build/src/obj-firefox/_virtualenv/Scripts/python.exe config.status =============================== Visual Studio Support Available You are building Firefox on Windows. Please help us test the experimental Visual Studio project files (yes, IntelliSense works) by running the following: mach build-backend --backend=VisualStudio =============================== Reticulating splines... Finished reading 2990 moz.build files in 7.74s Processed into 9663 build config descriptors in 4.79s RecursiveMake backend executed in 24.09s 2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile FasterMake backend executed in 0.58s 10 total backend files; 0 created; 0 updated; 10 unchanged; 0 deleted Total wall time: 62.71s; CPU time: 62.71s; Efficiency: 100%; Untracked: 25.51s
Attachment #8743208 -
Flags: feedback?(cmanchester)
Assignee | ||
Comment 54•8 years ago
|
||
examples of above errors in treeherder: https://treeherder.mozilla.org/#/jobs?repo=try&revision=435b2a2db2ed47413b5e8f6fa3dd33140e793da9 https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e00422e69c53be32afd056ebd3df36907cf0aea
Assignee | ||
Comment 55•8 years ago
|
||
Attachment #8742874 -
Attachment is obsolete: true
Comment 56•8 years ago
|
||
Comment on attachment 8743208 [details]
build looping on RecursiveMake
This seems to be related to the fix for the c:\ vs C:\ paths, so we've pulled that out, and instead skip over the check for now. We'll fix it properly once we've got the green builds working.
Attachment #8743208 -
Flags: feedback?(cmanchester)
Comment 57•8 years ago
|
||
> RecursiveMake backend executed in 24.08s
> 2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile
That is absurdly slow :/
Assignee | ||
Comment 58•8 years ago
|
||
when i checked on the instance it was running 87 mozmake processes. I assume the slowness is related to the recursion...
Assignee | ||
Comment 59•8 years ago
|
||
Attachment #8743209 -
Attachment is obsolete: true
Assignee | ||
Updated•8 years ago
|
Attachment #8743759 -
Attachment is patch: true
Comment 60•8 years ago
|
||
Comment on attachment 8743759 [details] [diff] [review] patch rewrite for mozharness builds Review of attachment 8743759 [details] [diff] [review]: ----------------------------------------------------------------- just some drive by comments...more first impressions than official review. ::: testing/mozharness/configs/builds/releng_base_windows_32_builds.py @@ +34,4 @@ > ], > "make": [ > sys.executable, > + os.path.join(os.getcwd(), 'build', 'src', 'build', 'pymake', 'make.py') before this was split to reduce the line length, I would think that would still be desireable. @@ +38,4 @@ > ], > 'virtualenv': [ > sys.executable, > + os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build', 'buildbotve', 'virtualenv.py') why do we need a os.path.sep here? I thought os.path.join() would include that automatically? @@ +44,5 @@ > 'app_ini_path': '%(obj_dir)s/dist/bin/application.ini', > # decides whether we want to use moz_sign_cmd in env > 'enable_signing': True, > 'enable_ccache': False, > + 'vcs_share_base': os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'builds', 'hg-shared'), can we pull os.getenv('SystemDrive', 'C:') out as a variable to make things more readable? that would help in many of these calls. In fact, making something like "moz_build = os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build'" would make things easier. ::: testing/taskcluster/scripts/builder/build-windows.cmd @@ +29,5 @@ > + > +:: test required parameters are supplied > +if not defined MOZHARNESS_SCRIPT exit /b 1 > +if not defined MOZHARNESS_CONFIG exit /b 1 > +if "%NEED_XVFB%" == "true" exit /b 1 do we have xvfb for windows? I thought that was a linux thing @@ +46,5 @@ > + ::mkdir %WORKSPACE%\build\tools || echo ERROR && exit /b > + ::hg clone -U %TOOLS_BASE_REPOSITORY% %WORKSPACE%\build\tools || echo ERROR && exit /b > + ::hg pull -u -R %WORKSPACE%\build\tools --rev %TOOLS_HEAD_REV% %TOOLS_HEAD_REPOSITORY% || echo ERROR && exit /b > + ::hg update -R %WORKSPACE%\build\tools --rev %TOOLS_HEAD_REV% || echo ERROR && exit /b > +::) not sure if we are doing gaia anymore @@ +73,5 @@ > + --no-upload-files^ > + --no-sendchange^ > + --log-level=debug^ > + --work-dir=%WORKSPACE%\build^ > + --no-action=generate-build-stats^ we needed generate-build-stats for linux builds in tc, I assume we can tweak these defaults when we get the builds working. ::: testing/taskcluster/tasks/builds/base_win32.yml @@ +10,5 @@ > + payload: > + env: > + CommandPromptType: "Cross" > + LIB: "C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\x86;" > + LIBPATH: "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319;C:\\Program Files (x86)\\Windows Kits\\8.1\\References\\CommonConfiguration\\Neutral;" we have c:\ hardcoded here, is that ok?
Assignee | ||
Comment 61•8 years ago
|
||
https://hg.mozilla.org/try/rev/1ee783d30721cf277a61f7f95ba5633fbe5adadc https://treeherder.mozilla.org/#/jobs?repo=try&revision=1ee783d30721cf277a61f7f95ba5633fbe5adadc
Attachment #8743759 -
Attachment is obsolete: true
Assignee | ||
Comment 62•8 years ago
|
||
Thanks Joel! Much appreciated. testing/mozharness/configs/builds/releng_base_windows_32_builds.py (& 64): now omitted from the patch. testing/taskcluster/scripts/builder/build-windows.cmd: the check on NEED_XVFB flag causes the build to script to exit with an error if set. Not strictly necessary, but we aren't saying "use xfvb" rather, "fail if xfvb set" will pull gaia if thats not needed. yes, happy to add generate-build-stats (not that I know what it does). testing/taskcluster/tasks/builds/base_win32.yml: I will experiment with removing these variable settings completely, because I believe its taken care of by logic further down (after tooltool pulls in VS2015)
Comment 63•8 years ago
|
||
generate-build-stats gives num_constructors from the build, we should leave it off for now, it might not be needed on windows- I can follow up with the build team and other interested parties if they want it. Looking forward to the next patch, feel free to ask for feedback, myself, chmanchester, gbrown, or whoever you would normally go to :)
Comment 64•8 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #32) > The vs2015 archive in tooltool contains *everything* Microsoft needed to > build. No Visual Studio or SDKs need to be installed at the system level. > > It is possible to build Firefox with just a MozillaBuild install and the > vs2015 tooltool archive. (Although there is a bug in MozillaBuild where a > vcruntimeXXX.dll file is missing and I /think/ yasm.exe fails to run, so > watch out for that.) See bug 1267975 about this.
Comment 65•8 years ago
|
||
For reference, my latest green build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=43e6cfaf8d36a5cdec6c57851c40ce826d78fa00 Please note that is still using `mach build -j1` rather than `mach build` (bug 1269809) and does not check the build with `make check` after the build completes (bug 1274676). In the end we should go with Rob's builds/AMIs etc - but this try push might be useful reference material when we do code review.
Assignee | ||
Comment 66•8 years ago
|
||
Attachment #8743795 -
Attachment is obsolete: true
Assignee | ||
Comment 67•8 years ago
|
||
Assignee | ||
Updated•8 years ago
|
Attachment #8758702 -
Attachment is obsolete: true
Assignee | ||
Comment 68•8 years ago
|
||
Attachment #8742765 -
Attachment is obsolete: true
Attachment #8742766 -
Attachment is obsolete: true
Attachment #8743208 -
Attachment is obsolete: true
Attachment #8759220 -
Attachment is obsolete: true
Assignee | ||
Comment 69•8 years ago
|
||
Attachment #8714742 -
Attachment is obsolete: true
Attachment #8759223 -
Attachment is obsolete: true
Assignee | ||
Comment 70•8 years ago
|
||
Attachment #8760218 -
Attachment is obsolete: true
Assignee | ||
Comment 71•8 years ago
|
||
Attachment #8760226 -
Attachment is obsolete: true
Comment 72•8 years ago
|
||
This will be the last part to land, after all dependent bugs have landed. This is the change that just enables the Windows™ 32/64 bit opt/debug builds of Firefox Desktop in TaskCluster.
Attachment #8760682 -
Attachment is obsolete: true
Attachment #8762495 -
Flags: review?(bstack)
Comment 73•8 years ago
|
||
@bstack Note, initially we just want to land the builds on try, not on mozilla-inbound / mozilla-central or elsewhere. After we see that things are working well on try, we can enable other branches.
Comment 74•8 years ago
|
||
Comment on attachment 8762495 [details] [diff] [review] bug1244750_gecko_v1.patch As bstack is going to be stuck on a workweek, I'll hand to garndt as I know he loves reviews. :)
Attachment #8762495 -
Flags: review?(bstack) → review?(garndt)
Comment 75•8 years ago
|
||
Comment on attachment 8762495 [details] [diff] [review] bug1244750_gecko_v1.patch Review of attachment 8762495 [details] [diff] [review]: ----------------------------------------------------------------- As long as these tasks exist in tree, this looks good to me.
Attachment #8762495 -
Flags: review?(garndt) → review+
Comment 76•8 years ago
|
||
Per IRC conversation, it appears that this patch is taking a conservative approach to enabling these builds and limiting them to try. What would be done instead is to enable them in base_jobs.yml instead of try/job_flags.yml and have them run on all trees, BUT either remove task.extra.treeherderEnv entirely from the task definition (will default to staging) or make it only list staging there so that way the jobs only appear on treeherder staging. This way you have a lot more datapoints to work with to determine stability, accuracy, etc.
Comment 77•8 years ago
|
||
Incorporated review comments from garndt. Thanks Greg!
Attachment #8762495 -
Attachment is obsolete: true
Attachment #8762692 -
Flags: review?(garndt)
Comment 78•8 years ago
|
||
Now with `tier: 2` specified in yaml file for treeherder metadata.
Attachment #8762741 -
Flags: review?(garndt)
Updated•8 years ago
|
Attachment #8762692 -
Attachment is obsolete: true
Attachment #8762692 -
Flags: review?(garndt)
Comment 79•8 years ago
|
||
Comment on attachment 8762741 [details] [diff] [review] bug1244750_gecko_v3.patch you might want to push to try just to ensure it behaves as you expect, but I think this looks good.
Attachment #8762741 -
Flags: review?(garndt) → review+
Comment 80•8 years ago
|
||
(In reply to Greg Arndt [:garndt] from comment #79) > Comment on attachment 8762741 [details] [diff] [review] > bug1244750_gecko_v3.patch > > you might want to push to try just to ensure it behaves as you expect, but I > think this looks good. Try push here: https://treeherder.allizom.org/#/jobs?repo=try&revision=39209ef0fb22d50fb58f83e5c249cd4be8dde495
Comment 81•8 years ago
|
||
The above url is for *staging* treeherder where we want jobs to appear. And to check it doesn't get added to production treeherder (where we don't want the jobs to appear): https://treeherder.mozilla.org/#/jobs?repo=try&revision=39209ef0fb22
Comment 82•8 years ago
|
||
Note, the builds might not pass since other bugs remain open - but resolving that is handled by bugs hanging off this one....
Comment 83•8 years ago
|
||
Looks great to me, sir.
Comment 85•8 years ago
|
||
Pushed by pmoore@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/d37a9c4dd98f enable windows 32/64 bit builds on taskcluster, report only in treeherder staging env; r=garndt
Comment 86•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/d37a9c4dd98f
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•