Closed Bug 1244750 Opened 8 years ago Closed 8 years ago

Port 2015 TaskCluster Windows build POC to current mozilla-central default head

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox50 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox50 --- fixed

People

(Reporter: grenade, Assigned: grenade)

References

Details

Attachments

(1 file, 38 obsolete files)

bug1244750_gecko_v3.patch
2.49 KB, patch
garndt
: review+
Details | Diff | Splinter Review 
- adapt POC patch to work with current m-c
- output to include binary builds:
 - 32 bit opt
 - 64 bit opt
 - 32 bit debug
 - 64 bit debug
- determine if a new platform is needed (whether try jobs can run TC/BB builds independently)
> grenade@quadbrat ~/hg/hg.mozilla.org/mozilla-central hg:default:282529 ?+$ hg add .
> adding testing/taskcluster/tasks/builds/firefox_linux_base.yml
> adding testing/taskcluster/tasks/builds/firefox_win32_debug.yml
> adding testing/taskcluster/tasks/builds/firefox_win32_opt.yml
> adding testing/taskcluster/tasks/builds/firefox_win64_debug.yml
> adding testing/taskcluster/tasks/builds/firefox_win64_opt.yml
> adding testing/taskcluster/tasks/builds/firefox_windows_base.yml
> adding testing/taskcluster/tasks/linux_build.yml
> adding testing/taskcluster/tasks/windows_build.yml

> grenade@quadbrat ~/hg/hg.mozilla.org/mozilla-central hg:default:282529 +$ hg commit -m "try: -b o -p win32,win64 -u none -t none"

> grenade@quadbrat ~/hg/hg.mozilla.org/mozilla-central hg:default:282530 $ hg push -f ssh://hg.mozilla.org/try
> pushing to ssh://hg.mozilla.org/try
> searching for changes
> remote: adding changesets                                                                                                                               
> remote: adding manifests
> remote: adding file changes
> remote: added 1 changesets with 20 changes to 20 files (+1 heads)
> remote: recorded push in pushlog
> remote: 
> remote: View your change here:
> remote:   https://hg.mozilla.org/try/rev/1d543c583e70
> remote: 
> remote: Follow the progress of your build on Treeherder:
> remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=1d543c583e70
> remote: recorded changegroup in replication log in 0.056s

        Attachment #8714745 -
        Attachment is obsolete: true

    
    

    
  


  
revert generic-worker version to 1.0.11

        Attachment #8716890 -
        Attachment is obsolete: true

    
    

    
  


  
use new worker type "y-2012"

        Attachment #8719741 -
        Attachment is obsolete: true

    
    

    
  
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284334 $ hg import --no-commit https://bug1244750.bmoattachments.org/attachment.cgi?id=8719743
> applying https://bug1244750.bmoattachments.org/attachment.cgi?id=8719743
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284334 +$ hg add .
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284334 +$ hg commit -m "try: -b do -p win32,win64 -u none -t none"
> grenade@localhost ~/hg/hg.mozilla.org/mozilla-central hg:default:284335 $ hg push -f ssh://hg.mozilla.org/try
> pushing to ssh://hg.mozilla.org/try
> searching for changes
> remote: adding changesets                                                                                                                                                                                                                  
> remote: adding manifests
> remote: adding file changes
> remote: added 5 changesets with 22 changes to 68 files (+1 heads)
> remote: recorded push in pushlog
> remote: 
> remote: View your changes here:
> remote:   https://hg.mozilla.org/try/rev/f5df9f4596dc
> remote:   https://hg.mozilla.org/try/rev/03fc467e1002
> remote:   https://hg.mozilla.org/try/rev/53029769feef
> remote:   https://hg.mozilla.org/try/rev/6ea654cad929
> remote:   https://hg.mozilla.org/try/rev/c2a12e796c42
> remote: 
> remote: Follow the progress of your build on Treeherder:
> remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2a12e796c42
> remote: recorded changegroup in replication log in 0.065s
> 
> [taskcluster:error] Error calling 'stopped' for extendTaskGraph : Error encountered when attempting to extend task graph. Graph server error while extending task graph id bExcSj11R9GH-z_w2FmFjQ : You didn't give the task-graph scopes allowing it define tasks on the queue., [{"message":"You do not have sufficient scopes. This request requires you\nto have one of the following sets of scopes:\n[\n [\n \"queue:define-task:aws-provisioner-v1/y-2012\"\n ],\n [\n \"queue:create-task:aws-provi

    
    

    
  


  

        Attachment #8719743 -
        Attachment is obsolete: true

    
    

    
  
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 ?+$ hg pull
pulling from ssh://hg.mozilla.org/mozilla-central
searching for changes
no changes found
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 ?+$ hg update -r default -C
13 files updated, 0 files merged, 0 files removed, 0 files unresolved
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 ?$ hg purge
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 $ hg import --no-commit https://bug1244750.bmoattachments.org/attachment.cgi?id=8722460
applying https://bug1244750.bmoattachments.org/attachment.cgi?id=8722460
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285025 +$ hg commit -m "try: -b do -p win32,win64 -u none -t none"
grenade@dhcppc9 ~/hg/hg.mozilla.org/mozilla-central hg:default:285026 $ hg push -f ssh://hg.mozilla.org/try
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets                                                                                                                                                                                         
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 2 files (+1 heads)
remote: recorded push in pushlog
remote: 
remote: View your change here:
remote:   https://hg.mozilla.org/try/rev/9011535fbd44
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=9011535fbd44
remote: recorded changegroup in replication log in 0.070s

    
    

    
  


  

        Attachment #8722460 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8724019 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8724021 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8736654 -
        Attachment is obsolete: true

        Attachment #8736753 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8736761 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8736769 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8736830 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8737731 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8737772 -
        Attachment is obsolete: true

    
    

    
  
Hey Ted -- could you have a look at this latest patch and the try output? :pmoore, could you have a look too?

I see this in the log output at https://public-artifacts.taskcluster.net/bi9kSwbcQF6fq66PG8q_5A/0/public/logs/command_000003.log:  

17:03:29     INFO -  Creating Python environment
17:03:30     INFO -  New python executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python2.7.exe
17:03:30     INFO -  Also creating executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python.exe
17:03:32     INFO -  Installing setuptools, pip, wheel...done.
17:03:33     INFO -  running build_ext
17:03:33     INFO -  building 'psutil._psutil_windows' extension
17:03:33     INFO -  error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
17:03:33     INFO -  Error processing command. Ignoring because optional. (optional:setup.py:python/psutil:build_ext:--inplace)
17:03:33     INFO -  C:\home\worker\workspace\build\src\python\mozbuild\mozbuild\virtualenv.py:361: UserWarning: Hacking environment to allow binary Python extensions to build. You can make this warning go away by installing Visual Studio 2008. You can download the Express Edition installer from http://go.microsoft.com/?linkid=7729279
17:03:33     INFO -    warnings.warn('Hacking environment to allow binary Python '
17:03:33     INFO -  Reexecuting in the virtualenv
17:03:33     INFO -  Creating Python environment
17:03:33     INFO -  Please use the *system* python to run this script
17:03:33     INFO -  Installing setuptools, pip, wheel...
17:03:33     INFO -    Error [Error 5] Access is denied while executing command c:\home\worker\works...uild\src\obj-firefox - setuptools pip wheel
17:03:33     INFO -  ...Installing setuptools, pip, wheel...done.
17:03:33     INFO -  Traceback (most recent call last):
17:03:33     INFO -    File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 2316, in <module>
17:03:33     INFO -      main()
17:03:33     INFO -    File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 708, in main
17:03:33     INFO -      symlink=options.symlink)
17:03:33     INFO -    File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 941, in create_environment
17:03:33     INFO -      download=download,
17:03:33     INFO -    File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 897, in install_wheel
17:03:33     INFO -      call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
17:03:33     INFO -    File "C:/home/worker/workspace/build/src\python\virtualenv\virtualenv.py", line 748, in call_subprocess
17:03:33     INFO -      cwd=cwd, env=env)
17:03:33     INFO -    File "c:\Python27\Lib\subprocess.py", line 710, in __init__
17:03:33     INFO -      errread, errwrite)
17:03:33     INFO -    File "c:\Python27\Lib\subprocess.py", line 958, in _execute_child
17:03:33     INFO -      startupinfo)
17:03:33     INFO -  WindowsError: [Error 5] Access is denied
17:03:33     INFO -  Traceback (most recent call last):
17:03:33     INFO -    File "C:/home/worker/workspace/build/src/configure.py", line 94, in <module>
17:03:33     INFO -      sys.exit(main(sys.argv))
17:03:33     INFO -    File "C:/home/worker/workspace/build/src/configure.py", line 22, in main
17:03:33     INFO -      sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 172, in run
17:03:33     INFO -      self.exec_file(path)
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 165, in exec_file
17:03:33     INFO -      exec(code, self)
17:03:33     INFO -    File "C:/home/worker/workspace/build/src/moz.configure", line 7, in <module>
17:03:33     INFO -      include('build/moz.configure/init.configure')
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 354, in include_impl
17:03:33     INFO -      self.exec_file(what)
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 165, in exec_file
17:03:33     INFO -      exec(code, self)
17:03:33     INFO -    File "C:/home/worker/workspace/build/src/build/moz.configure/init.configure", line 165, in <module>
17:03:33     INFO -      @advanced
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\configure\__init__.py", line 337, in decorator
17:03:33     INFO -      self._results[func] = func(*resolved_args)
17:03:33     INFO -    File "C:/home/worker/workspace/build/src/build/moz.configure/init.configure", line 215, in virtualenv_python
17:03:33     INFO -      manager.build(python)
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\virtualenv.py", line 424, in build
17:03:33     INFO -      self.create(python)
17:03:33     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\virtualenv.py", line 188, in create
17:03:33     INFO -      'Failed to create virtualenv: %s' % self.virtualenv_root)
17:03:33     INFO -  Exception: Failed to create virtualenv: c:/home/worker/workspace/build/src/obj-firefox\_virtualenv
17:03:33     INFO -  *** Fix above errors and then restart with\
17:03:33     INFO -                 "C:/home/worker/workspace/build/src/mozmake.EXE -f client.mk build"
17:03:33     INFO -  C:/home/worker/workspace/build/src/client.mk:382: recipe for target 'configure' failed
17:03:33     INFO -  mozmake.EXE[2]: *** [configure] Error 1
17:03:33     INFO -  mozmake.EXE[2]: Leaving directory 'C:/home/worker/workspace/build/src'
17:03:33     INFO -  C:/home/worker/workspace/build/src/client.mk:396: recipe for target 'C:/home/worker/workspace/build/src/obj-firefox/Makefile' failed
17:03:33     INFO -  mozmake.EXE[1]: *** [C:/home/worker/workspace/build/src/obj-firefox/Makefile] Error 2
17:03:33     INFO -  mozmake.EXE[1]: Leaving directory 'C:/home/worker/workspace/build/src'
17:03:33     INFO -  client.mk:181: recipe for target 'build' failed
17:03:33     INFO -  mozmake.EXE: *** [build] Error 2
17:03:33     INFO -  0 compiler warnings present.
17:03:34     INFO -  2
17:03:34    ERROR - Return code: 1
17:03:34  WARNING - setting return code to 2
17:03:34    FATAL - 'mach build' did not run successfully. Please check log for errors.
Flags: needinfo?(ted)

    
    

    
  
I think gps is better equipped to help diagnose this try error, but if he doesn't have time for it or can't get it sorted I will take a look. (We're both in TRIBE all day tomorrow.)
Flags: needinfo?(gps)

    
    

    
  


  
leave buildbot intact...

        Attachment #8738610 -
        Attachment is obsolete: true

        Attachment #8738925 -
        Attachment is patch: true

    
    

    
  
My debug efforts for error shown in comment 21:
 - Got rid of the "Microsoft Visual C++ 9.0" warning by setting all the lib/include/path environment variables that point to directories under the vs2015 install. This didn't affect the virtualenv call (or stop it failing).
 - Modified line 188 of src/python/mozbuild/mozbuild/virtualenv.py to read "'Failed to create virtualenv: %s (%s)' % (self.virtualenv_root, args))" so that i could see what call was failing.
 - The failing call turned out to be (fails with same error when run manually): 
c:\\home\\worker\\workspace\\build\\src\\obj-firefox\\_virtualenv\\Scripts\\python.exe C:/home/worker/workspace/build/src\\python\\virtualenv\\virtualenv.py --no-download c:/home/worker/workspace/build/src/obj-firefox\\_virtualenv
 - When run with system python manually (c:\Python27\python.exe C:/home/worker/workspace/build/src\\python\\virtualenv\\virtualenv.py --no-download c:/home/worker/workspace/build/src/obj-firefox\\_virtualenv), call succeeds.
- When virtualenv.py is modified to call system python instead, it still fails.

    
    

    
  


  

        Attachment #8738925 -
        Attachment is obsolete: true

        Attachment #8739049 -
        Attachment is obsolete: true

    
    

    
  
For anyone who wants to help debug this, you can create a windows builder from a vanilla amazon ec2 windows 2012 r2 ami by running the following two lines of powershell:

> Set-ExecutionPolicy RemoteSigned -f
> Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/MozRelOps/OpenCloudConfig/master/userdata/win2012-vs2015.ps1')

Then run this script in a cmd prompt:
https://gist.github.com/grenade/1e63fcb938ff4fc22924da23127e1d81#file-build-firefox-cmd

    
    

    
  
(In reply to Rob Thijssen (:grenade - GMT) from comment #25)
> vs 2015 lib/include/path env vars:
> https://hg.mozilla.org/try/rev/f8ae5c527fc56349ed8d3789b3435c949f98ef0b

I'm not a fan of this patch because it is reimplementing logic in the in-tree mozconfigs. The order of directories for the MSVC toolchain environment variables is pretty sensitive. I'd prefer we not have this logic implemented in multiple places.

    
    

    
  
The configure rewrite recently changed the way virtualenvs are installed. The error in comment #21 possibly points to a regression from that refactor.

What's alarming about comment #21 is "Creating Python environment" is printed twice. Specifically, the wonkiness is:

16:47:24     INFO - Calling ['C:\\Python27\\python.exe', 'mach', '--log-no-times', 'build', '-v'] with output_timeout 4800
17:03:04     INFO -  C:/home/worker/workspace/build/src/configure
17:03:29     INFO -  Creating Python environment
17:03:30     INFO -  New python executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python2.7.exe
17:03:30     INFO -  Also creating executable in c:\home\worker\workspace\build\src\obj-firefox\_virtualenv\Scripts\python.exe
17:03:32     INFO -  Installing setuptools, pip, wheel...done.
17:03:33     INFO -  Reexecuting in the virtualenv
# This is where things get weird
17:03:33     INFO -  Creating Python environment
17:03:33     INFO -  Please use the *system* python to run this script

`configure` is supposed to create the virtualenv then re-execute itself from the newly-created virtualenv. However, when it re-executes itself, it attempts to install the virtualenv over the existing virtualenv (it is somehow not detecting it is running from the just-created virtualenv). This leads to a permission denied error because Windows won't let you modify/replace an executable that is running.

glandium: since this is likely fallout from configure rewrite, can you take a look?
Flags: needinfo?(gps) → needinfo?(mh+mozilla)

    
    

    
  
Greg: re comment 29, it'd be great to have a chat when you have time about the changes introduced by having vs2015 downloaded via tooltool. In response to that change, I dropped the installation of any Visual Studio version to the taskcluster worker type. We had previously been installing either 2013 or 2015 community edition.

The patch to set lib/include paths was more because of my misunderstanding of how that all works now, and motivated by wanting to get rid of environment variables that reference `C:\Program Files (x86)\Visual Studio 14.0` which no longer exists. If we don't need them because they're already in mozconfig, i'll just rip'em back out.

I'd also like to understand if the builders/workertypes still need installations of windows sdks or if that requirement is deprecated too.
Flags: needinfo?(gps)

    
    

    
  
The vs2015 archive in tooltool contains *everything* Microsoft needed to build. No Visual Studio or SDKs need to be installed at the system level.

It is possible to build Firefox with just a MozillaBuild install and the vs2015 tooltool archive. (Although there is a bug in MozillaBuild where a vcruntimeXXX.dll file is missing and I /think/ yasm.exe fails to run, so watch out for that.)

The in-tree mozconfigs (e.g. build/win64/mozconfig.vs2015) set all the environment variables needed to find the MSVC and SDK files from the tooltool archive.
Flags: needinfo?(gps)

    
    

    
  
Bug 1255585 sounds like a candidate.
Flags: needinfo?(mh+mozilla) → needinfo?(cmanchester)

    
    

    
  
The symptom looks related (the virtualenv is always considered invalid). I'll push to try with some diagnostic logging to see why that might be.
Flags: needinfo?(ted)
Flags: needinfo?(cmanchester)

    
    

    
  
The difference between the TC builds and buildbot builds causing this failure is that TC is calling os.path.getsize on the Python interpreter at c:\Python27\python2.7.exe and finding its size is 0, while buildbot is calling os.path.getsize on the interpreter at c:\mozilla-build\python27\python2.7.exe and finding an actual size. Why this is, I'm not sure.

We compare this size to the size of the Python interpreter in the objdir virtualenv to attempt to detect incompatible Python versions, and consider the virtualenv to be invalid when they don't match.

    
    

    
  
Thanks Chris!
That solves it for me. The TC builders had a symlink at c:\Python27\python2.7.exe pointing to c:\Python27\python.exe. That's now patched to be a copy of the executable instead.

    
    

    
  


  

        Attachment #8739053 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8739905 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 8739921 [details] [diff] [review]
patch rewrite for mozharness builds

Review of attachment 8739921 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/builds/releng_base_windows_32_builds.py
@@ +30,4 @@
>          ],
>          "buildbot": [
>              sys.executable,
> +            os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build', 'buildbotve', 'scripts', 'buildbot')

Do we need the reference to os.path.sep here, or would this also work? If it can be removed, note there are some more references below that should also be updated.

os.path.join(os.getenv('SystemDrive', 'C:'), 'mozilla-build', 'buildbotve', 'scripts', 'buildbot')

@@ +38,4 @@
>          ],
>          'virtualenv': [
>              sys.executable,
> +            os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build', 'buildbotve', 'virtualenv.py')

What is the reason to use os.path.join rather than a literal backslash? Isn't this config only useful for a Windows machine, which always uses a backslash?

::: testing/taskcluster/scripts/builder/build-windows.cmd
@@ +13,5 @@
> +::if not defined MH_CUSTOM_BUILD_VARIANT_CFG set MH_CUSTOM_BUILD_VARIANT_CFG=
> +if not defined MH_BRANCH                   set MH_BRANCH=mozilla-central
> +if not defined MH_BUILD_POOL               set MH_BUILD_POOL=staging
> +
> +if not defined WORKSPACE                   set WORKSPACE=%SystemDrive%\home\worker\workspace

What is WORKSPACE for? Should this be in the task folder created by the worker?

@@ +43,5 @@
> +
> +set custom_build_variant_cfg_flag=
> +:: if defined MH_CUSTOM_BUILD_VARIANT_CFG set custom_build_variant_cfg_flag=--custom-build-variant-cfg=%MH_CUSTOM_BUILD_VARIANT_CFG%
> +
> +::if "%CHECKOUT_GAIA%" == "true" (

Do we still need this?

::: testing/taskcluster/tasks/windows_build.yml
@@ +8,5 @@
> +  routes:
> +    - 'index.buildbot.branches.{{project}}.{{build_name}}'
> +    - 'index.buildbot.revisions.{{head_rev}}.{{project}}.{{build_name}}'
> +
> +  #routes:

did you mean to comment this out?

@@ +22,5 @@
> +    # All builds share a common artifact directory for ease of uploading.
> +    #artifacts:
> +      #'public/build':
> +        #type: directory
> +        #path: 'C:\\home\\worker\\artifacts\\'

Should this commented section be here? If it should be here, shouldn't it use the Task directory, not C:\home\worker\artifacts?

::: testing/windows/desktop-build/checkout-sources.cmd
@@ +49,5 @@
> +if exist %GECKO_DIR% rmdir %GECKO_DIR% /s /q || echo ERROR && exit /b
> +mkdir %GECKO_DIR% || echo ERROR && exit /b
> +hg clone -U %GECKO_BASE_REPOSITORY% %GECKO_DIR% || echo ERROR && exit /b
> +hg pull -u -R %GECKO_DIR% --rev %GECKO_HEAD_REV% %GECKO_HEAD_REPOSITORY% || echo ERROR && exit /b
> +hg update -R %GECKO_DIR% --rev %GECKO_HEAD_REV% || echo ERROR && exit /b

Won't this script always clone the repositories, rather than using a vcs cache?

    
    

    
  


  

        Attachment #8739921 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8741364 -
        Attachment is obsolete: true

    
    

    
  
Latest build failure:

11:00:34     INFO -  js\src\ctypes\libffi> ********************************************************
11:00:34     INFO -  js\src\ctypes\libffi> * WARNING: Don't know the best CFLAGS for this system  *
11:00:34     INFO -  js\src\ctypes\libffi> * Use ./configure CFLAGS=... to specify your own flags *
11:00:34     INFO -  js\src\ctypes\libffi> * (otherwise, a default of CFLAGS=-O3 will be used)    *
11:00:34     INFO -  js\src\ctypes\libffi> ********************************************************
11:00:34     INFO -  js\src\ctypes\libffi>
11:00:34     INFO -  js\src\ctypes\libffi> checking whether C compiler accepts -O3... yes
11:00:34     INFO -  js\src\ctypes\libffi> checking CFLAGS for maximum warnings... -Wall
11:00:34     INFO -  js\src\ctypes\libffi> checking whether to enable maintainer-specific portions of Makefiles... no
11:00:34     INFO -  js\src\ctypes\libffi> checking sys/mman.h usability... no
11:00:34     INFO -  js\src\ctypes\libffi> checking sys/mman.h presence... no
11:00:34     INFO -  js\src\ctypes\libffi> checking for sys/mman.h... no
11:00:34     INFO -  js\src\ctypes\libffi> checking for mmap... no
11:00:34     INFO -  js\src\ctypes\libffi> checking for sys/mman.h... (cached) no
11:00:34     INFO -  js\src\ctypes\libffi> checking for mmap... (cached) no
11:00:34     INFO -  js\src\ctypes\libffi> checking for ANSI C header files... (cached) yes
11:00:34     INFO -  js\src\ctypes\libffi> checking for memcpy... no
11:00:34     INFO -  js\src\ctypes\libffi> checking for size_t... yes
11:00:34     INFO -  js\src\ctypes\libffi> checking for working alloca.h... no
11:00:34     INFO -  js\src\ctypes\libffi> checking for alloca... yes
11:00:34     INFO -  js\src\ctypes\libffi> checking size of double... 8
11:00:34     INFO -  js\src\ctypes\libffi> checking size of long double... 8
11:00:34     INFO -  js\src\ctypes\libffi> checking whether byte ordering is bigendian... no
11:00:34     INFO -  js\src\ctypes\libffi> checking assembler .cfi pseudo-op support... no
11:00:34     INFO -  js\src\ctypes\libffi> checking for _ prefix in compiled symbols... no
11:00:34     INFO -  js\src\ctypes\libffi> configure: updating cache c:/home/worker/workspace/build/src/obj-firefox/js/src/ctypes/libffi/config.cache
11:00:34     INFO -  js\src\ctypes\libffi> checking that generated files are newer than configure... done
11:00:34     INFO -  js\src\ctypes\libffi> configure: creating ./config.status
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating include/Makefile
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating include/ffi.h
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating Makefile
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating testsuite/Makefile
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating man/Makefile
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating libffi.pc
11:00:34     INFO -  js\src\ctypes\libffi> config.status: creating fficonfig.h
11:00:34     INFO -  js\src\ctypes\libffi> config.status: linking C:/home/worker/workspace/build/src/js/src/ctypes/libffi/src/x86/ffitarget.h to include/ffitarget.h
11:00:34     INFO -  js\src\ctypes\libffi> config.status: executing buildir commands
11:00:34     INFO -  js\src\ctypes\libffi> config.status: skipping top_srcdir/Makefile - not created
11:00:34     INFO -  js\src\ctypes\libffi> config.status: executing depfiles commands
11:00:34     INFO -  js\src\ctypes\libffi> config.status: executing libtool commands
11:00:34     INFO -  js\src\ctypes\libffi> config.status: executing include commands
11:00:34     INFO -  js\src\ctypes\libffi> config.status: executing src commands
11:00:36     INFO -  Reticulating splines...
11:00:36     INFO -  Traceback (most recent call last):
11:00:36     INFO -    File "config.status", line 992, in <module>
11:00:36     INFO -      config_status(**args)
11:00:36     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\config_status.py", line 156, in config_status
11:00:36     INFO -      definitions = list(definitions)
11:00:36     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\frontend\emitter.py", line 165, in emit
11:00:36     INFO -      for out in output:
11:00:36     INFO -    File "c:\home\worker\workspace\build\src\python\mozbuild\mozbuild\frontend\reader.py", line 1066, in read_mozbuild
11:00:36     INFO -      raise bre
11:00:36     INFO -  mozbuild.frontend.reader.BuildReaderError:
11:00:36     INFO -  ==============================
11:00:36     INFO -  ERROR PROCESSING MOZBUILD FILE
11:00:36     INFO -  ==============================
11:00:36     INFO -  The error occurred while processing the following file:
11:00:36     INFO -      C:/home/worker/workspace/build/src/moz.build
11:00:36     INFO -  The underlying problem is an illegal file access. This is likely due to trying to access a file outside of the top source directory.
11:00:36     INFO -  The path whose access was denied is:
11:00:36     INFO -      C:/home/worker/workspace/build/src/js/src/moz.build
11:00:36     INFO -  Modify the script to not access this file and try again.
11:00:36     INFO -  Creating config.status
11:00:36     INFO -  *** Fix above errors and then restart with\
11:00:36     INFO -                 "C:/home/worker/workspace/build/src/mozmake.EXE -f client.mk build"
11:00:36     INFO -  C:/home/worker/workspace/build/src/client.mk:370: recipe for target 'configure' failed
11:00:36     INFO -  mozmake.EXE[2]: *** [configure] Error 1
11:00:36     INFO -  mozmake.EXE[2]: Leaving directory 'C:/home/worker/workspace/build/src'
11:00:36     INFO -  C:/home/worker/workspace/build/src/client.mk:384: recipe for target 'C:/home/worker/workspace/build/src/obj-firefox/Makefile' failed
11:00:36     INFO -  mozmake.EXE[1]: *** [C:/home/worker/workspace/build/src/obj-firefox/Makefile] Error 2
11:00:36     INFO -  mozmake.EXE[1]: Leaving directory 'C:/home/worker/workspace/build/src'
11:00:36     INFO -  client.mk:168: recipe for target 'build' failed
11:00:36     INFO -  mozmake.EXE: *** [build] Error 2
11:00:36     INFO -  1 compiler warnings present.
11:00:36     INFO -  2
11:00:36    ERROR - Return code: 1
11:00:36  WARNING - setting return code to 2
11:00:36    FATAL - 'mach build' did not run successfully. Please check log for errors.

    
    

    
  
I created a diff between the broken TC env and the working BB env:
http://people.mozilla.org/~pmoore/Screen%20Shot%202016-04-19%20at%2013.53.15.png

I'll upload source files too...

    
    

    
  

      
      
      
      

        Attached file
          
        broken (obsolete)
        — 
      

    


  
Env for failing TC build

    
    

    
  

      
      
      
      

        Attached file
          
        working (obsolete)
        — 
      

    


  
Env for passing BB build

    
    

    
  
I just remembered, I had this error too and solved it a different way! It was a horrid mach bug. Long story short, if you execute mach with an absolute path, you need to specify a lowercase drive letter. I had a problem because I was executing:

C:\\mozilla-build\\msys\\bin\\bash --login %CD%\mach.exe

Instead I now specify a relative path to avoid the drive being included in the path.

The reason for this is when mach tries to work out the topsrcdir variable, it takes inspiration from `$0`, i.e. the command used to call mach. When it compares the location of a mozconfig relative to this location, it expects to find it underneath. However, by that point it has used python function os.getcwd() which returns a lower case drive letter, or at least it does when called from msys.

Since you're calling mozharness, I'm guessing the location of mach is going to be in a mozharness config file. Probably this should be specified in msys format (/c/....) with a lower case drive letter.

Note, to get it working for me, where I call mach directly, I changed the mach call to be:

C:\\mozilla-build\\msys\\bin\\bash --login -c "cd '%CD%'; ./mach build"

Nasty little bug. :)

    
    

    
  
Pushed a new commit to get more debug, to confirm whether this is exactly the same cause that I had, or a subtly different reason:

  * https://treeherder.mozilla.org/#/jobs?repo=try&revision=912139a9c662

    
    

    
  
This is similar to bug 1177634 and should be fixed the same way, in fact, in the same function.

    
    

    
  
Yep, that was it:

0:01.22 CAUSE OF FAILURE: C:/home/worker/workspace/build/src/js/src/moz.build falls outside of c:/home/worker/workspace/build/src

https://public-artifacts.taskcluster.net/eDAjycrnQtW9qQDbdbIWUA/0/public/logs/command_000003.log

    
    

    
  
This is how it is invoked in the failing TC build:

Running command: ['C:\\mozilla-build\\python\\python.exe', 'mach', '--log-no-times', 'build', '-v'] in C:\home\worker\workspace\build\src


This is how it is invoked in the succeeding BB build:

Running command: ['c:\\mozilla-build\\python27\\python.exe', 'mach', '--log-no-times', 'build', '-v'] in c:\builds\moz2_slave\try-w64-0000000000000000000000\build\src


Notice the `C:` vs `c:`.

    
    

    
  


  

        Attachment #8742758 -
        Attachment is obsolete: true

    
    

    
  

      
      
      
      

        Attached file
          
        build looping on RecursiveMake (obsolete)
        — 
      

    


  
So in recent try pushes win64 builds don't complete (win32 is a different issue, related to DirectXSDK).

The logs are never uploaded to treeherder/taskinspector because the build just doesn't finish. An excerpt of the raw log (taken from the worker instance) showing the looping follows:

 Build configuration changed. Regenerating backend.
 c:/Users/Task_1461100687/build/src/obj-firefox/_virtualenv/Scripts/python.exe config.status
 ===============================
 Visual Studio Support Available
 You are building Firefox on Windows. Please help us test the experimental
 Visual Studio project files (yes, IntelliSense works) by running the
 following:
    mach build-backend --backend=VisualStudio
 ===============================
 Reticulating splines...
 Finished reading 2990 moz.build files in 7.75s
 Processed into 9663 build config descriptors in 4.85s
 RecursiveMake backend executed in 24.08s
   2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile
 FasterMake backend executed in 0.58s
   10 total backend files; 0 created; 0 updated; 10 unchanged; 0 deleted
 Total wall time: 62.56s; CPU time: 62.55s; Efficiency: 100%; Untracked: 25.29s
 Build configuration changed. Regenerating backend.
 c:/Users/Task_1461100687/build/src/obj-firefox/_virtualenv/Scripts/python.exe config.status
 ===============================
 Visual Studio Support Available
 You are building Firefox on Windows. Please help us test the experimental
 Visual Studio project files (yes, IntelliSense works) by running the
 following:
    mach build-backend --backend=VisualStudio
 ===============================
 Reticulating splines...
 Finished reading 2990 moz.build files in 7.69s
 Processed into 9663 build config descriptors in 4.80s
 RecursiveMake backend executed in 23.94s
   2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile
 FasterMake backend executed in 0.57s
   10 total backend files; 0 created; 0 updated; 10 unchanged; 0 deleted
 Total wall time: 62.71s; CPU time: 62.71s; Efficiency: 100%; Untracked: 25.71s
 Build configuration changed. Regenerating backend.
 c:/Users/Task_1461100687/build/src/obj-firefox/_virtualenv/Scripts/python.exe config.status
 ===============================
 Visual Studio Support Available
 You are building Firefox on Windows. Please help us test the experimental
 Visual Studio project files (yes, IntelliSense works) by running the
 following:
    mach build-backend --backend=VisualStudio
 ===============================
 Reticulating splines...
 Finished reading 2990 moz.build files in 7.74s
 Processed into 9663 build config descriptors in 4.79s
 RecursiveMake backend executed in 24.09s
   2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile
 FasterMake backend executed in 0.58s
   10 total backend files; 0 created; 0 updated; 10 unchanged; 0 deleted
 Total wall time: 62.71s; CPU time: 62.71s; Efficiency: 100%; Untracked: 25.51s

        Attachment #8743208 -
        Flags: feedback?(cmanchester)

    
    

    
  


  

        Attachment #8742874 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 8743208 [details]
build looping on RecursiveMake

This seems to be related to the fix for the c:\ vs C:\ paths, so we've pulled that out, and instead skip over the check for now. We'll fix it properly once we've got the green builds working.

        Attachment #8743208 -
        Flags: feedback?(cmanchester)
Depends on: 1266123

    
    

    
  
> RecursiveMake backend executed in 24.08s
>   2742 total backend files; 0 created; 0 updated; 2742 unchanged; 0 deleted; 67 -> 1024 Makefile

That is absurdly slow :/

    
    

    
  
when i checked on the instance it was running 87 mozmake processes. I assume the slowness is related to the recursion...

    
    

    
  


  

        Attachment #8743209 -
        Attachment is obsolete: true

        Attachment #8743759 -
        Attachment is patch: true

    
    

    
  
Comment on attachment 8743759 [details] [diff] [review]
patch rewrite for mozharness builds

Review of attachment 8743759 [details] [diff] [review]:
-----------------------------------------------------------------

just some drive by comments...more first impressions than official review.

::: testing/mozharness/configs/builds/releng_base_windows_32_builds.py
@@ +34,4 @@
>          ],
>          "make": [
>              sys.executable,
> +            os.path.join(os.getcwd(), 'build', 'src', 'build', 'pymake', 'make.py')

before this was split to reduce the line length, I would think that would still be desireable.

@@ +38,4 @@
>          ],
>          'virtualenv': [
>              sys.executable,
> +            os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build', 'buildbotve', 'virtualenv.py')

why do we need a os.path.sep here?  I thought os.path.join() would include that automatically?

@@ +44,5 @@
>      'app_ini_path': '%(obj_dir)s/dist/bin/application.ini',
>      # decides whether we want to use moz_sign_cmd in env
>      'enable_signing': True,
>      'enable_ccache': False,
> +    'vcs_share_base': os.path.join(os.getenv('SystemDrive', 'C:') + os.path.sep, 'builds', 'hg-shared'),

can we pull os.getenv('SystemDrive', 'C:') out as a variable to make things more readable?  that would help in many of these calls.  In fact, making something like "moz_build = os.getenv('SystemDrive', 'C:') + os.path.sep, 'mozilla-build'" would make things easier.

::: testing/taskcluster/scripts/builder/build-windows.cmd
@@ +29,5 @@
> +
> +:: test required parameters are supplied
> +if not defined MOZHARNESS_SCRIPT exit /b 1
> +if not defined MOZHARNESS_CONFIG exit /b 1
> +if "%NEED_XVFB%" == "true" exit /b 1

do we have xvfb for windows?  I thought that was a linux thing

@@ +46,5 @@
> +  ::mkdir %WORKSPACE%\build\tools || echo ERROR && exit /b
> +  ::hg clone -U %TOOLS_BASE_REPOSITORY% %WORKSPACE%\build\tools || echo ERROR && exit /b
> +  ::hg pull -u -R %WORKSPACE%\build\tools --rev %TOOLS_HEAD_REV% %TOOLS_HEAD_REPOSITORY% || echo ERROR && exit /b
> +  ::hg update -R %WORKSPACE%\build\tools --rev %TOOLS_HEAD_REV% || echo ERROR && exit /b
> +::)

not sure if we are doing gaia anymore

@@ +73,5 @@
> + --no-upload-files^
> + --no-sendchange^
> + --log-level=debug^
> + --work-dir=%WORKSPACE%\build^
> + --no-action=generate-build-stats^

we needed generate-build-stats for linux builds in tc, I assume we can tweak these defaults when we get the builds working.

::: testing/taskcluster/tasks/builds/base_win32.yml
@@ +10,5 @@
> +  payload:
> +    env:
> +      CommandPromptType: "Cross"
> +      LIB: "C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\x86;"
> +      LIBPATH: "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319;C:\\Program Files (x86)\\Windows Kits\\8.1\\References\\CommonConfiguration\\Neutral;"

we have c:\ hardcoded here, is that ok?

    
    

    
  
Thanks Joel! Much appreciated.

testing/mozharness/configs/builds/releng_base_windows_32_builds.py (& 64): now omitted from the patch.

testing/taskcluster/scripts/builder/build-windows.cmd: the check on NEED_XVFB flag causes the build to script to exit with an error if set. Not strictly necessary, but we aren't saying "use xfvb" rather, "fail if xfvb set"
will pull gaia if thats not needed.

yes, happy to add generate-build-stats (not that I know what it does).

testing/taskcluster/tasks/builds/base_win32.yml: I will experiment with removing these variable settings completely, because I believe its taken care of by logic further down (after tooltool pulls in VS2015)

    
    

    
  
generate-build-stats gives num_constructors from the build, we should leave it off for now, it might not be needed on windows- I can follow up with the build team and other interested parties if they want it.

Looking forward to the next patch, feel free to ask for feedback, myself, chmanchester, gbrown, or whoever you would normally go to :)
Depends on: 1266722
Depends on: 1266785
Depends on: 1267975

    
    

    
  
(In reply to Gregory Szorc [:gps] from comment #32)
> The vs2015 archive in tooltool contains *everything* Microsoft needed to
> build. No Visual Studio or SDKs need to be installed at the system level.
> 
> It is possible to build Firefox with just a MozillaBuild install and the
> vs2015 tooltool archive. (Although there is a bug in MozillaBuild where a
> vcruntimeXXX.dll file is missing and I /think/ yasm.exe fails to run, so
> watch out for that.)

See bug 1267975 about this.
Depends on: 1267992
Depends on: 1269339
Depends on: 1269809
Depends on: 1273549
Depends on: 1273806
Depends on: 1273812
Depends on: 1273815
Depends on: 1273817
Depends on: 1274676
Depends on: 1274844

    
    

    
  
For reference, my latest green build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=43e6cfaf8d36a5cdec6c57851c40ce826d78fa00

Please note that is still using `mach build -j1` rather than `mach build` (bug 1269809) and does not check the build with `make check` after the build completes (bug 1274676).

In the end we should go with Rob's builds/AMIs etc - but this try push might be useful reference material when we do code review.

    
    

    
  


  

        Attachment #8743795 -
        Attachment is obsolete: true

        Attachment #8758702 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8742765 -
        Attachment is obsolete: true

        Attachment #8742766 -
        Attachment is obsolete: true

        Attachment #8743208 -
        Attachment is obsolete: true

        Attachment #8759220 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8714742 -
        Attachment is obsolete: true

        Attachment #8759223 -
        Attachment is obsolete: true

    
    

    
  


  

        Attachment #8760218 -
        Attachment is obsolete: true
Depends on: 1278496

    
    

    
  


  

        Attachment #8760226 -
        Attachment is obsolete: true
No longer depends on: 1269809
Depends on: 1278990
Depends on: 1278995
Depends on: 1278999
Depends on: 1279011
Depends on: 1279016
Depends on: 1279018
Depends on: 1279019
Depends on: 1279021
Depends on: 1279027
No longer depends on: 1278995
Depends on: 1279260
No longer depends on: 1279260

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1244750_gecko_v1.patch (obsolete)
        — 
          — Splinter Review
      

    


  
This will be the last part to land, after all dependent bugs have landed. This is the change that just enables the Windows™ 32/64 bit opt/debug builds of Firefox Desktop in TaskCluster.

        Attachment #8760682 -
        Attachment is obsolete: true

        Attachment #8762495 -
        Flags: review?(bstack)

    
    

    
  
@bstack Note, initially we just want to land the builds on try, not on mozilla-inbound / mozilla-central or elsewhere. After we see that things are working well on try, we can enable other branches.

    
    

    
  
Comment on attachment 8762495 [details] [diff] [review]
bug1244750_gecko_v1.patch

As bstack is going to be stuck on a workweek, I'll hand to garndt as I know he loves reviews. :)

        Attachment #8762495 -
        Flags: review?(bstack) → review?(garndt)

    
    

    
  
Comment on attachment 8762495 [details] [diff] [review]
bug1244750_gecko_v1.patch

Review of attachment 8762495 [details] [diff] [review]:
-----------------------------------------------------------------

As long as these tasks exist in tree, this looks good to me.

        Attachment #8762495 -
        Flags: review?(garndt) → review+

    
    

    
  
Per IRC conversation, it appears that this patch is taking a conservative approach to enabling these builds and limiting them to try.

What would be done instead is to enable them in base_jobs.yml instead of try/job_flags.yml and have them run on all trees, BUT either remove task.extra.treeherderEnv entirely from the task definition (will default to staging) or make it only list staging there so that way the jobs only appear on treeherder staging.  This way you have a lot more datapoints to work with to determine stability, accuracy, etc.

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1244750_gecko_v2.patch (obsolete)
        — 
          — Splinter Review
      

    


  
Incorporated review comments from garndt.

Thanks Greg!

        Attachment #8762495 -
        Attachment is obsolete: true

        Attachment #8762692 -
        Flags: review?(garndt)

    
    

    
  


  
Now with `tier: 2` specified in yaml file for treeherder metadata.

        Attachment #8762741 -
        Flags: review?(garndt)

        Attachment #8762692 -
        Attachment is obsolete: true

        Attachment #8762692 -
        Flags: review?(garndt)

    
    

    
  
Comment on attachment 8762741 [details] [diff] [review]
bug1244750_gecko_v3.patch

you might want to push to try just to ensure it behaves as you expect, but I think this looks good.

        Attachment #8762741 -
        Flags: review?(garndt) → review+

    
    

    
  
(In reply to Greg Arndt [:garndt] from comment #79)
> Comment on attachment 8762741 [details] [diff] [review]
> bug1244750_gecko_v3.patch
> 
> you might want to push to try just to ensure it behaves as you expect, but I
> think this looks good.

Try push here:

https://treeherder.allizom.org/#/jobs?repo=try&revision=39209ef0fb22d50fb58f83e5c249cd4be8dde495

    
    

    
  
The above url is for *staging* treeherder where we want jobs to appear.

And to check it doesn't get added to production treeherder (where we don't want the jobs to appear):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=39209ef0fb22

    
    

    
  
Note, the builds might not pass since other bugs remain open - but resolving that is handled by bugs hanging off this one....

    
    

    
  
Looks great to me, sir.
See Also:  → 1269809
Depends on: 1282482
No longer depends on: 1279027
Depends on: 1280326
No longer depends on: 1274676
No longer depends on: 1279019
Depends on: 1283858
Depends on: 1283917
No longer depends on: 1283917

    
    

    
  
Pushed by pmoore@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d37a9c4dd98f
enable windows 32/64 bit builds on taskcluster, report only in treeherder staging env; r=garndt
Depends on: 1285222

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/d37a9c4dd98f
Status: NEW → RESOLVED
Closed: 8 years ago

          status-firefox50:
          --- → fixed
Resolution: --- → FIXED
No longer blocks: 1285576
Depends on: 1303974
No longer depends on: 1303974
Component: General Automation → General

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: