Closed Bug 1306614 Opened 3 years ago Closed 3 years ago

mach build fails with artifact builds

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox53 fixed)

RESOLVED FIXED
mozilla53
Tracking Status
firefox53 --- fixed

People

(Reporter: pascalc, Assigned: chmanchester)

References

Details

Attachments

(2 files)

Attached file output.txt
Following the instructions in https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Artifact_builds on a fresh mozilla-central clone, the build fails immediately if I have ac_add_options --enable-artifact-builds in my mozconfig.

----------------------------------------
Here is the error message:
 Error running mach:
 
     ['--log-no-times', 'artifact', 'install']
 
 The error occurred in code that was called by the mach command. This is either
 a bug in the called code itself or in the way that mach is calling it.
 
 You should consider filing a bug for this issue.
 
 If filing a bug, please include the full output of mach, including this error
 message.
 
 The details of the failure are as follows:
 
 AttributeError: 'unicode' object has no attribute 'iteritems'

----------------------------------------

I am attaching to this bug the full output as a txt file.

Following the same instructions, it worked fine the last time I did that like 3 weeks ago.
So you don't have to open the attachment:

 0:03.58 AttributeError: 'unicode' object has no attribute 'iteritems'
 0:03.58 
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 1552, in artifact_install
 0:03.58     return artifacts.install_from(source, self.distdir)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1052, in install_from
 0:03.58     return self.install_from_recent(distdir)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1015, in install_from_recent
 0:03.58     return self._install_from_hg_pushheads(hg_pushheads, distdir)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1001, in _install_from_hg_pushheads
 0:03.58     self._job, hg_hash)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 917, in find_pushhead_artifacts
 0:03.58     urls = task_cache.artifact_urls(tree, job, pushhead)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 452, in wrapper
 0:03.58     result = method(self, *args, **kwargs)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 641, in artifact_urls
 0:03.58     url = self._queue.buildUrl('getLatestArtifact', taskId, artifact_name)
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/obj-x86_64-pc-linux-gnu/_virtualenv/lib/python2.7/site-packages/taskcluster/baseclient.py", line 177, in buildUrl
 0:03.58     for key, value in six.iteritems(replDict):
 0:03.58   File "/home/pascalc/repos/hg/mozilla-central/obj-x86_64-pc-linux-gnu/_virtualenv/lib/python2.7/site-packages/six.py", line 599, in iteritems
 0:03.58     return d.iteritems(**kw)
I can't reproduce this on my Macbook, but it looks related to bug 1297585. It would be helpful to post the log output from a clobber, which should include logging of what packages get installed to the virtualenv. It also appears you're running |./mach artifact install| directly, does the same error occur if you use |./mach build|?
(In reply to Chris Manchester (:chmanchester) from comment #2)
> I can't reproduce this on my Macbook, but it looks related to bug 1297585.
> It would be helpful to post the log output from a clobber, which should
> include logging of what packages get installed to the virtualenv. It also
> appears you're running |./mach artifact install| directly, does the same
> error occur if you use |./mach build|?

I used ./mach build (didn't know about |./mach artifact install|)
There is no output to |./mach clobber| is that the command that you meant?
I am seeing this same issue right now when building locally.  I am at changeset https://hg.mozilla.org/mozilla-central/rev/34fce7c12173 (current tip of central).  I am using git cinnabar in my local repo.

./mach clobber did not fix the error.

$ ./mach build
 0:00.39 /usr/bin/make -f client.mk -s
 0:01.67 Adding client.mk options from /Users/jryans/projects/mozilla/gecko-dev-2/.mozconfig:
 0:01.67     AUTOCONF=autoconf213
 0:01.67     MOZ_OBJDIR=/Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact
 0:01.67     OBJDIR=/Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact
 0:01.67     FOUND_MOZCONFIG=/Users/jryans/projects/mozilla/gecko-dev-2/.mozconfig
 0:08.17 Attempting to find a pushhead containing 34fce7c12173bdd6dda54c2ebf6d344252f1ac48 on mozilla-central.
 0:08.17 Attempting to find a pushhead containing 34fce7c12173bdd6dda54c2ebf6d344252f1ac48 on integration/mozilla-inbound.
 0:08.17 Attempting to find a pushhead containing 34fce7c12173bdd6dda54c2ebf6d344252f1ac48 on releases/mozilla-aurora.
 0:08.83 Retrieving the last 50 pushheads starting with id 84302 on integration/mozilla-inbound
 0:08.83 Retrieving the last 50 pushheads starting with id 30986 on mozilla-central
 0:10.38 Error running mach:
 0:10.38
 0:10.38     ['--log-no-times', 'artifact', 'install']
 0:10.38
 0:10.38 The error occurred in code that was called by the mach command. This is either
 0:10.38 a bug in the called code itself or in the way that mach is calling it.
 0:10.38
 0:10.38 You should consider filing a bug for this issue.
 0:10.38
 0:10.38 If filing a bug, please include the full output of mach, including this error
 0:10.38 message.
 0:10.38
 0:10.41 The details of the failure are as follows:
 0:10.41
 0:10.41 AttributeError: 'unicode' object has no attribute 'iteritems'
 0:10.41
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/mach_commands.py", line 1559, in artifact_install
 0:10.41     return artifacts.install_from(source, self.distdir)
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 1064, in install_from
 0:10.41     return self.install_from_recent(distdir)
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 1026, in install_from_recent
 0:10.41     return self._install_from_hg_pushheads(hg_pushheads, distdir)
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 1012, in _install_from_hg_pushheads
 0:10.41     urls = self.find_pushhead_artifacts(task_cache, self._job, tree, hg_hash)
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 927, in find_pushhead_artifacts
 0:10.41     urls = task_cache.artifact_urls(tree, job, pushhead, self._download_symbols)
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 488, in wrapper
 0:10.41     result = method(self, *args, **kwargs)
 0:10.41   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 663, in artifact_urls
 0:10.41     url = self._queue.buildUrl('getLatestArtifact', taskId, artifact_name)
 0:10.41   File "/usr/local/lib/python2.7/site-packages/taskcluster/baseclient.py", line 177, in buildUrl
 0:10.41     for key, value in six.iteritems(replDict):
 0:10.41   File "/usr/local/lib/python2.7/site-packages/six.py", line 599, in iteritems
 0:10.41     return d.iteritems(**kw)
 0:10.41 make[4]: *** [recurse_artifact] Error 1
 0:10.41 make[3]: *** [artifact] Error 2
 0:10.41 make[2]: *** [default] Error 2
 0:10.41 make[1]: *** [realbuild] Error 2
 0:10.41 make: *** [build] Error 2
 0:10.42 0 compiler warnings present.
Here's a longer version of the log. :chmanchester said he would take a look at this today.

$ ./mach clobber && ./mach build
 0:00.24 /usr/bin/make -f client.mk -s configure
 0:00.61 client.mk:204: /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact/.mozconfig.mk: No such file or directory
 0:00.94 Clobber not needed.
 0:01.58 cd /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact
 0:01.59 /Users/jryans/projects/mozilla/gecko-dev-2/configure
 0:01.76 Creating Python environment
 0:03.74 New python executable in /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact/_virtualenv/bin/python2.7
 0:03.74 Also creating executable in /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact/_virtualenv/bin/python
 0:03.74 Installing setuptools, pip, wheel...done.
 0:03.99 running build_ext
 0:03.99 copying build/lib.macosx-10.11-x86_64-2.7/psutil/_psutil_osx.so -> psutil
 0:03.99 copying build/lib.macosx-10.11-x86_64-2.7/psutil/_psutil_posix.so -> psutil
 0:03.99
 0:03.99 Reexecuting in the virtualenv
 0:04.31 Adding configure options from /Users/jryans/projects/mozilla/gecko-dev-2/.mozconfig
 0:04.31   --enable-application=browser
 0:04.31   --disable-crashreporter
 0:04.31   --enable-debug-js-modules
 0:04.31   --disable-debug
 0:04.31   --enable-optimize
 0:04.31   --disable-strip
 0:04.31   --disable-install-strip
 0:04.31   --enable-crashreporter
 0:04.31   --enable-artifact-builds
 0:04.31   MOZ_ARTIFACT=1
 0:04.54 checking for a shell... /bin/sh
 0:04.62 checking for host system type... x86_64-apple-darwin16.1.0
 0:04.62 checking for target system type... x86_64-apple-darwin16.1.0
 0:04.63 checking whether cross compiling... no
 0:04.65 checking for tar... /usr/local/bin/gtar
 0:04.65 checking for unzip... /usr/bin/unzip
 0:04.65 checking for zip... /usr/bin/zip
 0:04.65 checking for the Mozilla API key... no
 0:04.65 checking for the Google API key... no
 0:04.65 checking for the Bing API key... no
 0:04.65 checking for the Adjust SDK key... no
 0:04.66 checking for awk... /usr/local/bin/gawk
 0:04.66 checking for perl... /usr/bin/perl
 0:04.67 checking for minimum required perl version >= 5.006... 5.018002
 0:04.70 checking for full perl installation... yes
 0:04.70 checking for gmake... /Applications/Xcode.app/Contents/Developer/usr/bin/make
 0:04.70 checking for doxygen... not found
 0:04.70 checking for xargs... /usr/bin/xargs
 0:04.70 checking for dsymutil... /usr/bin/dsymutil
 0:04.70 checking for genisoimage... not found
 0:04.70 checking for autoconf... /usr/local/bin/autoconf213
 0:04.99 creating cache ./config.cache
 0:05.05 checking host system type... x86_64-apple-darwin16.1.0
 0:05.08 checking target system type... x86_64-apple-darwin16.1.0
 0:05.11 checking build system type... x86_64-apple-darwin16.1.0
 0:05.42 checking for iOS target... no
 0:05.42 /Users/jryans/projects/mozilla/gecko-dev-2/old-configure: line 5726: test: argument expected
 0:05.62 checking for -dead_strip option to ld... yes
 0:05.65 checking NSPR selection... source-tree
 0:05.65 checking if app-specific confvars.sh exists... /Users/jryans/projects/mozilla/gecko-dev-2/browser/confvars.sh
 0:05.82 configure: warning: No assembler or assembly support for libav-fft.  Using unoptimized C routines.
 0:05.84 checking for wget... wget
 0:05.84 configure: warning: Cannot --enable-signmar with --disable-compile-environment
 0:06.10 updating cache ./config.cache
 0:06.10 creating ./config.data
 0:06.19 Creating config.status
 0:06.29 Reticulating splines...
 0:23.23 Finished reading 2945 moz.build files in 7.86s
 0:23.23 Processed into 7879 build config descriptors in 3.63s
 0:23.23 FasterMake+RecursiveMake backend executed in 3.01s
 0:23.23   2028 total backend files; 2028 created; 0 updated; 0 unchanged; 0 deleted
 0:23.23 Total wall time: 16.95s; CPU time: 11.64s; Efficiency: 69%; Untracked: 2.44s
Configure complete!
Be sure to run |mach build| to pick up any changes
 0:23.58 /usr/bin/make -f client.mk -s
 0:25.03 Adding client.mk options from /Users/jryans/projects/mozilla/gecko-dev-2/.mozconfig:
 0:25.03     AUTOCONF=autoconf213
 0:25.03     MOZ_OBJDIR=/Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact
 0:25.03     OBJDIR=/Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-release-artifact
 0:25.03     FOUND_MOZCONFIG=/Users/jryans/projects/mozilla/gecko-dev-2/.mozconfig
 0:31.57 Attempting to find a pushhead containing 8387a4ada9a5c4cab059d8fafe0f8c933e83c149 on mozilla-central.
 0:31.57 Attempting to find a pushhead containing 8387a4ada9a5c4cab059d8fafe0f8c933e83c149 on integration/mozilla-inbound.
 0:31.57 Attempting to find a pushhead containing 8387a4ada9a5c4cab059d8fafe0f8c933e83c149 on releases/mozilla-aurora.
 0:32.27 Retrieving the last 50 pushheads starting with id 84394 on integration/mozilla-inbound
 0:32.27 Retrieving the last 50 pushheads starting with id 31004 on mozilla-central
 0:33.49 Error running mach:
 0:33.49
 0:33.49     ['--log-no-times', 'artifact', 'install']
 0:33.49
 0:33.49 The error occurred in code that was called by the mach command. This is either
 0:33.49 a bug in the called code itself or in the way that mach is calling it.
 0:33.49
 0:33.49 You should consider filing a bug for this issue.
 0:33.49
 0:33.49 If filing a bug, please include the full output of mach, including this error
 0:33.49 message.
 0:33.49
 0:33.49 The details of the failure are as follows:
 0:33.49
 0:33.49 AttributeError: 'unicode' object has no attribute 'iteritems'
 0:33.49
 0:33.49   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/mach_commands.py", line 1559, in artifact_install
 0:33.49     return artifacts.install_from(source, self.distdir)
 0:33.49   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 1064, in install_from
 0:33.49     return self.install_from_recent(distdir)
 0:33.49   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 1026, in install_from_recent
 0:33.49     return self._install_from_hg_pushheads(hg_pushheads, distdir)
 0:33.49   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 1012, in _install_from_hg_pushheads
 0:33.49     urls = self.find_pushhead_artifacts(task_cache, self._job, tree, hg_hash)
 0:33.49   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 927, in find_pushhead_artifacts
 0:33.49     urls = task_cache.artifact_urls(tree, job, pushhead, self._download_symbols)
 0:33.49   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 488, in wrapper
 0:33.49     result = method(self, *args, **kwargs)
 0:33.50   File "/Users/jryans/projects/mozilla/gecko-dev-2/python/mozbuild/mozbuild/artifacts.py", line 663, in artifact_urls
 0:33.50     url = self._queue.buildUrl('getLatestArtifact', taskId, artifact_name)
 0:33.50   File "/usr/local/lib/python2.7/site-packages/taskcluster/baseclient.py", line 177, in buildUrl
 0:33.50     for key, value in six.iteritems(replDict):
 0:33.50   File "/usr/local/lib/python2.7/site-packages/six.py", line 599, in iteritems
 0:33.50     return d.iteritems(**kw)
 0:33.52 make[4]: *** [recurse_artifact] Error 1
 0:33.52 make[3]: *** [artifact] Error 2
 0:33.52 make[2]: *** [default] Error 2
 0:33.52 make[1]: *** [realbuild] Error 2
 0:33.52 make: *** [build] Error 2
 0:33.56 0 compiler warnings present.
Flags: needinfo?(cmanchester)
Looks like the interesting bit is at the end of the log, where it pulls from a system level copy of taskcluster, instead of using it from the virtualenv.  My virtualenv did not have taskcluster installed when checking with `pip freeze`.  If I activate the virtualenv and install it manually, then everything works.
Here are some more details:

At the system level, I have taskcluster==0.3.4 (because of mozregression CLI tool) installed.
./mach build, wants to use[1] taskcluster==0.0.32.

By adding some logging to `install_pip_package`[2] it skips installing taskcluster in the virtualenv since it thinks it's already there.  

[1]: http://searchfox.org/mozilla-central/rev/957458d8fa2328c2a760dbb30e7f1f1efa55b4d0/python/mozbuild/mozbuild/mach_commands.py#1505
[2]: http://searchfox.org/mozilla-central/rev/957458d8fa2328c2a760dbb30e7f1f1efa55b4d0/python/mozbuild/mozbuild/virtualenv.py#468
I think I see the issue here.
Assignee: nobody → cmanchester
Flags: needinfo?(cmanchester)
Duplicate of this bug: 1317327
Duplicate of this bug: 1297585
Comment on attachment 8815065 [details]
Bug 1306614 - Correctly detect when a conflicting version of a python package is found by the build system's virtualenv.

https://reviewboard.mozilla.org/r/96056/#review96208

Nice find.

Congratulations on becoming our new Python packaging expert!
Attachment #8815065 - Flags: review?(gps) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e9e311eb0f84
Correctly detect when a conflicting version of a python package is found by the build system's virtualenv. r=gps
https://hg.mozilla.org/mozilla-central/rev/e9e311eb0f84
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.