1369324 - Thunderbird treeherder build busted as of 2017-06-01 - Running setup.py bdist_wheel for blessings: finished with status 'error' - ImportError: No module named six

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Jorg K (CEST = GMT+2)]

https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-win32/1496301211/comm-central-win32-bm73-build1-build21.txt.gz

Building wheels for collected packages: blessings
  Running setup.py bdist_wheel for blessings: started
  Running setup.py bdist_wheel for blessings: finished with status 'error'
  Complete output from command c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\scripts\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\cltbld\\appdata\\local\\temp\\pip-build-q4dab7\\blessings\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d c:\users\cltbld\appdata\local\temp\tmpry6mjypip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\blessings
  copying blessings\tests.py -> build\lib\blessings
  copying blessings\__init__.py -> build\lib\blessings
  running egg_info
  writing blessings.egg-info\PKG-INFO
  writing top-level names to blessings.egg-info\top_level.txt
  writing dependency_links to blessings.egg-info\dependency_links.txt
  reading manifest file 'blessings.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'blessings.egg-info\SOURCES.txt'
  installing to build\bdist.win32\wheel
  running install
  running install_lib
  creating build\bdist.win32
  creating build\bdist.win32\wheel
  creating build\bdist.win32\wheel\blessings
  copying build\lib\blessings\tests.py -> build\bdist.win32\wheel\.\blessings
  copying build\lib\blessings\__init__.py -> build\bdist.win32\wheel\.\blessings
  running install_egg_info
  Copying blessings.egg-info to build\bdist.win32\wheel\.\blessings-1.6-py2.7.egg-info
  running install_scripts
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "c:\users\cltbld\appdata\local\temp\pip-build-q4dab7\blessings\setup.py", line 49, in <module>
      **extra_setup
    File "c:\mozilla-build\python27\Lib\distutils\core.py", line 152, in setup
      dist.run_commands()
    File "c:\mozilla-build\python27\Lib\distutils\dist.py", line 953, in run_commands
      self.run_command(cmd)
    File "c:\mozilla-build\python27\Lib\distutils\dist.py", line 972, in run_command
      cmd_obj.run()
    File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\wheel\bdist_wheel.py", line 235, in run
      self.run_command('install')
    File "c:\mozilla-build\python27\Lib\distutils\cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "c:\mozilla-build\python27\Lib\distutils\dist.py", line 972, in run_command
      cmd_obj.run()
    File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\setuptools\command\install.py", line 61, in run
      return orig.install.run(self)
    File "c:\mozilla-build\python27\Lib\distutils\command\install.py", line 575, in run
      self.run_command(cmd_name)
    File "c:\mozilla-build\python27\Lib\distutils\cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "c:\mozilla-build\python27\Lib\distutils\dist.py", line 972, in run_command
      cmd_obj.run()
    File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\setuptools\command\install_scripts.py", line 17, in run
      import setuptools.command.easy_install as ei
    File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\setuptools\command\easy_install.py", line 49, in <module>
      from setuptools.py27compat import rmtree_safe
    File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\setuptools\py27compat.py", line 7, in <module>
      import six
  ImportError: No module named six
  
  ----------------------------------------
  Running setup.py clean for blessings
c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\pip\_vendor\requests\packages\urllib3\util\ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\lib\site-packages\pip\_vendor\requests\packages\urllib3\util\ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Failed building wheel for blessings
Failed to build blessings

=== And further down ====

    ImportError: No module named six
    
    ----------------------------------------
Python: 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)]
Failure to install packages
Command "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\scripts\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\cltbld\\appdata\\local\\temp\\pip-49yswo-build\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\cltbld\appdata\local\temp\pip-zsrr5r-record\install-record.txt --single-version-externally-managed --compile --install-headers c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\_tests\mozmill-virtualenv\include\site\python2.7\manifestparser" failed with error code 1 in c:\users\cltbld\appdata\local\temp\pip-49yswo-build\
Makefile:29: recipe for target 'mozmill-virtualenv' failed
mozmake.exe[5]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb/mail/test/mozmill'
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/config/recurse.mk:100: recipe for target 'mail/test/mozmill/libs' failed
mozmake.exe[4]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb'
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/config/recurse.mk:32: recipe for target 'libs' failed
mozmake.exe[3]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb'
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/config/rules.mk:519: recipe for target 'default' failed
mozmake.exe[2]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb'
mozmake.exe[5]: *** [mozmill-virtualenv] Error 1
mozmake.exe[4]: *** [mail/test/mozmill/libs] Error 2
mozmake.exe[3]: *** [libs] Error 2
mozmake.exe[2]: *** [default] Error 2
mozmake.exe[1]: *** [build] Error 2
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/client.mk:397: recipe for target 'build' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build'
client.mk:217: recipe for target 'build' failed
mozmake.exe: *** [build] Error 2
program finished with exit code 2

Comment 1

[Jorg K (CEST = GMT+2)]

I wouldn't have a clue where to look and sadly we haven't hired our TB build/release engineer yet.

Comment 2

[Edmund Wong (:ewong)]

Attachment: not-tested-patch

The rationale for this is the following:

1) it busted right after https://hg.mozilla.org/mozilla-central/rev/bdb2387396b4a74dfefb7c983733eed3625e906a landed

2) it's downloading something that I've not seen before (this really
has no weight in here...) nor mentioned in previous logs and this something 
depends on the module 'six' which isn't present in virtualenv.

To Note:

1) I haven't touched the TB build config in a while and looking at the stuff
in the log, I am a tad bit overwhelmed.

2) I think rail and/or nthomas and even aki will probably know more.

3) This is the simplest patch that I can come up with without going
complicated by getting the virtualenv to install six and stuff.

Comment 3

[Jorg K (CEST = GMT+2)]

Edmund, thanks for looking into it. Strangely enough a retriggered job succeeded. So let's see what the daily build does which has just started.

Comment 4

[Jorg K (CEST = GMT+2)]

Aki, can you please take a look here. C-C treeherder is broken apart from Mac. Very similar to bug 1369250.

Comment 5

[ISHIKAWA, Chiaki]

Not only treeherder but the local build was busted. I hope that by now it is fixed (?). I am checking by downloading the latest tree via |make -f client.mk| and compiling locally.

Comment 6

[ISHIKAWA, Chiaki]

(In reply to ISHIKAWA, Chiaki from comment #5)
> Not only treeherder but the local build was busted. I hope that by now it is
> fixed (?). I am checking by downloading the latest tree via |make -f
> client.mk| and compiling locally.

It now looks "FIXED". The local build after a source refresh now worked!

Comment 7

[Aki Sasaki (not active)]

This is https://github.com/pypa/setuptools/issues/1042 .  We fixed it on the Fx side with https://bugzilla.mozilla.org/show_bug.cgi?id=1369250 , which would also fix tb if tb used mozharness.

Since setuptools is fixed upstream now, tb should be fixed.  The long term fix is to avoid downloading setuptools at venv creation, which would involve a similar fix to bug 1369250 (set VIRTUALENV_NO_DOWNLOAD in env) in whatever places are appropriate for tb.

Comment 8

[Jorg K (CEST = GMT+2)]

Indeed, the latest C-C push was successful. Perhaps Edmund can look into the "long term fix".

Comment 9

[Edmund Wong (:ewong)]

(In reply to Jorg K (GMT+2) from comment #3)
> Edmund, thanks for looking into it. Strangely enough a retriggered job
> succeeded. So let's see what the daily build does which has just started.

I'm not entirely sure what fixed it.

Moving TB to mozharness requires a *lot* of work; however, it will require
(aiui) some permission in sticking TB configs in the testing/ part of the
m-c tree  *OR* have mozharness forked into c-c and then TB (and by
extension SM) can run their builds under mozharness.

Comment 10

[ISHIKAWA, Chiaki]

Agha... We had python problem, but now we have rust problem?

make[2]: Circular /home/ishikawa/TB-SRC/comm-central/mozilla/CLOBBER <- /home/ishikawa/TB-SRC/comm-central/mozilla/CLOBBER dependency dropped.
force-cargo-library-build
   Compiling webrender v0.40.0 (file:///home/ishikawa/TB-SRC/comm-central/mozilla/gfx/webrender)
error: struct field shorthands are unstable (see issue #37340)
    --> /home/ishikawa/TB-SRC/comm-central/mozilla/gfx/webrender/src/renderer.rs:1061:13
     |
1061 |             default_font_render_mode,
     |             ^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `webrender`.

To learn more, run the command again with --verbose.
/home/ishikawa/TB-SRC/comm-central/mozilla/config/rules.mk:1024: recipe for target 'force-cargo-library-build' failed
make[4]: *** [force-cargo-library-build] Error 101

OK, it is more like gfx/webrender coding issue, I suppose.
I will investigate and file a separate bug.

Comment 11

[Jorg K (CEST = GMT+2)]

(In reply to ISHIKAWA, Chiaki from comment #10)
> Agha... We had python problem, but now we have rust problem?
We've been compiling with Rust for a while. I'm not sure that you can build without it these days. You could try options ac_add_options --disable-rust and ac_add_options --disable-webrender. But Rust will be compulsory soon (or already is) so it would be better to install it via |mach bootstrap|. In general, please don't post unrelated comments since it makes bugs hard to read.

(In reply to Edmund Wong (:ewong) from comment #9)
> Moving TB to mozharness requires a *lot* of work; however, it will require
> (aiui) some permission in sticking TB configs in the testing/ part of the
> m-c tree  *OR* have mozharness forked into c-c and then TB (and by
> extension SM) can run their builds under mozharness.
I know nothing about the build system, so I don't know how to interpret Aki's comment:
"a similar fix to bug 1369250 (set VIRTUALENV_NO_DOWNLOAD in env) in whatever places are appropriate for tb".
I don't see/know how that relates to mozharness, but yes, bug 1369250, https://hg.mozilla.org/mozilla-central/rev/bdb2387396b4, changed something in /testing/mozharness/mozharness/base/python.py.

Comment 12

[ISHIKAWA, Chiaki]

(In reply to Jorg K (GMT+2) from comment #11)
> (In reply to ISHIKAWA, Chiaki from comment #10)
> > Agha... We had python problem, but now we have rust problem?
> We've been compiling with Rust for a while. I'm not sure that you can build
> without it these days. You could try options ac_add_options --disable-rust
> and ac_add_options --disable-webrender. But Rust will be compulsory soon (or
> already is) so it would be better to install it via |mach bootstrap|. In
> general, please don't post unrelated comments since it makes bugs hard to
> read.
> 

Jorg, yes, I have been using rustc for several months locally now. 
I thought the bug here is a configuration issue caused by some toolchain issue. 
(and Aki's comment seems to confirm this general view.)
So I thought this rustc issue (obviously a version mismatch) was caused by the same root cause. Updating/upgrading some rustc-related files behind our back.

Well, it *is* a version mismatch, but it was not caused by this toolchain issue here.
Gory details are in:
https://bugzilla.mozilla.org/show_bug.cgi?id=1369707
https://bugzilla.mozilla.org/show_bug.cgi?id=1369615

Comment 13

[Tom Prince [:tomprince]]

Attachment: Tell virtualenv not to download virtualenv.

There appear to be two places where virtualenvs are created (based on looking at https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-linux/1496916864/comm-central-linux-bm73-build1-build63.txt.gz

1) https://dxr.allizom.org/comm-central/source/mail/test/resources/installmozmill.py#80-81 which is where the build here failed. I've changed this to pass --no-download. Since this uses a vendored copy of virtualenv we don't need to worry about versions of virtualenv that don't understand the option.

2) https://dxr.allizom.org/comm-central/source/mozilla/build/moz.configure/init.configure#206-207 which sets --no-download already (https://dxr.allizom.org/comm-central/source/mozilla/python/mozbuild/mozbuild/virtualenv.py#191-197)

I've adjusted the former to match the later, but haven't test, since I'm still downloading a copy of mozilla-central.

Comment 14

[ISHIKAWA, Chiaki]

C-C TB tryserver still seems to bomb out.

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=a73e06cd94e0d451d2f839bd53eba84e55e2d614

Oh, I see, I have to add Tom&s patch in comment 13 to my push, correct?

Comment 15

[Jorg K (CEST = GMT+2)]

No. You need to rebase. You're caught by bug 1371300 which fixed a broken build yesterday.

Comment 16

[Jorg K (CEST = GMT+2)]

Comment on attachment 8875822 [details] [diff] [review]
Tell virtualenv not to download virtualenv.

Tom, you'll ask for review once you think that it's good to go, right? Meanwhile let's ask FRG for his opinion. The TB team doesn't have much build expertise, but FRG handles SeaMonkey, so he's across many issues.

Comment 17

[ISHIKAWA, Chiaki]

(In reply to Jorg K (GMT+2) from comment #15)
> No. You need to rebase. You're caught by bug 1371300 which fixed a broken
> build yesterday.

rebase?

I ran |make -f client.mk checkout| only about an hour ago or so before I tried the latest push.
Maybe I missed something???

TIA

Comment 18

[ISHIKAWA, Chiaki]

(In reply to ISHIKAWA, Chiaki from comment #17)
> (In reply to Jorg K (GMT+2) from comment #15)
> > No. You need to rebase. You're caught by bug 1371300 which fixed a broken
> > build yesterday.
> 
> rebase?
> 
> I ran |make -f client.mk checkout| only about an hour ago or so before I
> tried the latest push.
> Maybe I missed something???
> 
> TIA

Between the time I updated the tree by |make -f client.mk checkout| and
the C-C tryserver submission is processed, I think M-C there imported these patches.

	291cefb09dff	Philipp Kewisch — Bug 1347211 - Remove some deprecated functions from calUtils.js. r=MakeMyDay
	4bafb6c2ddaf	Philipp Kewisch — Bug 1347149 - Remove inclusion of calUtils.js and use cal prefix for all calUtils functions. r=MakeMyDay
	693678f602cf	Philipp Kewisch — Bug 1344068 - Use alarmLastAck from parent item in the alarm service. r=Decathlon
	316de32a4217	Jorg K — Keep tooltool manifests in sync (Port 1371372: Update win64 builders to rust 1.18.0 stable). rs=bustage-fix
	998749e6ed4e	Jorg K — Bug 1371300 - Port bug 1346025 to mail: move /python to /third_party/python. rs=bustage-fix CLOSED TREE
	274f513e0e06	Richard Marti — Bug 1369850 - Remove the gap in version text when prefs are in window. r=jorgk
	fb2d9e053f28	tbirdbld — No bug, Automated blocklist update from host bld-linux64-spot-303 - a=blocklist-update
	e8b8d373ef7b	Sebastian Hengst — Bug 1303719 - Use addEventListener(..., {once: true}) in calendar. r=philipp
	16d84314fbf7	Jorg K — Bug 1364977 - remove obs_documentCreated observer in ComposeUnload(). r=aceman
	502e9f359c37	Richard Marti — Bug 1370334 - Port bug 1352366 to TB [Implement new location and search bar appearance]. r=aceman,jorgk


That suggests to me that there could have been similar updates in C-C tree AFTER my upgrade.
I will try updating C-C tree locally again and repush.

TIA

Comment 19

[Aki Sasaki (not active)]

See bug 1369250.

Try

    export VIRTUALENV_NO_DOWNLOAD=1

then rerun.  To automate, set that env var in your builders.

Comment 20

[Jorg K (CEST = GMT+2)]

You need:
998749e6ed4e	Jorg K — Bug 1371300 - Port bug 1346025 to mail: move /python to /third_party/python. rs=bustage-fix 
It clearly says: Bustage fix.

Comment 21

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8875822 [details] [diff] [review]
Tell virtualenv not to download virtualenv.

Jorg, ewong is the build boss:) I have mostly experience with locals builds and not much with tests. That said I think it should work here. TB does not use mozharness and only has this call here. VIRTUALENV_NO_DOWNLOAD is the same but operating globally. So no difference if you you use either one. YMMV

Comment 22

[Jorg K (CEST = GMT+2)]

Comment on attachment 8875822 [details] [diff] [review]
Tell virtualenv not to download virtualenv.

Looks like this needs review to move this forward.

Comment 23

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 8875822 [details] [diff] [review]
Tell virtualenv not to download virtualenv.

Review of attachment 8875822 [details] [diff] [review]:
-----------------------------------------------------------------

Patch and reasoning sound good to me. I just skimmed the bug comments, but if this hasn't already happened a try run would probably be good. r=philipp

Comment 24

[Jorg K (CEST = GMT+2)]

I may just land this since a try run will cost the same as a landing on C-C. The patch doesn't look very destructive to me ;-)

Comment 25

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/comm-central/rev/61ad7098e066d51dd5980c8f110eeb648dcd8d95

Sorry, I forgot to land this.