1605128 - build fails with "AttributeError: 'module' object has no attribute 'ensure_text'"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nathan Froyd [:froydnj]]

I just tried building (with Python 2.7), and got:

 0:00.71 /usr/bin/make -f client.mk MOZ_PARALLEL_BUILD=50 -s
 0:00.81 Traceback (most recent call last):
 0:00.81   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
 0:00.81     "__main__", fname, loader, pkg_name)
 0:00.81   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
 0:00.81     exec code in run_globals
 0:00.81   File "/home/froydnj/src/gecko/python/mozbuild/mozbuild/action/preprocessor.py", line 25, in <module>
 0:00.81     main(sys.argv[1:])
 0:00.82   File "/home/froydnj/src/gecko/python/mozbuild/mozbuild/action/preprocessor.py", line 21, in main
 0:00.82     pp.handleCommandLine(args, True)
 0:00.82   File "/home/froydnj/src/gecko/python/mozbuild/mozbuild/preprocessor.py", line 509, in handleCommandLine
 0:00.82     self.processFile(input=input, output=out)
 0:00.82   File "/home/froydnj/src/gecko/python/mozbuild/mozbuild/preprocessor.py", line 420, in processFile
 0:00.82     self.do_include(input, False)
 0:00.82   File "/home/froydnj/src/gecko/python/mozbuild/mozbuild/preprocessor.py", line 821, in do_include
 0:00.82     self.includes.add(six.ensure_text(abspath))
 0:00.82 AttributeError: 'module' object has no attribute 'ensure_text'
 0:00.82 /home/froydnj/src/gecko/config/rules.mk:1151: recipe for target '.cargo/config' failed

Are we using an in-tree six?

Comment 1

[Nathan Froyd [:froydnj]]

Er, apparently some other bug modified preprocessor.py to use ensure_text.

Comment 2

[Ricky Stewart]

Yeah, six lives in-tree (third_party/python/six) and the version we're using is new enough:

rickystewart-a5lvdq:mozilla-unified rickystewart$ rg ensure_text third_party/python/six
third_party/python/six/six.py
903:def ensure_text(s, encoding='utf-8', errors='strict'):

third_party/python/six/test_six.py
1048:    def test_ensure_text(self):
1049:        converted_unicode = six.ensure_text(self.UNICODE_EMOJI, encoding='utf-8', errors='strict')
1050:        converted_binary = six.ensure_text(self.BINARY_EMOJI, encoding="utf-8", errors='strict')

third_party/python/six/documentation/index.rst
451:.. function:: ensure_text(s, encoding='utf-8', errors='strict')

third_party/python/six/CHANGES
25:- Pull request #204: Add `six.ensure_binary`, `six.ensure_text`, and

I can't reproduce locally and automation is fine (of course, or else this wouldn't have made it to central).

mars, ahal: any idea why this might happen?

Comment 3

[Nathan Froyd [:froydnj]]

Apparently I had an old copy in my virtualenv (or on my system itself, not sure), and blowing away my objdir fixed things. I'd like to understand why things didn't work in the first place, though.

Comment 4

[Andrew Halberstadt [:ahal]]

Not sure. Could some tool have installed an older six into the build system's virtualenv which was getting picked up ahead of the in-tree one?

Comment 5

[Ricky Stewart]

FWIW, if your Python virtualenvs get borked (happens to me not infrequently working on Python 3 stuff), you can just rm -rf objdir/_virtualenvs and the next build will recreate them. You don't have to trash your entire objdir in that case.

Comment 6

[Māris Fogels [:mars] (please needinfo)]

:frodynj what likely happened is you objectdir was built with a version of the mozilla source tree that included our previous version of the six library. If you built an objectdir with an old revision of mozilla-central that included six 1.10, and then updated to the latest mozilla-central, then you'd trigger this error.

We've run with six v1.10 until recently and v1.10 does not have the ensure_text() function. We recently upgraded to six v1.13 and people have started using its new features liberally. The virtualenv should have detected the change in library version requirements and clobbered itself. Perhaps you found a case where that didn't happen.

Unfortunately there's no way to tell what version of the source tree your virtualenv was built with now that the objectdir has been refreshed.

Comment 7

[Michael Froman [:mjf]]

I've seen this happen after running a mochitest and then rebuilding.

Comment 8

[Michael Froman [:mjf]]

(In reply to Michael Froman [:mjf] from comment #7)

I've seen this happen after running a mochitest and then rebuilding.

I'm on current head (3a51a5d07942) as of 2020/01/07. Specifically, I can repro with the following:

./mach build
./mach mochitest --headless --keep-open=false dom/media/tests/mochitest/identity/test_getIdentityAssertion.html &> log.txt
./mach build

If I follow the advice in Comment 5, I can rebuild successfully, but another mochitest run will cause the build failure again.

:mars, please let me know if there is anything you need from me to help figure out what is going on.

Comment 9

[Michael Froman [:mjf]]

One bit of additional info, the first few lines of the output from the mochitest output:

mfroman@moz:~/mozilla/moz-central$ ./mach mochitest --headless --keep-open=false dom/media/tests/mochitest/identity/test_getIdentityAssertion.html &> log.txt
tail: log.txt: file truncated
 /home/mfroman/mozilla/obj/deb/backend.TestManifestBackend is out of date with respect to /home/mfroman/mozilla/obj/deb/config.status
Test configuration changed. Regenerating backend.
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting six==1.10.0
  Using cached https://files.pythonhosted.org/packages/c8/0a/b6723e1bc4c516cb687841499455a8505b44607ab535be01091c0f24f079/six-1.10.0-py2.py3-none-any.whl

Seems like ./mach mochitest is causing an old version of six to be used.

Comment 10

[Ricky Stewart]

callek pointed this out: https://searchfox.org/mozilla-central/source/testing/mozharness/configs/unittests/mac_unittest.py#15

Not sure why we wouldn't just be using the vendored six here.

Comment 11

[Māris Fogels [:mars] (please needinfo)]

Punting to ahal, I don't know why mozharness would hardcode its own six module version.

I think the fix is trivial: bump six to v1.13.0. I don't see anything in the six changelog that would be a risk.

Comment 12

[Andrew Halberstadt [:ahal]]

It's possible we can remove the mozharness copy now, though you should follow-up with releng. Historically there were some places that plucked testing/mozharness directly from hg.m.o and for some reason that I can't remember it would have been very annoying to update this code to also pluck things from other places in the tree. So instead modules that mozharness dependend on were duplicated to testing/mozharness.

IIRC, this was fixed, but please don't take my word for it. Maybe Callek knows more.

Comment 13

[Andrew Halberstadt [:ahal]]

Oops, should have checked the link first. Looks like six isn't duplicated but installed at runtime somehow. Though this likely exists for the same reason I mentioned in the last comment. Things used to (and possibly still do) pluck mozharness from out of tree.

Comment 14

[Byron Campen [:bwc]]

This is because the webrtc mochitests need six:

https://searchfox.org/mozilla-central/source/testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt

We could probably just remove the version, and it would work ok.

Comment 15

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9962d181c1719988ee838c4292a3c35aa1d536f1

Comment 16

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7579ff43d245ace430f25d2061597035083a7d8a

Comment 17

[Paul Adenot (:padenot)]

Byron, this works fine for me, can you land it?

Comment 18

[Byron Campen [:bwc]]

Attachment: Bug 1605128: Make websocket_process_bridge less picky about the version of six it wants. r?ahal

Comment 19

[Byron Campen [:bwc]]

Up for review.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e73c840984e6
Make websocket_process_bridge less picky about the version of six it wants. r=ahal

Comment 21

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/e73c840984e6

Comment 22

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/releases/mozilla-beta/rev/19f907e78218