Last Comment Bug 1279890 - make signtool.py python3 compatible
: make signtool.py python3 compatible
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: Tools (show other bugs)
: unspecified
: Unspecified Unspecified
-- normal (vote)
: ---
Assigned To: Aki Sasaki [:aki]
: Hal Wine [:hwine] (use NI)
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-06-13 04:36 PDT by Aki Sasaki [:aki]
Modified: 2016-07-08 10:30 PDT (History)
1 user (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
flake8.txt (47.79 KB, text/plain)
2016-06-13 04:40 PDT, Aki Sasaki [:aki]
no flags Details
build-tools-py3.diff (312.51 KB, patch)
2016-06-13 05:02 PDT, Aki Sasaki [:aki]
rail: review+
Details | Diff | Splinter Review

Description User image Aki Sasaki [:aki] 2016-06-13 04:36:39 PDT
I need to be able to sign things from scriptworker->signingscript.  Scriptworker and signingscript are using python3, so I had three options:

* call `python2 signtool.py`
* write my own python3 signing functions in signingscript.
* port signtool.py to python3

The first seemed wrong to me.  The second is tempting and may still be the future when all signing is done through signingscript, but we still use signtool a lot of places.

I ported signtool.py to py3, and got all the build/tools unittests green in both py27 and py35.  I left the bulk of the tools scripts untouched, as out of scope.
Comment 1 User image Aki Sasaki [:aki] 2016-06-13 04:40:11 PDT
Created attachment 8762507 [details]
flake8.txt

Output from `flake8 --exclude=vendor --max-line-length 160 > flake8.txt`
We still have work to do if we want all of build/tools to be python3 compatible and passing flake8.
Comment 2 User image Aki Sasaki [:aki] 2016-06-13 05:02:18 PDT
Created attachment 8762512 [details] [diff] [review]
build-tools-py3.diff

This patch:

* adds python3 compatibility to signtool.py and lib/python, so it passes all unittests
* adds py35 to tox.ini and .travis.yml
* adds git config items to init_gitrepo.sh, since I was hitting email/name errors despite having a ~/.gitconfig
* switched to using resp.app_iter instead of resp.body in the signingserver tests, because the latter tried to do a b''.join() on a str or bytes object in py3, which doesn't seem like it would ever work
* chmods now require an explicit octal notation
* new version of pefile!
* requirements-py3.txt.  This is so we can install unreleased versions of poster and redo.

I had the biggest problems with the user auth tests in signingserver, and testApplyAndPushRebaseFails in hg.

All green: https://github.com/mozilla/build-tools/pull/17

This is a pretty big patch, but it's largely simple changes.  Let me know if you have questions, or if you want someone else to take a look?
Comment 3 User image Aki Sasaki [:aki] 2016-06-13 05:05:41 PDT
Oh: testApplyAndPushRebaseFails: I think the update has to happen *after* the strip, otherwise we can update to the wrong head, strip that head, and then create a new head when applying the new changesets.  Moving the update after the strip fixes this.

This was working with the incorrect behavior in py27 tox, but was broken in py27 nosetests when only running the test_util_hg.py file, and broken across the board in py35.  The new behavior works in all 4 scenarios.
Comment 4 User image Rail Aliiev [:rail] ⌚️ET 2016-06-16 02:29:20 PDT
Comment on attachment 8762512 [details] [diff] [review]
build-tools-py3.diff

Review of attachment 8762512 [details] [diff] [review]:
-----------------------------------------------------------------

In overall the patch looks good to me. To avoid extra churn with new deps, I suggest to drop six into lib/python/vendor and adjust the pth file. Otherwise we would need to fix at least release runner's puppet manifest to install six, and probably many other scripts using libs from tools
Comment 5 User image Aki Sasaki [:aki] 2016-06-28 13:48:54 PDT
signtool.py is no longer working in py3 for me :(
I'm hitting poster multipart encoding errors.  I don't know how it ever worked, although I do know that I was editing my venv for debugging purposes.  That's a likely culprit as to why worked then but no longer works.

For the short term, I created a py2 venv to run signtool from build-tools.
Later I can revisit fixing up poster or rewriting signtool with aiohttp.
Comment 6 User image Aki Sasaki [:aki] 2016-07-08 10:30:26 PDT
https://pypi.python.org/pypi/signtool
https://github.com/escapewindow/signtool

py3 compatible after I ported it to use requests; 65% test coverage.

Note You need to log in before you can comment on or make changes to this bug.