1541166 - (vct-py3) [meta] Python 3 compatibility for version-control-tools

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect, P1)

Description

[Connor Sheehan [:sheehan]]

Tracking Python 3 compatibility in version-control-tools.

Edit June 7th 2019:

Useful link: https://www.mercurial-scm.org/wiki/Python3

Of particular interest is the "Using" section:

Mercurial's setup.py file refuses to run with Python 3 by default. This means that pip install Mercurial or python setup.py install will not work with Python 3 by default.

Setting the HGPYTHON3 environment variable will suppress this error and allow execution with Python 3. e.g. HGPYTHON3=1 pip3.7 install Mercurial or HGPYTHON3=1 python3.7 setup.py install.

No run-time environment variable or config option is required to use Python 3 with Mercurial: only the installation step / setup.py requires special action to override the Python version check.

Comment 1

[Connor Sheehan [:sheehan]]

Attachment: py3: remove Python 3 incompatible calls to `execfile` (Bug 1541166) r?lars

This commit replaces all calls to execfile with a file
read and call to exec, with file content as the parameter.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

Attachment: Python2/Python3 compatibility - try/except form for pylib/mercurial-support/killdaemons.py (Bug 1541166) r?sheehan

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/13daf68f9066
Python2/Python3 compatibility - try/except form for pylib/mercurial-support/killdaemons.py r=sheehan

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2c7002f9e788
py3: remove Python 3 incompatible calls to `execfile` r=lars

Comment 5

[Connor Sheehan [:sheehan]]

Attachment: py3: global replace `repo['` with `repo[b'` (Bug 1541166) r?lars

Using __getitem__ on classes that subclass repository (such
as localrepository) results in Python 3 compatibility failures
since we are currently not marking our strings with b''. This
commit does a global replace to assert all strings passed as such
are byte strings.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f1043f9d2312
py3: global replace `repo['` with `repo[b'` r=lars

Comment 7

[Connor Sheehan [:sheehan]]

Attachment: Apply this patch to make (some) things test using Python3. Requires hg 5.0

Comment 8

[Connor Sheehan [:sheehan]]

Attachment: py3: global replace `iteritems()` with `items()` (Bug 1541166) r?lars

iteritems() doesn't exist in Py3, with items() instead returning
a generator by default. This commit switches all instances of the
old iteritems method with items. In all cases except one, we
were evaluating the full list of items regardless of the generator
based function call, and the instance where we were only returned
a small number of items, so I don't expect any performance hit here
while we are still on Python 2.

The same operation has been performed for iterkeys() and keys(),
but only a single instance was found and is included in this commit.

There are still some instances of iteritems in the repo, in the
third_party directory of vendored libraries. These will be handled
in bug 1557783.

Comment 9

[Connor Sheehan [:sheehan]]

Attachment: updated patch to use for testing Python3

Updated version of the Python 3 testing patch. This version works in docker and installs hg 5.0.1 using Python 3.6.

Edit: tests aren't passing, I made an error when trying to test this out.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6b4c5f3ae719
py3: global replace `iteritems()` with `items()` r=lars

Comment 11

[Connor Sheehan [:sheehan]]

Attachment: hgtemplates: allow passing a path `hg` executable to `mozify-templates.py` (Bug 1541166) r?smacleod

In a later commit, I will upgrade our copy of the Mercurial test
harness for Python 3 compatibility. In doing so, the template
sync test fails as mozify-templates no longer uses the version
of hg in the test - instead finding the system hg and using that.
Since my version is 4.5.3 on the server and I am typically running
tests against 5.0.1 these days, the tests fail as 4.5.3 does not have
the sparse-revlog repository requirement.

This commit adds the ability to specify a specific version of hg
for use in the script, by passing a path in an environment variable.
It's a quick hack to make things work - we should probably add some
option parsing to mozify-templates.py at some point, at which time
we can convert the environment variable to an option.

Comment 12

[Connor Sheehan [:sheehan]]

Attachment: testing: update Mercurial support libraries (Bug 1541166) r?smacleod

These libraries have added more support for Python 3
compatibility under the new versions.

It turns out we have made some modifications to
run-tests.py that are documented in the commit
message for 85483f1cae73e27. I have added tracker
comments for these changes so they will be easier
to spot and preserve in future updates.

Comment 13

[Connor Sheehan [:sheehan]]

Attachment: py3: use `pycompat.getargspec` everywhere (Bug 1541166) r?smacleod

inspect.getargspec has changed under Python 3, and there is a helper
variable in mercurial.pycompat that papers over the differences.
This commit switches all instances in version-control-tools to use
pycompat.getargspec. We don't bother changing the single occurance
in pylib/pycodestyle since that is third party and will likely be
removed in the near future.

Since the pycompat.getargspec helper was introduced in Mercurial
4.6, we bump the minimumhgversion string to require at least
that version in both firefoxtree and push-to-try.

Comment 14

[Connor Sheehan [:sheehan]]

Attachment: push-to-try: print generic message about JSON decode error (Bug 1541166) r?smacleod

Getting the error message from within the ValueError or
JSONDecodeError will require different methods on Python
2 and 3, and will produce different outputs on each version
when using a solution that is compatible with both versions (thus
requiring a different output in the test). Instead, let's
just print a generic message about JSON decoding issues,
and let the user investigate further if their try_task_config.json
file is malformed (in many cases I don't find the error messages
useful anyways).

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/462e9d604451
hgtemplates: allow passing a path `hg` executable to `mozify-templates.py` r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6e0af7e5a70f
testing: update Mercurial support libraries r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/79a28ff70ead
py3: use `pycompat.getargspec` everywhere r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/dcc3ad8eab7c
push-to-try: print generic message about JSON decode error r=smacleod

Comment 16

[Connor Sheehan [:sheehan]]

Attachment: hgext/clang-format: add bytestring prefixes to clang-format extension (Bug 1541166) r?glob

Similar to previous, we need bytestrings everywhere that interfaces with
Mercurial internal APIs.

Comment 17

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxtree: add bytestring prefixes everywhere (Bug 1541166) r?glob

Same spirit as the previous commits. Several more of these will
be coming through the review pipeline before this is all over.

Comment 18

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxtree: use `util.safehasattr` to determine object type (Bug 1541166) r?glob

Previously we were using isinstance(obj, str). This solution isn't
compatible with Python 2 and 3. A simpler solution is to check if
the object has the local attribute. If it does, it is a repo
instance, and we want to call the function. Otherwise, we create
the local repo ourselves.

Comment 19

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxtree: convert `template` key to byte string before passing to template formatter (Bug 1541166) r?glob

The fxheads command allows passing the -T/--template argument.
This command line flag is converted to a Python keyword argument when
passed to the function that runs the specified command. As a result,
the key in the opts dict (which is essentially just a renamed
**kwargs call) is a unicode string under Python 3, while the template
formatter object is expecting a dict with a byte string template
argument. In this commit we manually pass a dict with the correct
byte string key.

Comment 20

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxtree: stop encoding `firefoxtree` values (Bug 1541166) r?glob

After marking all strings as byte-strings in this extension and
mozautomation, we don't need to do this.

Comment 21

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxtree: pass `rev` argument in a list (Bug 1541166) r?glob

This commit fixes what is actually a longstanding bug in firefoxtree
that by pure coincidence has been providing the intended behaviour.

Essentially the value of the rev option passed to most Mercurial
commands in the Python function definiton is supposed to be a list
of revsets (ie one entry for each -r <revset> used on the command
line). In firefoxtree's wrapping of hg outgoing, we override the
value of rev and always default to using . as our revset, except
we don't pass our overridden value in a list.

In many code paths that use the rev option (revset parsing code,
primarily) the elements of the rev list are iterated over, and
each is evaluated to be converted into symbols for the revset language.
Since we are passing a single string instead of a list of strings to
iterate over, we were accidentally iterating over the characters in
our single revset.

On Python 2, when iterating over elements of a str, you get the
string representation of those elements. For example, iterating over
'hello' would give the sequence ['h', 'e', 'l', 'l', 'o'].
For our revset of '.', by pure luck, our iteration would
end up giving the correct sequence, ['.'] (since we only have a
single element in the string).

On Python 3, we need to use bytes, as Mercurial assumes all strings
are represented in 8 bits. When iterating (or doing any accesses of
elements) over a byte string, instead of returning the string
representation as Python 2 does, we get the integer representation
of those bytes[0]. In the case of b'.', this means our iteration
sequence looks like [46]. Later down the line, the revset parser
sees 46 and assumes this is a revision number, which it later uses
to search the index, causing an IndexError.

The correct way to pass our revset override is to wrap it in a list.
So this commit fixes the bad option, and brings firefoxtree up
to Python 3 compatibility.

[0] https://docs.python.org/3.7/library/stdtypes.html?highlight=bytes#bytes.hex,
see directly below the definition of hex

Comment 22

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxtree: drop support for Mercurial 4.4 and 4.5 (Bug 1541166) r?glob

No longer compatible after the Py3 changes in this stack.

Comment 23

[Connor Sheehan [:sheehan]]

Attachment: hgext/pushlog: wraps values from `sqlite3` as compatible byte strings (Bug 1541166) r?glob

This commit adds parameters to the sqlite3 connection object
to force the returned objects into the correct values. The
detect_types keyword argument is set to PARSE_DECLTYPES,
which means that all returned values are returned as the type
in their schema defintion (ie something defined as "integer
primary key id" will be passed through the "integer" converter
before returning to the caller). The text_factory function,
which takes a single bytestring parameter and returns the
resulting object, is set to pycompat.bytestr to ensure all
TEXT values returned by sqlite are compatible with both
Python 2 and 3.

Comment 24

[Connor Sheehan [:sheehan]]

Attachment: hgext/hgmo: create a shallow copy of `entry` for iteration (Bug 1541166) r?glob

In the hgmo extension we iterate over a dict representing
a changeset "entry", mutating the state of the dict as we
do so. This includes some del statements and changes to
various values in the dict.

On Python 3, mutating a dict during iteration is not allowed
and throws a RuntimeError. This commit works around that
restriction by creating a copy of the dict and iterating over
it, while mutating the original object.

Comment 25

[Connor Sheehan [:sheehan]]

Attachment: hgext/firefoxreleases: add bytestring prefixes everywhere (Bug 1541166) r?glob

Comment 26

[Connor Sheehan [:sheehan]]

Attachment: hghooks: add byte-string prefix to raw strings (Bug 1541166) r?glob

As described in https://www.mercurial-scm.org/wiki/Python3,
we will need to ensure any raw strings used with core Mercurial
code are converted to byte strings in Python 3. This commit
starts converting many of these occurrances in the hghooks/
directory.

Comment 27

[Connor Sheehan [:sheehan]]

Attachment: hghooks: replace `.format` call with old-style byte string format (Bug 1541166) r?glob

.format doesn't exist on byte-strings in Py3, so
we need to use the old format.

Comment 28

[Connor Sheehan [:sheehan]]

Attachment: testing: make `url-intercept.py` compatible with Python 3 (Bug 1541166) r?glob

This file is a dummy extension used in testing to mock out
responses from various well-defined URLs.

Comment 29

[Connor Sheehan [:sheehan]]

Attachment: hghooks: use `range` instead of `xrange` everywhere (Bug 1541166) r?glob

xrange is dead in Python 3. Long live range!

Comment 30

[Connor Sheehan [:sheehan]]

Attachment: hghooks/changelog_correctness: use Python 2/3 compatible `StringIO` (Bug 1541166) r?glob

This object has been renamed in Python 3. Mercurial's pycompat
module papers over the rename, so import the object from there.

Comment 31

[Connor Sheehan [:sheehan]]

Attachment: hghooks/changelog_correctness: use `next(iter)` instead of `iter.next()` (Bug 1541166) r?glob

Using the builtin function is the correct way to advance an
iterator.

Comment 32

[Connor Sheehan [:sheehan]]

Attachment: hghooks: use Python 2/3 compatible `urlopen` (Bug 1541166) r?glob

urllib2/urllib was split up and moved around between Python
2 and 3. Use Mercurial's urllibcompat module to handle the
differences.

Comment 33

[Connor Sheehan [:sheehan]]

Attachment: testing: use `commitextras` extension shipped with core Mercurial (Bug 1541166) r?glob

test-prevent-vendored-changes.t uses the commitextras extension vendored
from the Facebook hg-experimental repo. Since the time that test was
written, commitextras has been moved into core Mercurial. Switch to using
that version of the extension, as it is compatible with Python 3.

Comment 34

[Connor Sheehan [:sheehan]]

Attachment: hghooks/treeclosure: use `as` syntax in `try/except` block (Bug 1541166) r?glob

For Python 3 compatibility.

Comment 35

[Connor Sheehan [:sheehan]]

Attachment: hghooks: use absolute imports in `treeclosure_comm_central.py` (Bug 1541166) r?glob

The comm-central hook re-uses the isPushAllowed logic from
the regular treeclosure hook. Use an absolute import to
make this work under Python 2 and 3.

Comment 36

[Connor Sheehan [:sheehan]]

Attachment: hgext/configwizard: add bytestring prefixes to many strings (Bug 1541166) r?glob

Many changes here. A few places that call into Python internals are
left as unicode strings (such as comparisons with os.name, etc).

Comment 37

[Connor Sheehan [:sheehan]]

Attachment: testing: add a Py3 test ignore list and teach `run-tests` to respect it by default (Bug 1541166) r?glob

Similar to our "flaky" tests list, we introduce a list of
tests which fail under Python 3 and enforce it's use when
running tests with a Python 3 hg. To do so we add a flag
--run-py3-failing which performs the same function as the
flaky test list (add the blacklist unless the flag is present)
except it also asserts the Python version used in the hg
binary is Python 3.

To get this information, we expand get_hg_version to return
the Python version as well as the Mercurial version for a
given hg. We also allow subprocess.CalledProcessError to
be called, since sometimes the error code can be 1 despite
the correct JSON output being emitted. We will stil fail when
attempting to decode the JSON if some fatal event has actually
occurred. Making this change requires us to move a check to an
earlier point in the script, which should have no effect.

Comment 38

[Connor Sheehan [:sheehan]]

Attachment: mozext: use `next(iter)` instead of `iter.next()` (Bug 1541166) r?glob

The iterator API is preferred.

Comment 39

[Connor Sheehan [:sheehan]]

Attachment: mozautomation: use `memoryview` in place of `buffer` on Python 3 (Bug 1541166) r?glob

buffer is removed in Python 3, and memoryview should be used instead.
Create an alias for memoryview when loaded under Python 3.

Comment 40

[Connor Sheehan [:sheehan]]

Attachment: mozext: use `util.safehasattr` instead of `hasattr` (Bug 1541166) r?glob

Comment 41

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator: convert `hg debugbase85obsmarkers` to use the templater (Bug 1541166) r?glob

Using json.loads to turn the data into JSON would require
type-checking all fields to ensure they are the str type
before attempting to serialize. We also switch to using the
formatter's formatdate function, which enforces the correct
Mercurial date format in it's output. As a result of this a
previous assumption that the first value in the date tuple is
a float is invalidated, so that value is wrapped in float
call.

Comment 42

[Connor Sheehan [:sheehan]]

Attachment: hgext/hgmo: use `functools.cmp_to_key` to work around Py3 `sorted` differences (Bug 1541166) r?glob

Python 3 changed the sorted function to take a key function
(a function which can be used as a sorting key for the items in
the sequence) instead of a cmp function (a function which takes
two arbitrary items and orders them). This commit papers over the
differences using functools.cmp_to_key. We can fully rewrite this
as a key function when we drop Python 2 support.

Comment 43

[Connor Sheehan [:sheehan]]

Attachment: pash: make compatible with Python 3 (Bug 1541166) r?glob

Paper over several changed import locations (configparser,
escape function, etc). Switch from print statements to the
print function, set raw_input to input under Python 3.

As a ride-along, make the code more Pythonic by removing some
braces around if-statement conditions.

Comment 44

[Connor Sheehan [:sheehan]]

Attachment: ldap: make all code interfacing with `python-ldap` Python 3 compatible (Bug 1541166) r?glob

python-ldap returns all LDAP keys as system strings, and values as
bytes. Encoding and decoding values before operating on search results,
as well as using system strings wherever possible makes us compatible
with Python 3.

The ldap-lookup-ssh-key script also uses python-ldap APIs, so we
take those changes as part of this commit.

Comment 45

[Connor Sheehan [:sheehan]]

Attachment: hgmolib: make s3 bundle generation code Python 3 compatible (Bug 1541166) r?glob

Some simple changes to read objects into strings instead of bytes,
and encode data from a subprocess.

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2a3b20a0996f
hgext/clang-format: add bytestring prefixes to clang-format extension r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f92ea6f2d36f
hgext/firefoxtree: add bytestring prefixes everywhere r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/82741c954238
hgext/firefoxtree: use `util.safehasattr` to determine object type r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4eb8d5e9625a
hgext/firefoxtree: convert `template` key to byte string before passing to template formatter r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7104cf32d338
hgext/firefoxtree: stop encoding `firefoxtree` values r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c1c68fe6e531
hgext/firefoxtree: pass `rev` argument in a list r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/30e89b0226f6
hgext/firefoxtree: drop support for Mercurial 4.4 and 4.5 r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/cd470e194f50
hgext/pushlog: wraps values from `sqlite3` as compatible byte strings r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/797f6b62fde3
hgext/hgmo: create a shallow copy of `entry` for iteration r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/52c16436d411
hgext/firefoxreleases: add bytestring prefixes everywhere r=glob

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ab777935ce15
hghooks: add byte-string prefix to raw strings r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/184d98f61265
hghooks: replace `.format` call with old-style byte string format r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/928837c8dba5
testing: make `url-intercept.py` compatible with Python 3 r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/bfc42a338ff4
hghooks: use `range` instead of `xrange` everywhere r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b3b0d95bf204
hghooks/changelog_correctness: use Python 2/3 compatible `StringIO` r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b57fb4a922d6
hghooks/changelog_correctness: use `next(iter)` instead of `iter.next()` r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d1bd8ce19387
hghooks: use Python 2/3 compatible `urlopen` r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d77cbc222634
testing: use `commitextras` extension shipped with core Mercurial r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/92faa0989cc6
hghooks/treeclosure: use `as` syntax in `try/except` block r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b8c3d12c40f9
hghooks: use absolute imports in `treeclosure_comm_central.py` r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6b81b05fee5f
hgext/configwizard: add bytestring prefixes to many strings r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/60b58a8a2c8e
testing: add a Py3 test ignore list and teach `run-tests` to respect it by default r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ab7f2f261477
mozext: use `next(iter)` instead of `iter.next()` r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6647a7ab9125
mozautomation: use `memoryview` in place of `buffer` on Python 3 r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/bc63385ca7c9
mozext: use `util.safehasattr` instead of `hasattr` r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7582e48bb042
vcsreplicator: convert `hg debugbase85obsmarkers` to use the templater r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/de1abdaa0dfc
hgext/hgmo: use `functools.cmp_to_key` to work around Py3 `sorted` differences r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/91b98a7c8dfb
pash: make compatible with Python 3 r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d3c8e2f2e946
ldap: make all code interfacing with `python-ldap` Python 3 compatible r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/76731091d482
hgmolib: make s3 bundle generation code Python 3 compatible r=glob

Comment 48

[Connor Sheehan [:sheehan]]

Attachment: testing: enable Python 3 compatibility with config option test scripts (Bug 1541166) r?glob

These scripts are used to set hgrc/config file options in the Docker
cluster tests. Paper over the configparser name change under Python
3.

Comment 49

[Connor Sheehan [:sheehan]]

Attachment: testing: make `sns-subscription-server` compatible with Python 3 (Bug 1541166) r?glob

We have a simple HTTP mock server to test the SNS notification
daemon. Many of the modules it uses were moved to the http
module of the standard library under Python 3. This commit
papers over the differences to achieve a Python 2/3 compatible
version.

Comment 50

[Connor Sheehan [:sheehan]]

Attachment: hgwsgi: make wsgi bootstrap script compatible with Python 3 (Bug 1541166) r?glob

This script is sourced and called by the other WSGI scripts found
in the hgwsgi/ directory. Setting a few values as bytes lets us
use it under Python 2 and 3.

Comment 51

[Connor Sheehan [:sheehan]]

Attachment: scripts: set shebang values for script to generic environment `python` (Bug 1541166) r?glob

Many of our utility scripts in scripts/ have python2.7 hard-coded
as the executable for use in their shebang line. This commit changes
many of these scripts to instead attempt to use python as the executable.

Comment 52

[Connor Sheehan [:sheehan]]

Attachment: testing: install a Py3 interpreter and Mercurial executables in test environment (Bug 1541166) r?glob

This commit teaches create-test-environment to install a Python 3
interpreter in the venv/ directory created by the script. We
introduce new test-requirements-3.{in,txt} files which are
installed into the interpreter after creation. These new files are
derived from test-requirements.{in,txt} and include all packages
necessary to run tests for v-c-t. The command assumes the user has
python3 aliased to a Python 3 interpreter in their shell.

The main use case for this change is to enabled running tests against
Python 3 Mercurial installations, like so:

$ ./run-tests hgserver/ --with-hg=venv/py3/mercurials/5.2rc0/bin/hg

Support for --all-hg-versions will be added in a later commit.
Targetting specific versions of Python can be achieved by aliasing
python3 to the desired interpreter in the users shell.

Comment 53

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2d34c57c6493
testing: enable Python 3 compatibility with config option test scripts r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/370a9a09ebda
testing: make `sns-subscription-server` compatible with Python 3 r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f161ff3fef11
hgwsgi: make wsgi bootstrap script compatible with Python 3 r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2cc2db4c3cc3
scripts: set shebang values for script to generic environment `python` r=glob

Comment 54

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9df718279ba6
testing: install a Py3 interpreter and Mercurial executables in test environment r=glob

Comment 55

[Connor Sheehan [:sheehan]]

Attachment: ansible: remove Py2 venv if installing Py3 venv in same directory (Bug 1541166) r?glob

Upgrading to Python 3 will involve re-installing many of the various
virtual environments on hgmo and upgrading dependencies for Python 3.
Rather than re-name all the environments, this commit teaches the
virtualenv3 Ansible task to check for the presence of a Python 2
virtual environment and delete it if present, writing the new
Python 3 virtual environment in it's place.

Comment 56

[Connor Sheehan [:sheehan]]

Attachment: ansible: install py3 development headers on hg hosts (Bug 1541166) r?glob

The headers are needed to install Mercurial using Python 3.

Comment 57

[Connor Sheehan [:sheehan]]

Attachment: ansible/hg-ssh: update `venv_bundles` to Python 3.6 (Bug 1541166, Bug 1593271) r?glob

This commit upgrades the bundles venv to Python 3.6. At the same time
we also upgrade the Mercurial version to Mercurial 5.2, since this
version is the first to remove the requirement of the HGPYTHON3
environment variable while compiling Mercurial.

This commit should be safe to deploy at any time, since the bundle
generation process runs once a day, overnight. Other similar changes
such as upgrading the virtualenv for hgweb/pash will require a rolling
deployment to ensure no production incidents.

Comment 58

[Connor Sheehan [:sheehan]]

Attachment: firefoxreleases: add byte-string prefix to another string (Bug 1541166) r?glob

Missed this one as it was in the setdefault call.

Comment 59

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/84a6ffcdd4e5
firefoxreleases: add byte-string prefix to another string r=glob

Comment 60

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/226672b8f4a2
ansible: remove Py2 venv if installing Py3 venv in same directory r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/94bbbdbab0bb
ansible: install py3 development headers on hg hosts r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b0a5721a3dd3
ansible/hg-ssh: update `venv_bundles` to Python 3.6 (Bug 1541166, Bug 1593271) r=glob

Comment 61

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e51a33329bdd
Backed out changeset b0a5721a3dd3 
https://hg.mozilla.org/hgcustom/version-control-tools/rev/880c69dbd89f
Backed out changeset 94bbbdbab0bb

Comment 62

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b49680c6385d
ansible: install py3 development headers on hg hosts r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b337f66ab146
ansible/hg-ssh: update `venv_bundles` to Python 3.6 (Bug 1541166, Bug 1593271) r=glob

Comment 63

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/eb9ad81bafaa
ansible/hg-web: remove installation of python3-devel package

Comment 64

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/14f2c6380770
ansible/hg-ssh-server: install `python3-devel` instead of `python36u-devel`

Comment 65

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/042266bf6b76
ansible/hg-ssh-server: modify Python 3 yum packages

Comment 66

[Connor Sheehan [:sheehan]]

Attachment: ansible/hg-ssh: make `check_kafka_hg_heartbeat` Python 3 compatible (Bug 1541166) r?smacleod

The check_kafka_hg_heartbeat Nagios check has a few encoding
issues in Python 3, since the values returned by subprocess.check_output
are byte-strings but the rest of the script assumes regular Python
str's. We also attempt to directly print an exception before
converting to a str, so this commit adds that conversion as well.

Comment 67

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/bc649592dc45
ansible/hg-ssh: make `check_kafka_hg_heartbeat` Python 3 compatible r=smacleod

Comment 68

[Connor Sheehan [:sheehan]]

Attachment: obsolescencehacks: convert `user` value from `pwd` module to byte string (Bug 1541166) r?mhentges,smacleod,zeid

Since the exchangeusers set will be composed of byte strings, we will
need to make sure this value is a byte string as well.

Comment 69

[Connor Sheehan [:sheehan]]

Attachment: hgmolib: mark as compatible with Python 3.6 (Bug 1541166) r?mhentges,smacleod,zeid

Full compatibility has landed for this module. Since this isn't on
PyPI it's mostly a cosmetic change, but we might as well mark it
anyways.

Comment 70

[Connor Sheehan [:sheehan]]

Attachment: ansible/hg-web: install `python3-devel` (Bug 1541166) r?mhentges,smacleod,zeid

We'll need the Python 3 development headers to compile Mercurial
on hgweb. hgssh already has this package installed.

Comment 71

[Connor Sheehan [:sheehan]]

Attachment: ansible/hg-web: upgrade `mod_wsgi` to 4.6.2 (Bug 1541166) r?mhentges,smacleod,zeid

Newer versions of mod_wsgi support Python 3. This commit upgrades
the version installed on hgweb to the latest, so we can seamlessly
upgrade to Python 3.

Comment 72

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/900e033bb5ae
obsolescencehacks: convert `user` value from `pwd` module to byte string r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/05b431f35a3a
hgmolib: mark as compatible with Python 3.6 r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b27606511259
ansible/hg-web: install `python3-devel` r=zeid,mhentges

Comment 73

[Connor Sheehan [:sheehan]]

Attachment: ansible/hg-web: upgrade all virtualenvs to Python 3 and Mercurial 5.2 (Bug 1541166) r?mhentges,smacleod,zeid

This commit upgrades all virtual environments on hgweb to Python 3.

The upgrade plan will involve deploying the repo state of v-c-t at
this commit to the hgweb mirrors in one AWS region, with the private
mirror rate for that region set to zero. Once the deploy has completed
successfully, we will set the rate to a low value (between 10-25% of
all CI traffic for that region) and verify there are no major bugs
with the deployment. We will then slowly increase the rate and ensure
the error rate for that region hasn't increased. Once we have increased
the rate to 100%, we will deploy this change to the remaining private
mirrors (set to 0 traffic, increasing directly to 100 once the deploy
completes successfully). Once our CI mirrors are fully upgraded to Py3,
we will conduct a rolling deployment to the public hg.mo interface in
MDC1, finally calling the upgrade complete.

We also upgrade test-template-sync.t to point to the new Mercurial
version being upgraded to, and change the hard-coded path to the template
folder to work with Python 3.

Comment 74

[Connor Sheehan [:sheehan]]

Attachment: ansible/hg-ssh*: upgrade to Python 3 and Mercurial 5.2 (Bug 1541166) r?mhentges,smacleod,zeid

This commit upgrades all virtual environments on hgssh to Python 3.

The upgrade plan for this commit is to request a tree closure window
from the sheriffs before conducting this upgrade. Once the trees are
closed, we will run through this deploy and wait until a successful
completion. After successful completion, we will make several test
pushes to user repositories to ensure no problems are encountered.
Once we have confidence that the upgrade has completed without a hitch,
we will re-open the trees and monitor performance of the system.

Comment 75

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a2d911db5097
ansible/hg-web: upgrade `mod_wsgi` to 4.6.2 r=zeid,mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/01fc71a60921
ansible/hg-web: upgrade all virtualenvs to Python 3 and Mercurial 5.2 r=smacleod

Comment 76

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2fead2184756
mozautomation: print unicode instead of bytes

Comment 77

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/67fb9524ba1f
hgwsgi: convert `HTTPS_PROXY` to a `str` before passing to `os.environ`

Comment 78

[Connor Sheehan [:sheehan]]

Attachment: pushlog: ensure `start` and `end` of range are `int` (Bug 1541166) r?zeid

range expects integers, and this value can sometimes be a float. I noticed
a TypeError when scanning the logs during the Python 3 upgrade, but I'm
not actually sure if it is related.

Comment 79

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/68d8127a0a0d
firefoxreleases: add a byte-string prefix to key in `releases` dict

Comment 80

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8178c8336ea5
pushlog: ensure `start` and `end` of range are `int` r=zeid

Comment 81

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e971e086aec5
firefoxreleases: convert `changesetentry` to `str` kwargs before returning

Comment 82

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/86795ba50575
commitparser: pass groupname as `str` to `re.match.{end,start}` 
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f673a8ccd306
commitparser: pass byte-string to `startswith`

Comment 83

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/db888e6d3d88
vcsreplicator: glob over Py3 only output in `test-config.t`

Comment 84

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0a299902bdb9
ansible/hg-ssh*: upgrade to Python 3 and Mercurial 5.2 r=zeid,smacleod

Comment 85

[Connor Sheehan [:sheehan]]

Attachment: hooks: fix bug references check and advanced url intercept Python 3 issues (Bug 1541166) r?zeid

While testing Mercurial 5.3 against our extensions and mock
server cluster, I noticed the bug references check and the
helper "advanced url intercept" mock tool are not Python 3
compatible. These scripts landed in an awkward time where
the hgssh server was not on Py3 yet, and was disabled before
the upgrade was deployed to the server.

This commit changes all hard-coded string accesses for dicts
loaded from json.load* to use Python system strings as accessors,
since the json module will convert them to regular strings.

We switch from using a StringIO.StringIO to a mercurial.pycompat.bytesio
which will paper over the Python 2/3 differences.

Also, under Python 3 the object returned by an installed URL opener
is assumed to have all required methods, which includes both the
"file-like" interfaces methods provided by StringIO, and some
custom methods such as getcode. We add a simple monkey patch
to correctly return 200 from a call to getcode() on the
returned object.

As part of debugging this code I flattened the main code path in
advanced url interceptor and this change has been left in place
for clarity.

Comment 86

[Connor Sheehan [:sheehan]]

Attachment: url-intercept: patch over bustage caused by chemspill patch (Bug 1541166) r?zeid

During the Python 3 upgrade an issue with the treeclosure hook
was discovered and a minor outage took place while a patch
was created to fix it. In rushing around I patched over an issue
where the url was a Request object instead of a string as
expected, but did not account for the difference between
b'url' and 'url' on Python 3. This commit wraps the patched
value of url to be a byte-string, fixing tests.

Depends on D70957

Comment 87

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7aa02c9e7ac2
hooks: fix bug references check and advanced url intercept Python 3 issues r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f08999cbf4ed
url-intercept: patch over bustage caused by chemspill patch r=zeid

Comment 88

[Connor Sheehan [:sheehan]]

Attachment: testing: create Py2/3 compatible json pretty-print alias (Bug 1541166) r?zeid

On Python 2, python -m json.tool prints the json with keys
sorted. On Python 3 the same command prints the json with keys
in the same order as inputted, and has a flag to enable sorting.

This commit adds an alias ppjson that papers over the differences
to the tests can be run with either Python version.

Depends on D74756

Comment 89

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c562cd27e434
testing: create Py2/3 compatible json pretty-print alias r=zeid

Comment 90

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3cdb27092967
robustcheckout: fix `test-perfherder.t` failing regex 
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6db81c3ff751
testing: remove Py3 failing test list and associated code

Comment 91

[Connor Sheehan [:sheehan]]

We've been Py3 compatible for some time. There are still a few lingering instances of Py2 compat that I find from time-to-time. However we use Py3 in production and are close to being able to drop Py2 support altogether.