Add byte string prefixes to all strings that interface with Mercurial
Categories
(Developer Services :: Mercurial: hg.mozilla.org, task)
Tracking
(Not tracked)
People
(Reporter: sheehan, Assigned: sheehan)
References
Details
(Keywords: leave-open)
Attachments
(30 files)
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review |
All Python string constants that interface directly with Mercurial internal APIs will need to have byte-string prefixes added to them. Mercurial has assumed the "string" type will be encoded bytes since it's inception, and thus we will need to explicitly tell Python that our strings are in fact bytes.
I've already started a lot of the work here, but here are so many commits that will need to be submitted that I'm attaching them all to a single bug.
|
Assignee | |
Comment 1•5 years ago
|
||
Byte-string prefixes will be needed almost everywhere in
version-control-tools. Here we update mozhg for Python 3
compatibility.
|
|
Assignee | |
Comment 2•5 years ago
|
||
This commit adds byte-string prefixes to all necessary strings
in the push-to-try extension.
|
|
Assignee | |
Updated•5 years ago
|
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/7ede0a306189 mozhg: add byte string prefixes everywhere r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/fbe004e04405 push-to-try: add byte string prefixes r=smacleod
|
|
Assignee | |
Comment 4•5 years ago
|
||
|
|
Assignee | |
Comment 5•5 years ago
|
||
This file makes modifications to repositories in tests, and needs
to be updated for Python 3 compatibility in order to test other
extensions.
|
|
Assignee | |
Comment 6•5 years ago
|
||
This commit adds byte string prefixes just about everywhere in
mozautomation.
I haven't touched a few of the files, specifically those relating
to things like BuildBot and Bugzilla. I'm going to see if those
need to be upgraded to get us to Py3 compatibility (ie does their
lack of byte prefixes cause any bustage), and if not, they are
candidate for deletion.
|
|
Assignee | |
Comment 7•5 years ago
|
||
This makes this extension compatible with Py3.
|
|
Assignee | |
Comment 8•5 years ago
|
||
|
|
Assignee | |
Comment 9•5 years ago
|
||
cgi.escape is deprecated on Python 3, in favor of html.escape.
html.escape doesn't exist on Python 2, and only supports unicode
on Python 3. Mercurial has backported a copy of html.escape for
use with byte strings, and this commit copies that function into
mozautomation to support
|
|
Assignee | |
Comment 10•5 years ago
|
||
So the returned values are compatible with Mercurial running
on Python 2 and 3.
|
|
Assignee | |
Comment 11•5 years ago
|
||
|
|
Assignee | |
Comment 12•5 years ago
|
||
|
|
Assignee | |
Comment 13•5 years ago
|
||
|
|
Assignee | |
Comment 14•5 years ago
|
||
test-json-extradata.t includes an inline Python class,
which contains some hard-coded strings. These values must
be byte strings just as all other strings interfacing with
Mercurial.
|
|
Assignee | |
Comment 15•5 years ago
|
||
Changing most strings to byte-strings has the unfortunate side effect
of making those strings unusable as keyword arguments via dictionary
unpacking. The pycompat module has two helper functions that can
convert the top-level keys in a dict to unicode strings (for passing
to Python functions) or to byte strings (for passing to various
Mercurial functions, or passing the whole dict as an argument). This
commit wraps occurrances of mismatched keys to ensure compatibility
with Python 3.
In the case of the wrapped webutil.changesetentry, the function is
only called once in upstream Mercurial code, and it's return value
is immediately unpacked to be passed as kwargs to another function.
Since we want to pass the value of that function to addmetadata,
we need to first convert the keys to bytestrings, then re-convert
the keys to unicode strings before passing back to Mercurial. The
wrapper for webutil.changelistentry is called from within Mercurial
code and is not passed as keyword arguments, so we don't wrap those
values before passing to addmetadata.
|
|
Assignee | |
Comment 16•5 years ago
|
||
|
|
Assignee | |
Comment 17•5 years ago
|
||
|
|
Assignee | |
Comment 18•5 years ago
|
||
This gets us to all tests passing with one exception.
In that case the test output is correct, but the hghave
for Py3 seems to be failing to recognise the test is
running against Python 3. It appears the Py3 check
compares against the value of sys.version_info,
and I believe in this case is grabbing the version
from the interpreter of the test runner instead of
the Python 3 hg.
|
||
Comment 19•5 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/e1d74189a807 readonly: add byte string prefixes r=glob
|
|
||
Comment 20•5 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/5ea05f3fd0c2 testing: add byte string prefixes to `local-mozilla-repos.py` r=glob
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 21•5 years ago
|
||
Bytes-mageddon continues.
|
|
Assignee | |
Comment 22•5 years ago
|
||
More bytestring changes in this commit, this time for the
serverlog extension.
|
|
Assignee | |
Comment 23•5 years ago
|
||
|
|
Assignee | |
Comment 24•5 years ago
|
||
|
|
Assignee | |
Comment 25•5 years ago
|
||
Mercurial's urllibcompat exists to paper over the major differences
of urllib between Python 2 and 3.
|
|
Assignee | |
Comment 26•5 years ago
|
||
Add byte-strings to a few places.
|
|
Assignee | |
Comment 27•5 years ago
|
||
This commit wraps exception values in pycompat.bytestr to allow
displaying them under both Python 2 and 3.
|
|
Assignee | |
Comment 28•5 years ago
|
||
We have an extension in use that forces robustcheckout
to hit error conditions in a test. Use byte strings in
all interactions with Mercurial APIs, as with other
extensions.
|
|
Assignee | |
Comment 29•5 years ago
|
||
util.version is patched over to simulate different versions
of Mercurial. Ensure the function returns a byte-string to avoid
errors during calls to util.versiontuple.
|
|
Assignee | |
Comment 30•5 years ago
|
||
|
|
Assignee | |
Comment 31•5 years ago
|
||
|
|
Assignee | |
Comment 32•5 years ago
|
||
|
|
||
Comment 33•5 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/06b064b0aaef mozautomation: add byte string prefixes everywhere r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/b907b9d06dc2 mozautomation: remove `Build` namedtuple in favor of raw dict and remove `artifacts_url` parameter r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/2ff9009c05e4 mozautomation: vendor bytes-friendly `escape` from Mercurial r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/7dc746255f22 mozautomation: support passing a `text_factory` function to `FirefoxReleaseDatabase` r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/49aed22df635 hgext/js-format: add byte string prefixes everywhere r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/487c9f180feb testing: add byte-string prefixes to `testing/hghave.py` r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/7cfd8c61ddc7 pushlog: add byte string prefixes r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/cbda0c3640b4 hgext/hgmo: add bytesstring prefixes everywhere r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/a05f21993acf hgext/hgmo: add byte-string prefixes to inline code in `test-json-extradata.t` r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/6027ea6224d5 py3: use `pycompat.{strkwargs,byteskwargs}` to ensure arguments are correct type r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/36cc9ba83658 hgext/manualoverlay: add bytestring prefixes everywhere r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/95c981a14b19 hgext/obsolescencehacks: add byte string prefixes everywhere r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/bb6c695293ef hgext/overlay: add byte string prefixes everywhere r=glob
|
|
||
Comment 34•5 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/82f8c810dfa9 replicateowner: add byte-string prefixes to strings r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/f8b8fb6f918b serverlog: add byte string prefixes r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/91a486e93f76 unifyrepo: add byte-string prefixes to most strings r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/02741ec85de4 robustcheckout: add byte string prefixes to strings interfacing with Mercurial r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/c466f1409951 robustcheckout: use `mercurial.urllibcompat` instead of `urllib2` r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/8acdfaa72009 robustcheckout: make `test-locks` inline extension compatible with Python 3 r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/2538d277860a robustcheckout: get exception values as byte strings r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/5856e61ce85d robustcheckout: add byte string prefixes to `badserver.py` r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/eb0a328230d2 configwizard: use a byte-string when monkeypatching `util.version` r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/db378efc91e0 mozext: add byte-string prefixes everywhere r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/16dcd541c5f6 vcsreplicator: add byte-string prefixes r=glob https://hg.mozilla.org/hgcustom/version-control-tools/rev/1d45e023d637 mozautomation: support passing a text factory function to `ChangeTracker` r=glob
|
|
Assignee | |
Comment 35•4 years ago
|
||
Despite reading the original file as a bytes (hence the rb file mode),
passing values through json.loads will convert them to Python strings,
causing issues when we later attempt to use %s formatting to append
them to a byte-string. This commit wraps instances where we do this
formatting to avoid TypeErrors.
|
|
||
Comment 36•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/589de4aa124a hgmo: convert cloud region to `pycompat.bytestr` r=smacleod
|
|
||
Comment 37•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/5e76933c432a commitparser: use `%` formatting instead of `.format` for byte-strings
|
|
||
Comment 38•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/f218b146891a mozautomation: pass format string as a str and convert to bytes after https://hg.mozilla.org/hgcustom/version-control-tools/rev/ae3ce85db5de firefoxreleases: pass `path` as a system string
|
|
||
Comment 39•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/a3e1bba435a2 releasescraper: pass `response.content` to bytes regex
|
|
||
Comment 40•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/ed365b2f7ab5 mozautomation: pass regex group names as `str`
|
|
||
Comment 41•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/66dc8911b04f releasescraper: pass key as a byte-string
|
|
||
Comment 42•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/fa6e42e93be6 releasescraper: fix encoding issues when referencing values from `response.json()`
|
|
||
Comment 43•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/5c668a0a695f releasescraper: don't encode platform
|
|
||
Comment 44•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/eb02c21a764e releasescraper: use unicode to split url
|
|
||
Comment 45•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/520f32cad30d commitparser: use `htmlescape` for GitHub links too
|
|
||
Comment 46•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/cda4db0fd638 pushlog: pass datestring to `cal.parse` as a `str`
|
|
||
Comment 47•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/8932aa40dcf4 pushlog: convert `isoformat` string to bytes
|
|
||
Comment 48•4 years ago
|
||
Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/dfad904b4771 unifyrepo: use a byte-string for exit message when no pullnodes are found
|
|
Assignee | |
Comment 49•4 years ago
|
||
This is effectively complete, and we can just file new bugs for any overlooked cases.
Description
•