Closed
Bug 1406703
Opened 7 years ago
Closed 6 years ago
[tracker] switch to python 3
Categories
(Socorro :: General, task, P2)
Socorro
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: willkg, Assigned: willkg)
References
Details
Attachments
(4 files)
Python 2 is going away. It behooves us to think about this now so that we can figure out our library situation, rewrite and rearchitect things, and generally have time to switch to Python 3 leisurely.
This tracker covers all the work required to switch to Python 3.
Assignee | ||
Comment 1•7 years ago
|
||
I'm thinking this is a good plan of attack:
1. make a list of requirements which don't support Python 3; write up a bug for each one
2. deal with all those bugs: remove code, rewrite code, update libraries, switch to different libraries, etc
3. update socorro code and Docker infrastructure to work with Python 3
Maybe this is a 2018q1 project? In the meantime, as we make changes, we should be asking ourselves, "Will this work with Python 3?"
Comment 2•7 years ago
|
||
Step 1 is probably a good 2017q4 goal
Assignee | ||
Comment 3•6 years ago
|
||
We're all set with requirements now--all the requirements support Python 3.
We have a rough infrastructure for running tests in Python 3. This will let us work through updating code and tests incrementally.
We should fix linting with flake8 in Python 3 first. Once that's done, one way to break up the work in chunks is this:
1. socorro
* socorro.lib
* socorro.external
* socorro.external.boto
* socorro.external.postgres
* socorro.external.es
2. crontabber
3. processor
4. webapp
Helpful resources:
Python porting guide:
https://docs.python.org/3/howto/pyporting.html
particularly this section: https://docs.python.org/3/howto/pyporting.html#update-your-code
Cheat sheet: Writing Python 2-3 compatible code
http://python-future.org/compatible_idioms.html#essential-syntax-differences
Supporting Python 3: An in-depth guide
http://python3porting.com/
Six documentation:
https://pythonhosted.org/six/
Assignee | ||
Comment 4•6 years ago
|
||
Just so I don't forget where I wrote it again, I wrote up some docs on iteration steps here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1460035#c2
A big change in python 2 and python 3 is the base string data type. ascii(python 2) and unicode(python 3). Many errors will come from pivoting the desired behavior of these data types. Many of the test failed because of the discrepancies in data types for what these test were expecting. Majority of the task will be finding where the data changed to fix it accordingly and modifying or removing test. Code isn't commented very much so stay in constant contact with the lead dev asking about the stack traces from error output and each individual unit test if need be. Steps to solving I used were:
If you don't know python start here:
https://github.com/gregmalcolm/python_koans (learn the language as much as possible)
- Will help sparingly because the code is written in sophisticated way beyond any of these test suites but it still helps
Problem solving:
1) Check python 2/3 cheat sheet
http://python-future.org/compatible_idioms.html#essential-syntax-differences
2) check six library for desired import functionality
3) Stack trace
- Check the code in working env. Trace functions looking for value modifications as far as possible
- If web services are involved look up what you can about whats going on with the code as it pertains to the web service
- Consider using conditional statements for doing something if its a Python 2 env or python 3 env.
- talk to lead dev on what you can assume about the test in possibly changing the test in the testing and expected values.
There is more commentary on the specific problems I faced on the work I did. They haven't all been the same but the methods I used in solving them were similar.
https://github.com/pulls?q=is%3Apr+author%3Aceddy-cedd+archived%3Afalse+is%3Aclosed
(In reply to cmiller from comment #5)
> A big change in python 2 and python 3 is the base string data type.
> ascii(python 2) and unicode(python 3). Many errors will come from pivoting
> the desired behavior of these data types. Many of the test failed because of
> the discrepancies in data types for what these test were expecting. Majority
> of the task will be finding where the data changed to fix it accordingly and
> modifying or removing test. Code isn't commented very much so stay in
> constant contact with the lead dev asking about the stack traces from error
> output and each individual unit test if need be. Steps to solving I used
> were:
>
> If you don't know python start here:
> https://github.com/gregmalcolm/python_koans (learn the language as much
> as possible)
> - Will help sparingly because the code is written in sophisticated way
> beyond any of these test suites but it still helps
>
>
> Problem solving:
>
> 1) Check python 2/3 cheat sheet
>
> http://python-future.org/compatible_idioms.html#essential-syntax-differences
>
> 2) check six library for desired import functionality
>
> 3) Stack trace
> - Check the code in working env. Trace functions looking for value
> modifications as far as possible
> - If web services are involved look up what you can about whats going
> on with the code as it pertains to the web service
> - Consider using conditional statements for doing something if its a
> Python 2 env or python 3 env.
> - talk to lead dev on what you can assume about the test in possibly
> changing the test in the testing and expected values.
>
> There is more commentary on the specific problems I faced on the work I did.
> They haven't all been the same but the methods I used in solving them were
> similar.
>
https://github.com/mozilla-services/socorro/pull/4487
https://github.com/mozilla-services/socorro/pull/4508
Assignee | ||
Comment 7•6 years ago
|
||
Assignee: nobody → willkg
Comment 8•6 years ago
|
||
Commit pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/3308852985f0d8ec3234bfa5eab0b9fa7cd45252
Merge pull request #4600 from willkg/cleanup-makefile
bug 1406703: Cleanup makefile
Assignee | ||
Comment 9•6 years ago
|
||
Comment 10•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/e27372eac6fccc8e13c8be8cd2a550194f216d16
bug 1406703: add webapp tests to python3 test scaffolding
https://github.com/mozilla-services/socorro/commit/bece727d51d4b70e63a00fa4c3ce540f71fcd282
Merge pull request #4651 from willkg/1406703-python3-webapp
bug 1406703: add webapp tests to python3 test scaffolding
Assignee | ||
Updated•6 years ago
|
Priority: -- → P2
Assignee | ||
Comment 11•6 years ago
|
||
Comment 12•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/b249ef6684346f3e2f4f523bc0762e26867b002e
bug 1406703: add comments for what's remaining
This adds commented out lines for what remains to get working in Python 3.
This will make it easier to make sure we've got everything.
https://github.com/mozilla-services/socorro/commit/3f6d58554bd442ac58b661a6210ffffed54f83ad
Merge pull request #4697 from willkg/1406703-python3
bug 1406703: add comments for what's remaining
Assignee | ||
Comment 13•6 years ago
|
||
Comment 14•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/6d0fa71f14e4bcc8ad8fb66ca5cc72ccb5971505
bug 1406703: redo image with Python 3 and remove Python 2-isms
https://github.com/mozilla-services/socorro/commit/76faea5b078a69feb936d516479567d3ae4fa7e4
bug 1406703: fix socorro_aws_s3.sh script
This was using print statement which doesn't exist in Python 3.
https://github.com/mozilla-services/socorro/commit/45defda60cc2eb4ee7b09de3039ad3c284fdfa2a
bug 1406703: fix fetch_crash_data.py
This was trying to serialize a dictkeys--this fixes that.
https://github.com/mozilla-services/socorro/commit/340c9137ffdba2996bb76813c0dff1b4bc314dc5
bug 1406703: fix rabbitmq crashstorage
The body from a channel.basic_get is a bytes. Since that always consists
of a crash id and that's always a string, this adjusts the code to decode
it first.
https://github.com/mozilla-services/socorro/commit/d2670a82f5b82a9993ac443bd81f323969e20372
fix bug 1406703: update Python version in metadata
https://github.com/mozilla-services/socorro/commit/90214cd165fdb972d15f84934f2ce0efc2bf4f24
fix bug 1406703: Remove commented out code in .circleci/config.yml
https://github.com/mozilla-services/socorro/commit/eec0c4ed742e3e9802d46abb49eb9171f9460ed4
fix bug 1406703: switch to python:3.6.7-slim-jessie
https://github.com/mozilla-services/socorro/commit/6ec93f01df1ef387c3f60bcd70537a0a1d535385
Merge pull request #4740 from willkg/1406703-python3
fix bug 1406703: migrate to python 3
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•