1436639 - [flake8] Explicitly use version of python run with mach

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, enhancement, P1)

Description

[Andrew Halberstadt [:ahal]]

I noticed the flake8 linter will use whatever the default 'python' is. For example, the machine reviewbot runs on has python 3 by default, so we're seeing 'undefined name basestring' errors.

It looks like instead of:
flake8 ...

We need to run:
python2.7 -mflake8 ...

(Though using sys.executable instead of hardcoding anything)

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1436639 - [lint] Make sure flake8 is run with same python as |mach lint| was,

This fixes a bug which can happen when the default version of python differs from
the version of python used with mach.

For example, mach explicitly looks for python2.7. This means running |mach lint -l flake8|
should also run flake8 with version 2.7. But if the default is python3, and flake8 is also
installed there, the subprocess call that invokes flake8 will run under python3. This can
lead to errors like "undefined name 'basestring'" and other 2to3 gotchas.

This patch ensures that we run:
python2.7 -m flake8

which explicitly runs flake8 against the same interpreter as mach, no matter the default.

Review commit: https://reviewboard.mozilla.org/r/218640/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/218640/

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8949275 [details]
Bug 1436639 - [lint] Make sure flake8 is run with same python as |mach lint| was,

https://reviewboard.mozilla.org/r/218640/#review224496

looks good

Comment 3

[Andrew Halberstadt [:ahal]]

Try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ccea03775c6d951d7702c4411b5937947182a29

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c9d150401eea
[lint] Make sure flake8 is run with same python as |mach lint| was, r=jmaher

Comment 5

[Andrew Halberstadt [:ahal]]

Oops, I forgot that sys.executable doesn't point to the virtualenv python due to bug 1436777. Out of an abundance of caution I'm going to ask for this to be backed out, fix bug 1436777, and then re-land.

Comment 6

[Noemi Erli[:noemi_erli]]

Backed out changeset c9d150401eea (bug 1436639) for potential local workflow issues r=ahal on a CLOSED TREE

Push with issues: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c9d150401eea734abb9815f237591475f3a98df4&filter-classifiedState=unclassified&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable

Backout push: https://hg.mozilla.org/integration/autoland/rev/5df904a4b6965791f5989f0c610872ea72486990

Comment 7

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/c9d150401eea

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout for potential workflow issues:
https://hg.mozilla.org/integration/autoland/rev/5df904a4b6965791f5989f0c610872ea72486990
https://hg.mozilla.org/mozilla-central/rev/5df904a4b696

Comment 9

[Andrew Halberstadt [:ahal]]

Apparently modifying sys.executable can cause problems in some cases, so it's not something we want to do all the time. Luckily sys.prefix *does* get modified, so I can come up with a new patch that uses that instead.

Enough has changed that this should get a re-review. Patch coming shortly.

Comment 10

[Andrew Halberstadt [:ahal]]

Comment on attachment 8949275 [details]
Bug 1436639 - [lint] Make sure flake8 is run with same python as |mach lint| was,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/218640/diff/1-2/

Comment 11

[Andrew Halberstadt [:ahal]]

Comment on attachment 8949275 [details]
Bug 1436639 - [lint] Make sure flake8 is run with same python as |mach lint| was,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/218640/diff/2-3/

Comment 12

[Andrew Halberstadt [:ahal]]

Comment on attachment 8949275 [details]
Bug 1436639 - [lint] Make sure flake8 is run with same python as |mach lint| was,

Sorry, the old patch you reviewed had a problem, and I think this warrants a re-review.

Basically sys.executable doesn't get updated to the build system python environment, but sys.prefix does.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8949275 [details]
Bug 1436639 - [lint] Make sure flake8 is run with same python as |mach lint| was,

https://reviewboard.mozilla.org/r/218640/#review225628

this doesn't look scary

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/365a91ab41e2
[lint] Make sure flake8 is run with same python as |mach lint| was, r=jmaher

Comment 15

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/365a91ab41e2