1244746 - pip 7.1.2 -> 8.0.2 errors make using treeherder's log viewer more difficult

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Nathan Froyd [:froydnj]]

https://treeherder.mozilla.org/logviewer.html#?job_id=16163275&repo=try

Jumps me to an error about pip versions:

 15:31:33    ERROR - Errors received:
 15:31:33     INFO - Reading from file tmpfile_stderr
 15:31:33    ERROR -  You are using pip version 7.1.2, however version 8.0.2 is available.
 15:31:33    ERROR -  You should consider upgrading via the 'pip install --upgrade pip' command.

I don't really care about this error, it's out of my control, and it doesn't actually cause problems for the build.  Worse, Treeherder's log viewer thinks this is the interesting error that caused my build to fail, and wants to direct my attention to it.  Can we either ignore stderr from pip invocations, patch pip to not emit this message, or upgrade?  (Why is our version of pip even informing us about new versions, anyway?)

Comment 1

[Selena Deckelmann :selenamarie :selena]

Greg - was this fixed in recent changes to th/tc log processing?

Comment 2

[Greg Arndt [:garndt]]

This is an error matching regular expression that was not added by our team and is something living in treeherder.  It catches all errors and displays them rather than just when the task itself has failed.  The harness outputs this error but chooses to not treat it as fatal.

As far as something on the taskcluster side, the image should/could be updated to have a newer version of pip.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Moving over to TH. I'll add a bug to update pip.

Comment 4

[Ed Morley [:emorley]]

This is a mozharness (or some other upstream script) issue.

Treeherder is correctly highlighting lines that have the mozharness error prefix, such as:

15:31:33 ERROR - Errors received: 

This line should have been output as:

15:31:33 INFO - Errors received: 

...if it wasn't an issue.

(Or to be more precise, the mozharness log strings should match the exit code; either it's an error or it's not).

Comment 5

[Aki Sasaki (not active)]

Upgrading pip is overall the best solution.  Mozharness could run |pip install --upgrade pip| after venv creation, or we can upgrade pip in bug 1269853.  I think the latter is cleaner.

To stem the bleeding sooner, we may be able to add |--disable-pip-version-check| to the |pip freeze| command in mozharness.  (Looks like :emorley fixed this in travis: https://github.com/travis-ci/travis-ci/issues/5499 :)  That fix would address this symptom and make upgrading pip less urgent.

Comment 6

[Aki Sasaki (not active)]

Attachment: MozReview Request: bug 1244746 - silence mozharness pip upgrade errors in pip freeze. r?jlund

Review commit: https://reviewboard.mozilla.org/r/50269/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/50269/

Comment 7

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=49132010f819

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 8748418 [details]
MozReview Request: bug 1244746 - silence mozharness pip upgrade errors in pip freeze. r?jlund

From try: |pip: error: no such option: --disable-pip-version-check|
https://treeherder.mozilla.org/logviewer.html#?job_id=20316661&repo=try#L396

Comment 9

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b05608eb2c8d

Comment 10

[Aki Sasaki (not active)]

Setting PIP_DISABLE_PIP_VERSION_CHECK=1 in env seems to break windows for some reason.
Trying ignore_errors=True, which is my last quick'n'dirty fix idea.

Comment 11

[Aki Sasaki (not active)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=507fa74b623b

Comment 12

[Aki Sasaki (not active)]

Attachment: MozReview Request: bug 1244746 - mozharness pip freeze ignore_errors=True. r?jlund

Review commit: https://reviewboard.mozilla.org/r/50319/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/50319/

Comment 13

[Jordan Lund (:jlund)]

Comment on attachment 8748487 [details]
MozReview Request: bug 1244746 - mozharness pip freeze ignore_errors=True. r?jlund

https://reviewboard.mozilla.org/r/50319/#review47285

r+ as a stop gap for Bug 1269853

Comment 14

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/56596c60064e

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/56596c60064e

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Aki Sasaki [:aki] from comment #10)
> Setting PIP_DISABLE_PIP_VERSION_CHECK=1 in env seems to break windows for
> some reason.

Perhaps needed to set "1" instead of 1. The other issue I've hit is unicode strings making it barf.

Comment 17

[Aki Sasaki (not active)]

https://hg.mozilla.org/try/rev/b05608eb2c8d3d8cedb4ca5eb6bd4b1586eaee48
Probably the latter.  Or maybe I needed to self.query_env(...) to not drop the other env vars on the floor.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Wow, that's really weird, and quite different from what i hit.