Innocuous "ERROR - You are using pip version 7.1.0, however version 7.1.2 is available" (which steals log-viewer focus from real errors)

NEW
Unassigned

Status

Release Engineering
Mozharness
3 years ago
2 years ago

People

(Reporter: Tomcat, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
seems like in https://treeherder.mozilla.org/logviewer.html#?job_id=8649349&repo=mozilla-inbound we got some log spew like:

11:37:05 INFO - Reading from file tmpfile_stderr
11:37:05 ERROR - You are using pip version 6.0.8, however version 6.1.1 is available.
11:37:05 ERROR - You should consider upgrading via the 'pip install --upgrade pip' command.
11:37:05 ERROR - Return code: 0 

so far just log spew it seems
This is an informative message of pip (its phoning home somehow, I guess) but should not be reporting as an error.

I'm unsure if this falls as build config or automation/mozharness. But cc'd a few people just in case they have ideas.

(solve in my mind is making the warning suppressed, or disabling phone-home entirely)
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
Falls as Testing::TaskCluster, I'd think.

Comment 3

3 years ago
To avoid recent version of pip to error out we can use it with --disable-pip-version-check or implied with --no-index.

The job seems to be a TC job.

We can also use a config file:
http://pip.readthedocs.org/en/latest/user_guide.html#configuration
In the end I landed a small pip config change to TC which disables this check...

https://hg.mozilla.org/integration/b2g-inbound/rev/ef8ca1b1f56a
Deep inside mozharness, it calls "pip freeze". Recent pip versions, by default, check if there is an update available for pip, printing a message to stderr if so.

For most of our systems, the version of pip is controlled elsewhere (system packages, puppet, etc). So the message is of no value to automation.

The downside is stderr output is logged as an error, and (with luck) as the only error for a run, generates a spurious email.

Since there are non TC users of mozharness, let's fix this upstream :)
Component: General Automation → Mozharness
QA Contact: catlee → jlund
Created attachment 8616343 [details] [diff] [review]
bz1152749.patch

All tests pass, manually applying to vcs-sync production box shows that it works to eliminate the stderr output in the mozharness log files.
Assignee: nobody → hwine
Status: NEW → ASSIGNED
Attachment #8616343 - Flags: review?(jlund)

Comment 7

3 years ago
Comment on attachment 8616343 [details] [diff] [review]
bz1152749.patch

Review of attachment 8616343 [details] [diff] [review]:
-----------------------------------------------------------------

looks good. does this mean that TC jobs will have --disable-pip-version-check in the pip freeze command twice?
Attachment #8616343 - Flags: review?(jlund) → review+
(In reply to Jordan Lund (:jlund) from comment #7)
> Comment on attachment 8616343 [details] [diff] [review]
> bz1152749.patch
> 
> looks good. does this mean that TC jobs will have
> --disable-pip-version-check in the pip freeze command twice?

Yes, it will be doubled. Doubling on the command line does not produce a failure (tested on version 6.0.8). 

For jobs using the approach in comment 4, they will will have the "supported case" of a setting in a conf file being overridden by the same setting on the command line. Definite pip bug if that doesn't work.
Comment on attachment 8616343 [details] [diff] [review]
bz1152749.patch

tl;dr: this patch broke the build farm

I tested on modern AWS machines, which have pip v6, which understands that option.

Many of the machines are running pip 1.x which does not understand the option, and thus errors out.

Backing out of default & production:
   https://hg.mozilla.org/build/mozharness/rev/eef8d170f11c
   https://hg.mozilla.org/build/mozharness/rev/1a041ea00cf3
Attachment #8616343 - Flags: checked-in+ → checked-in-
Fixing mozharness doesn't work, due to wide variety of pip versions in play across the buildfarm.

Creating/modifying the pip.conf file as in comment 4 does work on all pip versions (at least from 1.3 on).
Assignee: hwine → nobody
Status: ASSIGNED → NEW
pip now warns that we are running version 7.1.0 when 7.1.2 is available. Treeherder's log viewer jumps to the first error in the log, which is often this pip "error" instead of the real error later in the log.

ERROR - Errors received:
 INFO - Reading from file tmpfile_stderr
ERROR -  The directory '/home/worker/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
ERROR -  You are using pip version 7.1.0, however version 7.1.2 is available.
ERROR -  You should consider upgrading via the 'pip install --upgrade pip' command.
OS: Windows 7 → Unspecified
Hardware: x86 → Unspecified
Summary: ERROR - You are using pip version 6.0.8, however version 6.1.1 is available. → ERROR - You are using pip version 7.1.0, however version 7.1.2 is available.
I've run into the issue cpeterson describes (log viewer jumping to this "error" instead of the real error).  I also just ran into someone else on IRC getting confused about what this "error" means in his build-failure output (when really, there was a different error that broke his build).

Expanding bug-summary to hint at the actual human problem that this causes. A fix here would be much appreciated!
Summary: ERROR - You are using pip version 7.1.0, however version 7.1.2 is available. → Innocuous "ERROR - You are using pip version 7.1.0, however version 7.1.2 is available" (which steals log-viewer focus from real errors)
See Also: → bug 1148847
Created attachment 8676967 [details] [diff] [review]
[puppet] use the pip.conf instead

This approach doesn't fail on older pip versions.

Updated

2 years ago
Attachment #8676967 - Flags: review?(jlund)
Comment on attachment 8676967 [details] [diff] [review]
[puppet] use the pip.conf instead

Review of attachment 8676967 [details] [diff] [review]:
-----------------------------------------------------------------

sweet. so iiuc, this will only fix the buildbot case, specifically posix machines?
Attachment #8676967 - Flags: review?(jlund) → review+
This will fix the case on any machine we have puppetized.

(so not our production windows ones).

We could have it updated on the windows side, but imo they have more pressing issues to contend with.

This should work on our windows machines once they are puppetized though.
You need to log in before you can comment on or make changes to this bug.