1387970 - Permafail pkg_resources.DistributionNotFound: requests for Firefox UI tests

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: hskupin [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=121350251&repo=autoland

https://queue.taskcluster.net/v1/task/PLM8h1CZTr62PHn8e_I43Q/runs/0/artifacts/public/logs/live_backing.log

All Firefox UI tests on OS X started to fail with this failure because the requests package cannot be downloaded from the internal pypi mirror:

22:44:42     INFO -  Downloading/unpacking requests (from mozInstall==1.13->-r /Users/cltbld/tasks/task_1502083007/build/tests/config/mozbase_requirements.txt (line 8))
22:44:42     INFO -    http://pypi.pvt.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pvt.build.mozilla.org has it available

Strangely we also install a very old version of pypi:

22:44:21     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/Users/cltbld/tasks/task_1502083007/build/venv'] in /Users/cltbld/tasks/task_1502083007/build
22:44:21     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /Users/cltbld/tasks/task_1502083007/build/venv
22:44:21     INFO - Using partial env: {'VIRTUALENV_NO_DOWNLOAD': '1'}
22:44:21     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
22:44:21     INFO -  Using real prefix '/tools/python27'
22:44:21     INFO -  New python executable in /Users/cltbld/tasks/task_1502083007/build/venv/bin/python
22:44:22     INFO -  Installing distribute.............................................................................................................................................................................................done.
22:44:24     INFO -  Installing pip.................done.
22:44:24     INFO - Return code: 0
22:44:24     INFO - Getting output from command: ['/Users/cltbld/tasks/task_1502083007/build/venv/bin/pip', '--version']
22:44:24     INFO - Copy/paste: /Users/cltbld/tasks/task_1502083007/build/venv/bin/pip --version
22:44:25     INFO - Reading from file tmpfile_stdout
22:44:25     INFO - Output received:
22:44:25     INFO -  pip 1.5.5 from /Users/cltbld/tasks/task_1502083007/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg (python 2.7)

So not sure which version of virtualenv is used here.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This has actually started to permafail this morning. So I wonder if something for the TC worker on OS X has changed. Nothing is obvious on any integration branch, so it must be an external change.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Looks like Pete is out. So forwarding the ni? to wcosta.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

I wonder if this is related to bug 1387730 fixed earlier this morning where we uploaded latest version of requests[1]. I see before requests-2.18.3 the latest one was 2.13.0 which seems oldish. Could it be that some of the CI is not correctly pinned and grabs by default whatever it finds in the internal pypi mirror, hence a newer version of requests that automatically pulled more-up-to-date dependencies?

I'm happy to remove those files if needed to see if this is the culprit.
Let me know what you think.

[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1387730#c1

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The real underlying issue should really be requests, and not the INFO line which I previously mentioned. So updating the summary.

Mihai will temporarily remove the recently uploaded requests package so we can check if that causes the failure.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Henrik Skupin (:whimboo) from comment #4)
> The real underlying issue should really be requests, and not the INFO line
> which I previously mentioned. So updating the summary.
> 
> Mihai will temporarily remove the recently uploaded requests package so we
> can check if that causes the failure.

1. removing requests 2.18.3 from internal pypi
2. rerunning https://tools.taskcluster.net/groups/XOCX-yu3SlW65VEz6C5npA/tasks/HsRXZ8ghRsyweoZ9jHQhTQ/details worked.

Sounds like we can close this and we need another bug to track mozharness testing scripts[1] virtualenv issues and python versions pinning. 

[1]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Moving this to Mozharness as most likely there's something under the hood that needs to get fixed. Either some older version of requests is not pinned or we need to upgrade this script[1]'s deps. But from Buildduty standpoint, this is no longer an issue. I've blocked in for bug 1387730. 

[1]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So this is firefox-ui-tests only, and I wonder why we see this problem because each and every test suite is using mozinstall to get the application installed. And we don't do something differently.

I wonder if we need a new release of mozinstall given that requests support has been added to it via bug 1048446, and since then no new release has been created. Andrew, do you have an idea here why this is happening?

Comment 8

[Ed Morley [:emorley]]

I agree it looks like an unpinned dep. I would also guess that the breakage is due to recent versions of requests unbundling their sub-dependencies and various other package changes, that I think are not compatible with ancient pip/setuptools, eg:
https://github.com/requests/requests/issues/4006

...though note that issue was partially fixed in:
https://github.com/requests/requests/pull/4014

It's worth pointing out that these jobs are using pip 1.5.5 which is so old it's not even funny, and are also using the "you should absolutely never use this nowadays" `distribute` package too.

I think a `pip install -U pip setuptools` would fix this and many many other issues.

As for pinning deps, the firefox UI tests use a requirements file here:
https://hg.mozilla.org/mozilla-central/file/bb8de16ce00c/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py#l144

Which is here:
https://dxr.mozilla.org/mozilla-central/rev/52285ea5e54c73d3ed824544cef2ee3f195f05e6/testing/config/firefox_ui_requirements.txt

Also I see mozharness doesn't handle erroring log output very well:
`TypeError: argument of type 'NoneType' is not iterable`
https://treeherder.mozilla.org/logviewer.html#?job_id=121350251&repo=autoland&lineNumber=678

Comment 9

[Intermittent Failures Robot]

27 failures in 124 pushes (0.218 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 26
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1387970&startday=2017-08-07&endday=2017-08-07&tree=all

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Linux docker images are using pip 9.x, while the new Windows TC workers have pip 8.1.1 installed.

Wander, why are we so much behind on a recent version of pip on the OS X workers? Can we get this updated easily?

Comment 11

[Wander Lairson Costa]

(In reply to Henrik Skupin (:whimboo) from comment #10)
> Linux docker images are using pip 9.x, while the new Windows TC workers have
> pip 8.1.1 installed.
> 
> Wander, why are we so much behind on a recent version of pip on the OS X
> workers? Can we get this updated easily?

pip version is deployed by puppet, I guess. The migration to TC worker didn't touch any of this stuff, so the pip version is the same as always have been in BB time.

In summary, I don't know :)

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Wander Lairson Costa [:wcosta] from comment #11)
> > Wander, why are we so much behind on a recent version of pip on the OS X
> > workers? Can we get this updated easily?
> 
> pip version is deployed by puppet, I guess. The migration to TC worker
> didn't touch any of this stuff, so the pip version is the same as always
> have been in BB time.

Oh, I see. So maybe Dustin knows who currently works on puppet scripts or if there is already a bug open for that.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

PuppetAgain is currently managed by relops.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So I found https://hg.mozilla.org/build/puppet/file/tip/modules/python/manifests/virtualenv/settings.pp

>    $pip_version         = '1.5.5'

Last time we updated it was 2014 by Justin via bug 1003729.

Lets wait for a reply from jwatkins.

Comment 15

[Kendall Libby [:fubar] (he/him)]

Swapping NI on jake for blocking bug

Comment 16

[Intermittent Failures Robot]

22 failures in 901 pushes (0.024 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 21
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 22

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1387970&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Henrik Skupin (:whimboo) from comment #14)
> So I found
> https://hg.mozilla.org/build/puppet/file/tip/modules/python/manifests/
> virtualenv/settings.pp
> 
> >    $pip_version         = '1.5.5'
> 
> Last time we updated it was 2014 by Justin via bug 1003729.
> 
> Lets wait for a reply from jwatkins.

That is the pip version used by puppet to install virtualenvs for servers.  Not the droids you're looking for.  See bug 1388816.

Comment 18

[Intermittent Failures Robot]

1 failures in 1032 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1387970&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #5)
> > Mihai will temporarily remove the recently uploaded requests package so we
> > can check if that causes the failure.
> 
> 1. removing requests 2.18.3 from internal pypi
> 2. rerunning
> https://tools.taskcluster.net/groups/XOCX-yu3SlW65VEz6C5npA/tasks/
> HsRXZ8ghRsyweoZ9jHQhTQ/details worked.

So it looks like that we have an up-to-date version of pip running for Firefox ui tests on integration branches now, which is great to see. But sadly this will only fix our issue here for Firefox 58, but leaves out older branches. Andrew, are there attempts to get the fix on bug 1297515 uplifted to older and still supported branches?

Comment 20

[Andrew Halberstadt [:ahal]]

No, I wasn't planning on doing any uplifts. Given 57 and the risk of this patch I'd prefer not to, and think it should probably get some kind of approval. If this is something you think would be important, feel free to drive it. Otherwise I'd vote we just let it ride the trains.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrew Halberstadt [:ahal] from comment #20)
> No, I wasn't planning on doing any uplifts. Given 57 and the risk of this
> patch I'd prefer not to, and think it should probably get some kind of
> approval. If this is something you think would be important, feel free to
> drive it. Otherwise I'd vote we just let it ride the trains.

So the patches got all uplifted to any supported branch by Ryan. That means that the problem as reported here should not be present anymore.

Mihai, could you please upload requests 2.18.3 to the internal pypi mirror, and try again if the jobs are passing now?

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Henrik Skupin (:whimboo) from comment #21)
> Mihai, could you please upload requests 2.18.3 to the internal pypi mirror,
> and try again if the jobs are passing now?

Done. I added following files to the internal PyPi used by CI

-rw-r--r-- 1 root      root         88626 Oct 30 03:56 requests-2.18.3-py2.py3-none-any.whl
-rw-r--r-- 1 root      root        126008 Oct 30 03:55 requests-2.18.3.tar.gz

I've retriggered a maple-based test-macosx64-nightly/opt-firefox-ui-functional-local-e10s task[1] to avoid messing with central builds. The downside of this is that the task is currently pending with low-priority. If you feel like we can retrigger a central one with higher priority, please do so or let me know to assist. 

[1]: https://tools.taskcluster.net/groups/dD1E_3MxSDG87WcABjs-xg/tasks/afy8ZP0eRhmdG-7A1luwNg/details

Comment 23

[Mihai Tabara [:mtabara]⌚️GMT]

Retriggering resulted in green build[0] with 2.18.3 requests python package installed as logs[1] depict. 

[0]: https://tools.taskcluster.net/groups/dD1E_3MxSDG87WcABjs-xg/tasks/afy8ZP0eRhmdG-7A1luwNg/details
[1]: https://public-artifacts.taskcluster.net/afy8ZP0eRhmdG-7A1luwNg/1/public/logs/live_backing.log

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's great to see. So this actually is fixed now by using a more modern pip release, as updated by Andrew on bug 1297515.

Comment 25

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.