Closed Bug 852473 Opened 11 years ago Closed 11 years ago

We should not rely on third party sites as part of the build (B2G uses http://pypi.python.org/packages/source/m/mozInstall/mozInstall-1.5.tar.gz)

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: mozilla)

References

Details

(Keywords: intermittent-failure, Whiteboard: [buildduty])

Attachments

(1 file)

--no-index
11.72 KB, patch
armenzg
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review 
Yet again we sadly have tree bustage because a dependency for the build has snuck in, that relies on an external site.

In addition to fixing this one, can we vet for other uses of non-mozilla properties please :-)

https://tbpl.mozilla.org/php/getParsedLog.php?id=20816680&tree=Mozilla-Inbound#error0

{
03:14:24     INFO - #####
03:14:24     INFO - ##### Running create-virtualenv step.
03:14:24     INFO - #####
03:14:24     INFO - Creating virtualenv /home/cltbld/talos-slave/test/build/venv
03:14:24     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/home/cltbld/talos-slave/test/build/venv'] in /home/cltbld/talos-slave/test/build
03:14:24     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /home/cltbld/talos-slave/test/build/venv
03:14:27     INFO -  Using real prefix '/usr'
03:14:27     INFO -  New python executable in /home/cltbld/talos-slave/test/build/venv/bin/python
03:14:28     INFO -  Installing distribute....................................................................................................................................................................................done.
03:14:29     INFO - Return code: 0
03:14:29     INFO - Installing mozinstall into virtualenv /home/cltbld/talos-slave/test/build/venv
03:14:29     INFO - Running command: ['/home/cltbld/talos-slave/test/build/venv/bin/pip', 'install', '-r', 'tests/b2g/b2g-unittest-requirements.txt', '--find-links', 'http://puppetagain.pub.build.mozilla.org/data/python/packages', 'mozinstall'] in /home/cltbld/talos-slave/test/build
03:14:29     INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/pip install -r tests/b2g/b2g-unittest-requirements.txt --find-links http://puppetagain.pub.build.mozilla.org/data/python/packages mozinstall
03:14:47     INFO -  Downloading/unpacking mozinstall
03:14:47     INFO -    Error <urlopen error timed out> while getting http://pypi.python.org/packages/source/m/mozInstall/mozInstall-1.5.tar.gz#md5=14188bdc11065f86dd48497b66ce9dd8 (from http://pypi.python.org/simple/mozInstall/)
03:14:47     INFO -  Exception:
03:14:47    ERROR -  Traceback (most recent call last):
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/basecommand.py", line 130, in main
03:14:47     INFO -      self.run(options, args)
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/commands/install.py", line 223, in run
03:14:47     INFO -      requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/req.py", line 917, in prepare_files
03:14:47     INFO -      self.unpack_url(url, location, self.is_download)
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/req.py", line 1023, in unpack_url
03:14:47     INFO -      return unpack_http_url(link, location, self.download_cache, only_download)
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/download.py", line 429, in unpack_http_url
03:14:47     INFO -      resp = _get_response_from_url(target_url, link)
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/download.py", line 458, in _get_response_from_url
03:14:47     INFO -      resp = urlopen(target_url)
03:14:47     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/pip-0.8.2-py2.6.egg/pip/download.py", line 84, in __call__
03:14:47     INFO -      response = urllib2.urlopen(self.get_request(url))
03:14:47     INFO -    File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
03:14:47     INFO -      return _opener.open(url, data, timeout)
03:14:47     INFO -    File "/usr/lib/python2.6/urllib2.py", line 383, in open
03:14:47     INFO -      response = self._open(req, data)
03:14:47     INFO -    File "/usr/lib/python2.6/urllib2.py", line 401, in _open
03:14:47     INFO -      '_open', req)
03:14:47     INFO -    File "/usr/lib/python2.6/urllib2.py", line 361, in _call_chain
03:14:47     INFO -      result = func(*args)
03:14:47     INFO -    File "/usr/lib/python2.6/urllib2.py", line 1130, in http_open
03:14:47     INFO -      return self.do_open(httplib.HTTPConnection, req)
03:14:47     INFO -    File "/usr/lib/python2.6/urllib2.py", line 1105, in do_open
03:14:47     INFO -      raise URLError(err)
03:14:47     INFO -  URLError: <urlopen error timed out>
03:14:47     INFO -  Storing complete log in /home/cltbld/.pip/pip.log
}
Keywords: intermittent-failure
I'm not sure how to word this, but we shouldn't even be depending on resources within Mozilla in most cases. For example, we used to have tests that referenced made requests to www.mozilla.org. In general, requiring the any remote resource for any part of a compile step or actual test (as opposed to setting up a test) is very bad form (b2g builds that pull xulrunner, I'm looking at you).
Yeah I agree - it was more I couldn't think of a better way to describe "anything other than ftp.m.o, *.build.m.o, hg.m.o/*, ..." :-)
IIRC the public data/python/packages mirrors to some internal hosts for us to use and we should not puppetagain.pub directly.

On another note, I think we have to add --no-index with --find-links:

 -f, --find-links <URL>       URL to look for packages at
 --no-index                   Ignore package index (only looking at --find-links URLs instead)

Does this make sense?
IIUC we should aim for these hosts:
releng-puppet1.build.scl1.mozilla.com
releng-puppet1.build.mtv1.mozilla.com
releng-puppet1.srv.releng.scl3.mozilla.com

The problem I find is that we can't reach data/python/packages through a web head.
I think Dustin wants us to point at http://repos/python/packages/ , since "repos" is a dns CNAME that points at the nearest one of those three hosts.

Adding Jonathan and Andrew; did you guys realize we were adding a new dependency that wasn't in puppetagain?
It seems like it works for us:
[cltbld@talos-r3-fed-001 ~]$ host repos
repos.build.scl1.mozilla.com is an alias for releng-puppet1.build.scl1.mozilla.com.
releng-puppet1.build.scl1.mozilla.com has address 10.12.51.224
[cltbld@talos-r3-fed-001 ~]$ wget repos/python/packages/carrot-0.10.7.tar.gz--2013-03-19 09:51:47--  http://repos/python/packages/carrot-0.10.7.tar.gz
Resolving repos... 10.12.51.224
Connecting to repos|10.12.51.224|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62069 (61K) [application/x-gzip]
Saving to: `carrot-0.10.7.tar.gz'

100%[===========================================================================================>] 62,069      --.-K/s   in 0.009s  

2013-03-19 09:51:47 (6.53 MB/s) - `carrot-0.10.7.tar.gz' saved [62069/62069]
The main reason I was avoiding pointing at "repos" is it's another thing that keeps people outside our network from taking our configs and using them easily.  Maybe now that we have multiple config file support that's not as big an issue.
You should be using repos since that's fault tolerant.  Just specifying one puppet server or the pub website is not.
(In reply to Aki Sasaki [:aki] from comment #6)
> I think Dustin wants us to point at http://repos/python/packages/ , since
> "repos" is a dns CNAME that points at the nearest one of those three hosts.
> 
> Adding Jonathan and Andrew; did you guys realize we were adding a new
> dependency that wasn't in puppetagain?

We haven't added a new dependency here; 'mozinstall' is not a recent addition to the mozharness script, it's been around for a long time:

http://hg.mozilla.org/build/mozharness/file/7599724311e0/scripts/b2g_emulator_unittest.py#l118

And, it is in puppetagain:

http://puppetagain.pub.build.mozilla.org/data/python/packages/mozInstall-1.4.tar.gz

so, I'm not sure why it's intermittently failing.
Ok. I wondered if mozinstall==1.5 snuck in somewhere.
It's possible that puppetagain.pub.build.m.o was down at the time of these issues; pointing at repos/ would fix that.
(In reply to Aki Sasaki [:aki] from comment #8)
> The main reason I was avoiding pointing at "repos" is it's another thing
> that keeps people outside our network from taking our configs and using them
> easily.  Maybe now that we have multiple config file support that's not as
> big an issue.

The other issue is that people outside of rel-eng/IT can't determine what's in "repos", so we can't be sure when/if we need to update packages there.
(In reply to Armen Zambrano G. [:armenzg] from comment #4)
> IIRC the public data/python/packages mirrors to some internal hosts for us
> to use and we should not puppetagain.pub directly.
> 
> On another note, I think we have to add --no-index with --find-links:
> 
>  -f, --find-links <URL>       URL to look for packages at
>  --no-index                   Ignore package index (only looking at
> --find-links URLs instead)
> 
> Does this make sense?

I think --no-index makes sense if we want to hit our find-links only page only, yes.
Maybe we can start with --no-index everywhere.

Would changing the CNAME to "puppetagain-internal" (or similar) instead of "repos" be a more clear CNAME?
Assignee: nobody → aki
Ok.  I'm going to use this bug for --no-index only, and bug 843568 for puppetagain -> repos.
Attached patch --no-index — — Splinter Review 
* adds pip_index = False to all config files with a find_links

* adds --pip-index and --no-pip-index commandline options.  I debated on naming for a while.  The fact that we'll want a --no-___ option made me avoid --pip-no-index and --no-pip-no-index (or --no-index --no-no-index, or --no-index --index).

* pep8-ified mozharness.base.python


I could have also assumed that if we ever specify --find-links, we'll want --no-index.  I'm not sure that's the case for developers, so I went this route.  I don't have strong feelings about it, though.
Attachment #726824 - Flags: review?(armenzg)
Comment on attachment 726824 [details] [diff] [review]
--no-index

Review of attachment 726824 [details] [diff] [review]:
-----------------------------------------------------------------

r+ after addressing questions below.

::: mozharness/base/python.py
@@ +208,5 @@
>          for link in c.get('find_links', []):
>              command.extend(["--find-links", link])
>  
> +        if not c["pip_index"]:
> +            command += ['--no-index']

Should this be --no-pip-index?
Attachment #726824 - Flags: review?(armenzg) → review+
(In reply to Armen Zambrano G. [:armenzg] from comment #17)
> Comment on attachment 726824 [details] [diff] [review]
> --no-index
> 
> Review of attachment 726824 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ after addressing questions below.
> 
> ::: mozharness/base/python.py
> @@ +208,5 @@
> >          for link in c.get('find_links', []):
> >              command.extend(["--find-links", link])
> >  
> > +        if not c["pip_index"]:
> > +            command += ['--no-index']
> 
> Should this be --no-pip-index?

No, it's |pip install --no-index| .

pip install --help
<snip>
 --no-index                   Ignore package index (only looking at --find-links URLs instead)
Comment on attachment 726824 [details] [diff] [review]
--no-index

http://hg.mozilla.org/build/mozharness/rev/c840a99c6b7d
Merged to production.
Attachment #726824 - Flags: checked-in+
This should be fixed.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Looks like now that we have --no-index, we get this for Marionette:

14:21:38     INFO -  Downloading/unpacking mozcrash>=0.5 (from marionette-client)
14:21:38     INFO -    Could not find a version that satisfies the requirement mozcrash>=0.5 (from marionette-client) (from versions: )
14:21:38     INFO -  No distributions matching the version for mozcrash>=0.5 (from marionette-client)
Depends on: 852741
(In reply to Aki Sasaki [:aki] from comment #20)
> This should be fixed.

Thank you for doing this :-)
(In reply to Aki Sasaki [:aki] from comment #6)
> I think Dustin wants us to point at http://repos/python/packages/ , since
> "repos" is a dns CNAME that points at the nearest one of those three hosts.

(In reply to Amy Rich [:arich] [:arr] from comment #9)
> You should be using repos since that's fault tolerant.  Just specifying one
> puppet server or the pub website is not.

Almost - 'repos' is just a CNAME for a puppet server, so it's no more fault-tolerant.  The correct fix here is to use a pip.conf that lists multiple mirrors (bug 843568) and to list explicit package versions everywhere, rather than pulling "latest" (bug 851305).
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: