Closed Bug 1318969 Opened 8 years ago Closed 8 years ago

Can't update buildapi because script expects to use puppetagain.pub.build.m.o

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nthomas, Assigned: nthomas)

References

Details

Bug 1291660 made that go away.
Do we have some information on what the script does or where to find it? I'm not sure why buildapi should be relying on a public interface of releng puppet. Likely the fix is just to have it use the private interface (and make sure the right flows are open) if it's actually required.
Flags: needinfo?(nthomas)
Specifically this in /data/releng/src/buildapi/update.sh: virtualenv/bin/pip install -U --no-index --find-links=http://puppetagain.pub.build.mozilla.org/data/python/packages/ --trusted-host puppetagain.pub.build.mozilla.org buildapi==$VERSION I've tried changeing it to use: virtualenv/bin/pip install -U --no-index --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages/ --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com buildapi==$VERSION but there appears to be no netflow from relengwebadm.private.scl3 to relengwebadm.private.scl3.
Blocks: 1306718
I did a grep for puppetagain from /data/releng and didn't find any other occurrences. I'm wondering why we handles this one thing differently than all the rest. It looks like other sites are using pypi.pub.build.mozilla.org Three questions: 1) Should we be retrieving things from the internal build network for this or should we even be hosting it on releng's puppet server (which we removed public access to for security reasons)? Is there a reason it doesn't live on the pypi mirror, too? 2) Is production functionality of building/testing/signing/shipping Firefox impacted by this right now, or is this just an impediment for releasing new versions of the tool (e.g. are things on fire)? 3) I know that we were talking about moving things off of relengweb. Is buildapi close enough to just migrate it?
re 1), idk for sure. I suspect we just changed deployment strategies with mozpool and relengapi. Personally I find it hard to reason about updating files into our pypi, given the complicated web of users and unknown set of requirements files (eg requests or other packages used by many things). buildapi doesn't have that problem though 2) We're only blocking some cleanup for treeherder (bug 1306718), nothing critical path 3) do you mean rok's new setup ? idk really know how he's going. buildapi is a little crufty though but could do with some more grunt
Flags: needinfo?(nthomas)
I talked a bit with Dustin, and he agrees that the correct path forward here is to use the pypi mirror (and isn't sure why anyone ever used the puppet server). So the path forward is to make sure the packages that are required are available there, and modify the update script to pull from there like it does with relengapi.
Assignee: relops → nobody
Component: RelOps → Tools
Product: Infrastructure & Operations → Release Engineering
QA Contact: arich → hwine
Flags: needinfo?(rgarbas)
FTR, after doing 'pip install -U --no-index --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages/ --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com buildapi==0.3.4', this is the output of 'pip freeze': amqp==1.4.6 anyjson==0.3.3 Beaker==1.5.4 buildapi==0.3.4 decorator==3.4.0 FormEncode==1.2.4 kombu==3.0.26 Mako==0.4.1 MarkupSafe==0.23 MySQL-python==1.2.5 nose==1.0.0 Paste==1.7.5.1 PasteDeploy==1.5.0 PasteScript==1.7.3 Pygments==1.4 Pylons==1.0 pytz==2015.6 Routes==1.12.3 simplejson==3.3.1 SQLAlchemy==0.8.3 Tempita==0.5.1 WebError==0.10.3 WebHelpers==1.3 WebOb==1.0.8 WebTest==1.2.3 Some are missing on our pypi (eg Pylons), some have different versions (SQLAlchemy would resolve to 0.8.0b2), others would be the same (eg pytz at 2015.6). We can backfill where necessary, and maybe pin some versions in setup.py to minimize change+validation.
In ``mozilla-releng/services`` setup official pypi index is used when creating binaries and those are stored in ``https://s3.amazonaws.com/releng-cache/`` so that we avoid rebuilding when re-deploying or when setting up local environment. you can imagine that this speeds up deployment quite a bit. also this binary cache is not only meant for python package but for anything we depend on in projects. I would actually but very happy if current relengapi is in non deployable state since it will - not that it doesn't already - make more pressure to switch to new services. Or to put it in words of wisdom that I learned in my first week when joining mozilla "Progress is made when things brake." (source of this quote is kept confidential :P) as for the blocking of treehurder. this is actually first service that is already running via mozilla-releng/seervices setup ... you can submit PR, and ping me review it. once it gets merged it will be automatically promoted to production (with a staging step in between). here is the sourcecode for treehurder now -> https://github.com/garbas/mozilla-releng-services/tree/master/src/releng_frontend/src/trychooser
Flags: needinfo?(rgarbas)
(In reply to Nick Thomas [:nthomas] from comment #7) > FTR, after doing 'pip install -U --no-index > --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/ > packages/ --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com > buildapi==0.3.4', this is the output of 'pip freeze': I should have specified 0.3.24 for buildapi here, but it doesn't actually affect any of the deps. So I tried switching to our pypi - the mechanics seem to work OK, but I haven't verified it actually runs OK. These packages were added to http://pypi.pub.build.mozilla.org/pub/: buildapi-0.3.24, Pylons-1.0, WebTest-1.2.3, Routes-1.12.3, Routes-1.12.3, Beaker-1.5.4, FormEncode-1.2.4, Paste-1.7.5.1, PasteDeploy-1.5.0, PasteScript-1.7.3, Routes-1.12.3, WebError-0.10.3, WebHelpers-1.3, WebTest-1.2.3 (everything following Pylons is a requirements of that) So now if I run the 'pip install -U ... buildapi-0.3.24' against pypi it works, with the following version differences: amqq 1.4.6 --> 1.4.9 kombu 3.0.26 --> 3.0.37 Mako 0.4.1 --> 1.0.1 MarkupSafe 0.23 --> 0.18 !! nose 1.0.0 --> 1.3.7 Pygments 1.4 --> 2.0.2 simplejson 3.3.1 --> 3.3.0 !! SQLAlchemy 0.8.3 --> 0.7.9 !! Three downgrades are marked with '!!', and are potential footguns if we need to build from scratch in the future. Should probably update those on pypi too, if we know what else consumes them on pypi. To make things fun the existing virtualenv (relengwebadm:/data/releng/src/buildapi/virtualenv/) has packages not listed in setup.py for buildapi: celery==3.1.22, ConcurrentLogHandler==0.9.1, gviz-api.py==1.8.2, newrelic==2.68.0.50, taskcluster==0.3.4 and also has kombu 3.0.30 (no idea where that came from). I made a backup of .../src/buildapi/virtualenv/ and tried running 'virtualenv/bin/pip install -U --no-index --find-links=http://pypi.pub.build.mozilla.org/pub/ --trusted-host pypi.pub.build.mozilla.org buildapi==0.3.24' on it. The new buildapi was pulled in, along with amqp, kombu, Mako, nose and Pygments; it left SQLAlchemy, simplejson, and MarkupSafe alone ('Requirement already up-to-date'). kombu failed to build a wheel, but pip returned an exit status of 0. I'm not going to deploy to the webheads right now, and restored the virtualenv backup.
Do you know when you might be able to take another look at this? I'm just keen to get bug 1306718 landed so we can continue the Treeherder short SHA deprecation :-)
Flags: needinfo?(nthomas)
I noticed a way to work around the pip issue, so I'll attempt to deploy bug 1306718 this week.
I'm not going to bother with pointing to pypi.pub.build.m.o for buildapi dependencies, just buildapi itself. Nor am I going to update the requirements list for buildapi to reflect what is actually installed. This is because buildapi is getting the minimum support needed to keep it running until we've migrated off buildbot. This avoids rebuilding the virtualenv and the risk of bustage from version shifts. What I have done is * upgraded pip from v8.0.3 to v9.0.1 in the production virtualenv (relengwebadm:/data/releng/src/buildapi/virtualenv). This had to be activated for this to work properly, there's something screwy going on. The motivation was to avoid an error with building a wheel for buildapi, not something we need * modified the update script so that it does virtualenv/bin/pip install -U --no-deps --no-index --find-links=https://pypi.pub.build.mozilla.org/pub/ --trusted-host pypi.pub.build.mozilla.org buildapi==$VERSION ** The --no-deps is new, and the host in --find-links and --trused-host has changed from puppetagain to our pypi * updated the docs for deploying buildapi at https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#BuildAPI Similar changes for staging, except the virtualenv at relengwebadm:/data/releng-stage/src/buildapi/virtualenv had a broken pip v1.5.4, and I had to easy_install v9.0.1 of pip. There are quite different versions of dependencies in this venv compared to production. :-/ Both staging and production have buildapi 0.3.26 from bug 1306718 deployed, so --> FIXED.
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(nthomas)
Resolution: --- → FIXED
FTR, the 'pip freeze --all' output for the current production virtualenv is amqp==1.4.6 anyjson==0.3.3 Beaker==1.5.4 buildapi==0.3.26 ConcurrentLogHandler==0.9.1 decorator==3.4.0 FormEncode==1.2.4 gviz-api.py==1.8.2 kombu==3.0.30 Mako==0.4.1 MarkupSafe==0.23 MySQL-python==1.2.5 newrelic==2.68.0.50 nose==1.0.0 Paste==1.7.5.1 PasteDeploy==1.5.0 PasteScript==1.7.3 pip==9.0.1 Pygments==1.4 Pylons==1.0 python-memcached==1.57 pytz==2015.6 Routes==1.12.3 setuptools==21.0.0 simplejson==3.3.1 six==1.10.0 SQLAlchemy==0.8.3 Tempita==0.5.1 WebError==0.10.3 WebHelpers==1.3 WebOb==1.0.8 WebTest==1.2.3 wheel==0.29.0 There are backup copies of the virtualenvs from before the changes at relengadm:/root/buildapi_backups/prod-virtualenv.backup/ relengadm:/root/buildapi_backups/staging-virtualenv.backup/
Assignee: nobody → nthomas
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.