1467002 - releaserunner virtualenv for FF52 & TB52 busted

Status: RESOLVED WORKSFORME

(Release Engineering :: Release Automation, enhancement)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Both prod-old (TB52) and prod (FF52) have
  ImportError: No module named urllib3

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: PR

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Also missing chardet and idna. Aki is doing some further checking.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: PR part duex

Aki's PR to revert us to older packages, where requests bundles urllib3, chardet, etc.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #3)
> Created attachment 8983645 [details] [review]
> PR part duex

We deployed this but hit problems due to the releasetasks==0.3.3 line in the requirements. We actually had been using /home/cltbld/releasetasks on tip of https://github.com/mozilla-releng/releasetasks. Aki did 'python setup.py develop --no-deps', and made sure the 0.3.3 was uninstalled (the develop call didn't do this). He also needed to install pgpy. FF 52.8.1esr could then start.

The final pip freeze output is:
arrow==0.5.4
buildbot===0.8.7p1
buildtools==1.0.6
certifi==0.0.8
cffi==1.10.0
chunkify==1.2
cryptography==0.6
decorator==3.4.0
docopt==0.6.2
ecdsa==0.10
enum34==1.0.4
Fabric==1.5.1
furl==1.1
futures==3.1.1
Jinja2==2.6
mohawk==0.3.1
orderedmultidict==0.7.11
paramiko==1.9.0
PGPy==0.3.0
pycparser==2.17
pycrypto==2.6.1
PyHawk-with-a-single-extra-commit==0.1.5
python-dateutil==1.5
python-jose==0.5.2
PyYAML==3.10
redo==1.5
-e git+https://github.com/mozilla-releng/releasetasks.git@953dddcba9e8fe917664b6567b488d21fbfb78b2#egg=releasetasks
requests==2.6.0
requests-hawk==1.0.0
simplejson==2.6.2
singledispatch==3.4.0.3
six==1.9.0
slugid==1.0.7
SQLAlchemy==0.8.0b2
sqlalchemy-migrate==0.7.2
taskcluster==0.0.24
Tempita==0.5.1
toposort==1.5
treeherder-client==4.0.0
Twisted==12.3.0
urllib3==1.22
zope.interface==4.0.2

puppet is not undoing this, thankfully. 

bm81 (old-prod) and bm83 (stage) are polling happily but haven't had the extra work with releasetasks and pgpy.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #4)
> treeherder-client==4.0.0

Removed this again. We added it while trying to make old releasetasks work, but not needed for tip.

> puppet is not undoing this, thankfully. 

Spoke too soon, it's undone the setup develop:

puppet-agent: (/Stage[main]/Releaserunner/Python::Virtualenv[/builds/releaserunner]/Python::Virtualenv::Package[/builds/releaserunner||releasetasks==0.3.3]/Exec[pip /builds/releaserunner||releasetasks==0.3.3]/returns) executed successfully

So we need to fix this. Could put a 0.4.0 release in our pypi - not likely to change it much now.

> bm81 (old-prod) and bm83 (stage) are polling happily but haven't had the
> extra work with releasetasks and pgpy.

Well, bm81 doesn't need it because it's buildbot reconfig & sendchange. Until we do a Thunderbird 52.8.x we'll not be quite sure if all is well there.

bm83 needs fixing still, but see puppet above.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I think we also need to provide build/tools in a develop-style. Can we do that in puppet Ben ?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Current status - can't start new ESR52 releases.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #6)
> I think we also need to provide build/tools in a develop-style. Can we do
> that in puppet Ben ?

We typically do this with an exec instead of defining it in the requirements file (like https://github.com/mozilla/build-puppet/blob/f5a07cb7e4338ebc187b8bcdc709a02c098e4b90/modules/buildmaster/manifests/queue.pp#L100).

Comment 9

[[e-mail] (:bhearsum)]

Thanks for all the investigative work + release unblocking.

I'm really surprised that release runner wasn't broken prior to this - we rebuilt its virtualenv multiple times through our python upgrade work, which means it _should_ have had releasetasks 0.3.3 installed. I think there's something here that I still don't quite understand.

In any case, I just tested https://github.com/mozilla/build-puppet/compare/master...mozbhearsum:releasetasks-upgrade?expand=1 against dev release runner, and it seemed to work OK. I submitted a release that it would pay attention to to ship-it-dev (Firefox-57.0esr-build1), and it got as far as doing sanity checks.

I also tested that patch with a completely empty /builds/releaserunner (I moved it to /builds/releaserunner.bak), and it appeared to rebuild correctly and work. "buildbot" appears to works in the virtualenv, too.

I'm going to hold off posting or pushing any patches until I understand the failures here a bit better.

Comment 10

[Tom Prince [:tomprince]]

I suspect part of the reason this didn't get noticed sooner is that releaserunner doesn't get restarted automatically. So, when the dependencies got updated, the existing releaserunner process continued running with the dependencies it has already loaded, polling ship-it for new versions. Since we haven't had a release since those upgrades, a new process never tried to start with the new dependencies.

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Tom Prince [:tomprince] from comment #10)
> I suspect part of the reason this didn't get noticed sooner is that
> releaserunner doesn't get restarted automatically. So, when the dependencies
> got updated, the existing releaserunner process continued running with the
> dependencies it has already loaded, polling ship-it for new versions. Since
> we haven't had a release since those upgrades, a new process never tried to
> start with the new dependencies.

I noticed this too while testing my patch - I can probably fix that pretty easily...

Comment 12

[[e-mail] (:bhearsum)]

I'm working on a fix here still. I think it's nearly ready, but I want to do some additional testing in staging before I post it for review. I wanted to get it done today, but that doesn't seem likely at this point. I think it's unlikely we'll have an esr build in the next ~12h, so I'll finish this up tomorrow.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: upgrade as much of the releaserunner deps as possible

PR has more details.

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 8984199 [details] [review]
upgrade as much of the releaserunner deps as possible

From the PR:

I think we also need build/tools - do we need to add something to run python setup.py develop in /builds/releaserunner/tools ?

If this doesn't work, we may want to stop maintaining that virtualenv via puppet, then create that virtualenv manually and hope the machine doesn't blow up before esr52 goes away.

Comment 15

[[e-mail] (:bhearsum)]

https://github.com/mozilla/build-puppet/pull/62 was also required on bm81, which runs different releaserunner code than dev (yay). It's up and running again, but I don't have a way to be 100% certain that it works until a release is triggered =\.

I'm going to leave this bug open for now. If things blow up in a way that isn't easily fixable, I think we should take aki's advice and unmanage this virtualenv, and rebuild it once by hand.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #14)
> Comment on attachment 8984199 [details] [review]
> upgrade as much of the releaserunner deps as possible
> 
> From the PR:
> 
> I think we also need build/tools - do we need to add something to run python
> setup.py develop in /builds/releaserunner/tools ?
> 
> If this doesn't work, we may want to stop maintaining that virtualenv via
> puppet, then create that virtualenv manually and hope the machine doesn't
> blow up before esr52 goes away.

We talked in IRC, but for posterity: I don't think we need to "setup.py develop" the tools repo. Release Runner sets up sys.path to include tools/lib/python before trying to import anything from it.

Comment 17

[Jordan Lund (:jlund)]

I don't believe we hit issue with the second esr52 gtb last week. Closing this for now.