1412342 - Golden instances not get puppetized - nose==1.3.1 , 1.3.4 could not be found

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

The golden AMI were no created today because we hit this error during the puppetize process.Seems lijke runner 2.0 have dependency for nose 1.3.1 and cleanslate have dependency for nose 1.3.4.

error: check 1.png and 2.png

I didn't found any puppet push that could have generate this error,also I remove some packages that I uploaded to Pypi in Bug 1412148 and re-run the script to generate the golden AMI but it seems like this isn't the cause.

Something has changed since yesterday and these dependency packages can't be found on the local or for download and Puppet is failing.

Comment 1

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: 1.png

Comment 2

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: 2.png

Comment 3

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: 3.png

Comment 4

[Jordan Lund (:jlund)]

I added 1.3.1 and 1.3.4 to our internal python package repo:

https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/nose-1.3.1.tar.gz
https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/nose-1.3.4.tar.gz

I suspect we were somehow relying on external pypi for these before even though we are seemingly passing `--no-deps --no-index --find-links https://releng-puppet1.srv.releng.use1.mozilla.com/python/packages` in the install call.

At any rate, 

This is too coincidental with the timing of https requirement in pypi. So if we make it farther in puppetizing now that the right version of nose is interanlly available and nobody seemingly removed it in the last 24h, I would suspect that http, old version of pip/virtualenv are to blame. In which case, we need https://bugzilla.mozilla.org/show_bug.cgi?id=1390946 to resolve.

Comment 5

[Jordan Lund (:jlund)]

dhouse: looks like we are still failing to install nose. Perhaps we are ignoring internal puppet repo completely. bug 1390946 is high priority now on our end since breaks future ami generations in our infra

Comment 6

[Jordan Lund (:jlund)]

Attachment: puppet_cleanslate_add_nose_dep.patch

Turns out we *are* installing cleanslate internally. This is coming from a package Morgan made for runner and uploaded to puppet python repo.

My assumption is that packages defined as deps within the cleanslate package ignore internal repo and instead look in global pypi https://github.com/FaustDeGoethe/build-cleanslate/blob/master/setup.py#L18

Which no longer works with our version of pip and http

solution: install nose 1.3.4 from internal via puppet first before installing cleanslate.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 8922940 [details] [diff] [review]
puppet_cleanslate_add_nose_dep.patch

We could also unpin nose in cleanslate, but this is probably faster for now.

Comment 8

[Jordan Lund (:jlund)]

https://hg.mozilla.org/build/puppet/rev/0294de6bfe62b3a5d90a695c6a3f055e623ad37b
Bug 1412342 - Golden instances not get puppetized nose==1.3.1 could not be found, r=aki

Comment 9

[Jordan Lund (:jlund)]

Attachment: puppet_cleanslate_and_runner_remove_nose_dep.patch

last attempt didn't work. As per #mozbuild, this removes nose dep entirely as I can't see it being used outside of CI

new versions uploaded:
https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/cleanslate-1.3.tar.gz
https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/runner-2.1.tar.gz

based on my own repo:
https://github.com/lundjordan/build-cleanslate/commit/802c9aa20c916791dc6863a6bb07a28f13e14746
https://github.com/lundjordan/build-runner/commit/5e873ccd87fad7aa2b9346f10904b0c8235d8a46

Comment 10

[Jordan Lund (:jlund)]

https://hg.mozilla.org/build/puppet/rev/17d5d931516b9b84033c0fad12c149c5209e4e93
Bug 1412342 - remove nose req from runner and cleanslate, r=aki

Comment 11

[Jordan Lund (:jlund)]

https://hg.mozilla.org/build/puppet/rev/6acefdc9a51dc365025dd902fe2c84b10f96753d
Bug 1412342 - remove nose req from runner and cleanslate, r=aki

Comment 12

[Jordan Lund (:jlund)]

I believe the stopgap has worked:

Oct 27 16:09:38 av-linux64-ec2-golden.build.releng.use1.mozilla.com puppet-agent: (/Stage[main]/Cleanslate/Python::Virtualenv[/opt/cleanslate]/Python::Virtualenv::Package[/opt/cleanslate||cleanslate==1.3]/Exec[pip /opt/cleanslate||cleanslate==1.3]/returns) executed successfully 
16:09:54 Oct 27 16:09:29 av-linux64-ec2-golden.build.releng.use1.mozilla.com puppet-agent: (/Stage[main]/Runner/Python::Virtualenv[/opt/runner]/Python::Virtualenv::Package[/opt/runner||runner==2.1]/Exec[pip /opt/runner||runner==2.1]/returns) executed successfully

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

We're still getting nagios alerts like
<nagios-releng> Sun 19:26:11 UTC [7439] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 2 crit, 0 warn out of 2 processes with args 'ec2-golden'

but that's due to these processes owned by aobreja on aws-manager2:

6536 SCREEN -S tst-linux64-ec2-golden-aobreja
6666 SCREEN -S av-linux64-ec2-golden-aobreja

I've downtimed that for 12 hours to give Andrei a chance to clean up.

Comment 14

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Seems like my screens are not available any more so clean up is no longer needed.
Thank you Jordan and Nick for watching over this bug.