1461303 - Beetmoverworkers updates for scriptworker 11.0.0

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Scriptworker 11.0.0 is shipped! This means we now support [most likely require] aiohttp>=3, and we've dropped support for py35.

Specifically, we need to update beetmoverworkers to:
- require py36; drop py35 support
- test against py36
- support aiohttp3; there are open PRs against the ones that need fixing

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

https://github.com/mozilla-releng/beetmoverscript/issues/138

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Similar approaches in other *scripts were done under shipitscript[1] and other scripts.

[1]: https://github.com/mozilla-releng/shipitscript/pull/6

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Pushapkscript was done here https://github.com/mozilla-releng/pushapkscript/pull/39

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

If things go well and fast, we could do the same for https://github.com/mozilla-releng/bouncerscript and https://github.com/mozilla-releng/balrogscript

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Lisa agreed to work on this so reassigning this to her!

Comment 6

[Lisa Guo [:lguo] EST]

Attachment: Dropped py35 support, added py37 tests

Comment 7

[Lisa Guo [:lguo] EST]

Attachment: did the same for bouncerscript

Comment 8

[Lisa Guo [:lguo] EST]

Attachment: update beetmoverworker puppet requirements file

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 8976981 [details] [diff] [review]
update beetmoverworker puppet requirements file

Looks good! We probably want a PR in the beetmoverscript repo to bump requirements-prod.txt as well.

I can land this on Monday.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/build/puppet/rev/ed1affbb2408
update beetmover deps. r=aki

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Hm, I see https://hg.mozilla.org/build/puppet/rev/f78fdc9188b1 landed in production. Do we need to land anything in-tree or the only leftover is the requirements-prod.txt?

Comment 12

[Lisa Guo [:lguo] EST]

Attachment: ammendment to last patch: diff for beetmoverscript and bouncerscript

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/build/puppet/rev/9a536661a1d7
bump bouncer scriptworker deps. r=aki

Comment 14

[Aki Sasaki (not active)]

Attachment: balrog_puppet.diff

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8979695 [details] [diff] [review]
balrog_puppet.diff

Review of attachment 8979695 [details] [diff] [review]:
-----------------------------------------------------------------

Two quick questions: 
1. Out of curiosity, how do we ensure puppet doesn't break for any possible missing package? Or have we already checked all these packages are present within our internal pypi when we rolled-out scriptworker which most likely introduced most of these?
2. Would it be a good time now to roll-out beetmoverscript bump too since we're here? In order to pick-up lisa's work from last 2 weeks - https://github.com/mozilla-releng/beetmoverscript/commits/master. hm, actually not blocking though, we can roll that out anytime.

::: modules/balrog_scriptworker/files/requirements-27.txt
@@ +24,5 @@
> +pycparser==2.18
> +python-dateutil==2.6.1
> +requests==2.18.4
> +toml==0.9.4
> +towncrier==18.5.0

Not sure why this is needed for production but it certainly can't hurt.

Comment 16

[Aki Sasaki (not active)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #15)
> Two quick questions: 
> 1. Out of curiosity, how do we ensure puppet doesn't break for any possible
> missing package? Or have we already checked all these packages are present
> within our internal pypi when we rolled-out scriptworker which most likely
> introduced most of these?

I generally

- create a file with all the new deps listed, e.g. `new`
mkdir download
cd download
pip download --no-deps -r ../new  # make sure to use the right pip,
                                  # needs to be py36 or py27 depending on which req file we're working against
# make sure i didn't download any mac-only wheels; if I did, I manually download the manylinux wheel instead
cd ..
scp -r download releng-puppet2.srv.releng.scl3.mozilla.com:
ssh releng-puppet2.srv.releng.scl3.mozilla.com
cd /data/python/packages-3.x
sudo mv -n ~/download/* .  # use -n to not overwrite existing
ls -lrt  # doublecheck
cd
rm -rf download  # cleanup for next time

> 2. Would it be a good time now to roll-out beetmoverscript bump too since
> we're here? In order to pick-up lisa's work from last 2 weeks -
> https://github.com/mozilla-releng/beetmoverscript/commits/master. hm,
> actually not blocking though, we can roll that out anytime.

Beetmoverscript and bouncerscript bumps are already rolled out.

> ::: modules/balrog_scriptworker/files/requirements-27.txt
> @@ +24,5 @@
> > +pycparser==2.18
> > +python-dateutil==2.6.1
> > +requests==2.18.4
> > +toml==0.9.4
> > +towncrier==18.5.0
> 
> Not sure why this is needed for production but it certainly can't hurt.

It's in requirements-dev.txt, so it shows up here. https://github.com/mozilla-releng/balrogscript/blob/master/requirements-dev.txt

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/build/puppet/rev/30ce934c839a
update balrog scriptworker deps. r=mtabara

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

@aki: thanks for providing all the useful detail from comment 16, that makes sense.
I think we can close this, I see the rollout is now complete - https://hg.mozilla.org/build/puppet/file/tip/modules/beetmover_scriptworker/files/requirements.txt

Feel free to reopen if I missed anything.
Thanks!