1533337 - migrate scriptworkers to cloudops

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Rok Garbas [:garbas]]

Existing scriptworkers:

• https://github.com/mozilla-releng/addonscript
• https://github.com/mozilla-releng/treescript
• https://github.com/mozilla-releng/pushsnapscript
• https://github.com/mozilla-releng/mozapkpublisher
• https://github.com/mozilla-releng/bouncerscript
• https://github.com/mozilla-releng/signingscript
• https://github.com/mozilla-releng/pushapkscript
• https://github.com/mozilla-releng/beetmoverscript
• https://github.com/mozilla-releng/shipitscript (already migrated)
• https://github.com/mozilla-releng/balrogscript

Comment 1

[Aki Sasaki (not active)]

mozapkpublisher isn't a scriptworker; it's part of pushapk scriptworker. We can ignore binary transparency as well (not on the list in comment 0, so no change needed for that).

pushsnap may be a strong candidate for one of the next to migrate. It's currently running an ubuntu tool on centos, and has some hacks to make that work. Once we're no longer using puppet and can choose our own OS in docker, we should be able to make this much nicer.

Comment 2

[Rok Garbas [:garbas]]

I added the initial plan for this under User Story and I will try to keep it up to date as I proceed. My current plan is to deploy all of the scriptworkers to cloudops in H1, but i'm not sure how fast we can replace old scriptworkers with new ones, so i'm hesitant to promise complete migration.

:aki I assume binary transparency is not planned to be migrated since it is not currently in use, correct?

:aki also can you provide an ordered list (by your opinion) how we should tackle the scriptworkers. you are much more familiar of how complex they are in terms of dependency on tree.

Comment 3

[Chris AtLee [:catlee]]

I would like to be able to migrate signingscript and beetmover script earlier, since both of them see pretty high (but bursty) load, and I want to be able to scale them up and down better.

Comment 4

[Aki Sasaki (not active)]

(In reply to Rok Garbas [:garbas] from comment #2)

:aki I assume binary transparency is not planned to be migrated since it is not currently in use, correct?

+1

:aki also can you provide an ordered list (by your opinion) how we should tackle the scriptworkers. you are much more familiar of how complex they are in terms of dependency on tree.

script	urgency	status	notes
pushsnap	high	not done	coordinate with jlorenzo. this will be ubuntu and will allow us to get rid of hacks
signing	med	not done	need netflows to signing servers, fw updates, ip allowlist updates
beetmover	med	not done	this will use a lot of network round-tripping to s3; watch this for bandwidth and $$
addon		not done	this needs some scaling, and access to amo.
pushapk		not done	talks to gplay. we can probably get away with a single prod instance
tree		not done	this needs write/flows to hg.m.o, could possibly go over internet
bouncer		not done	talks to bouncer - needs netflows and fw? updates
balrog		not done	py2 script. we may want to upgrade to py3 before we migrate? We need netflows to the balrog admin api

Pushsnap scriptworker would be my candidate for highest priority, since aiui the current setup is one big hack to get an ubuntu tool to run on centos. Docker (moving away from puppet) will allow us to run on ubuntu or debian.

I think the signing and beetmover scriptworkers may be on the trickier side because they're in heavy use. They will create a lot of network traffic between GCP and mdc{1,2} and/or s3, and the signing scriptworkers will require netflows and an allowlisted IP range. But agreed, we have scaling issues there, and moving them will be a big win.

Balrog scriptworker is on py2. We could move it as py2 with a py3 scriptworker (two separate virtualenvs), or we could upgrade it to py3 before GCP.

The other four are relatively straightforward, and can be done at any point.