Open Bug 1533337 Opened 1 year ago Updated 2 months ago

migrate scriptworkers to cloudops

Categories

(Release Engineering :: General, enhancement)

enhancement
Not set

Tracking

(Not tracked)

People

(Reporter: garbas, Unassigned)

References

(Depends on 5 open bugs, Blocks 1 open bug)

Details

User Story

scriptworkers:

1.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/shipitscript (deployed for 70 train, Bug 1581149)
2.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/pushsnapscript (deployed for 71.0b3, Bug 1588577)
3.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/addonscript (deployed for 71.0b1, Bug 1585603)
4.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/treescript (deployed for 71.0b3, Bug 1588392)
5.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/bouncerscript (deployed for 70 train, Bug 1580476)
6.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/signingscript (deployed for 71.0b1, Bug 1542819)
7.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/pushapkscript (WIP, Bug 1588590)
8.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript (deployed for 70 train, Bug 1548336)
9.) https://github.com/mozilla-releng/scriptworker-scripts/tree/master/balrogscript (deployed for 71.0b1, Bug 1580478)

mozapkpublisher isn't a scriptworker; it's part of pushapk scriptworker. We can ignore binary transparency as well (not on the list in comment 0, so no change needed for that).

pushsnap may be a strong candidate for one of the next to migrate. It's currently running an ubuntu tool on centos, and has some hacks to make that work. Once we're no longer using puppet and can choose our own OS in docker, we should be able to make this much nicer.

User Story: (updated)
Depends on: 1520588

I added the initial plan for this under User Story and I will try to keep it up to date as I proceed. My current plan is to deploy all of the scriptworkers to cloudops in H1, but i'm not sure how fast we can replace old scriptworkers with new ones, so i'm hesitant to promise complete migration.

:aki I assume binary transparency is not planned to be migrated since it is not currently in use, correct?

:aki also can you provide an ordered list (by your opinion) how we should tackle the scriptworkers. you are much more familiar of how complex they are in terms of dependency on tree.

Flags: needinfo?(aki)

I would like to be able to migrate signingscript and beetmover script earlier, since both of them see pretty high (but bursty) load, and I want to be able to scale them up and down better.

(In reply to Rok Garbas [:garbas] from comment #2)

:aki I assume binary transparency is not planned to be migrated since it is not currently in use, correct?

+1

:aki also can you provide an ordered list (by your opinion) how we should tackle the scriptworkers. you are much more familiar of how complex they are in terms of dependency on tree.

script urgency status notes
pushsnap high not done coordinate with jlorenzo. this will be ubuntu and will allow us to get rid of hacks
signing med not done need netflows to signing servers, fw updates, ip allowlist updates
beetmover med not done this will use a lot of network round-tripping to s3; watch this for bandwidth and $$
addon not done this needs some scaling, and access to amo.
pushapk not done talks to gplay. we can probably get away with a single prod instance
tree not done this needs write/flows to hg.m.o, could possibly go over internet
bouncer not done talks to bouncer - needs netflows and fw? updates
balrog not done py2 script. we may want to upgrade to py3 before we migrate? We need netflows to the balrog admin api

Pushsnap scriptworker would be my candidate for highest priority, since aiui the current setup is one big hack to get an ubuntu tool to run on centos. Docker (moving away from puppet) will allow us to run on ubuntu or debian.

I think the signing and beetmover scriptworkers may be on the trickier side because they're in heavy use. They will create a lot of network traffic between GCP and mdc{1,2} and/or s3, and the signing scriptworkers will require netflows and an allowlisted IP range. But agreed, we have scaling issues there, and moving them will be a big win.

Balrog scriptworker is on py2. We could move it as py2 with a py3 scriptworker (two separate virtualenvs), or we could upgrade it to py3 before GCP.

The other four are relatively straightforward, and can be done at any point.

Flags: needinfo?(aki)
Depends on: 1542819
Depends on: 1548336
Depends on: 1566740
User Story: (updated)
No longer depends on: 1566740
Depends on: 1566740
Depends on: 1574925
See Also: → 1581151
User Story: (updated)
User Story: (updated)
Depends on: 1585603
No longer depends on: 1580481
User Story: (updated)
User Story: (updated)
Depends on: 1587952
Depends on: 1588392
User Story: (updated)
User Story: (updated)
Depends on: 1581151
See Also: 1581151
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.