Closed Bug 1466627 Opened Last year Closed Last year

roll-out to production Ship-it v2 -v1 sync-up

Categories

(Release Engineering :: Applications: Shipit (backend), defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: mtabara)

References

Details

Attachments

(3 files, 4 obsolete files)

In bug 1461668, we tested in the staging environment how Ship-it v1 and Ship-it v2 can sync-up in terms of releases started. 

Since rolling this out to production depends on Ship-it v2 rollout, am tracking the landing of corresponding patches in this bug. Will lave everything here to have at hand when the moment comes.

As far as I remember from Rail, we want to have this ready for mozilla-beta only in Q3.
Assignee: nobody → mtabara
Status: NEW → ASSIGNED
Comment on attachment 8983133 [details] [review]
[shipitscript] Enrich shipitscript with new behavior to talk to Ship-it v1

@JohanLorenzo 

This is not urgent. We won't be shipping this for another few weeks. But I thought I'd add this to your plate since you're the original developer of this scriptworker and I tried to follow your best practices from this repo.

tl;dr - whenever we'll switch to Ship-it v2, we'll be limited to `mozilla-beta` for security reasons in the first place. Hence, we want that anything we schedule in the Shipit v2 UI, to be "synced" in the old Ship-it v1 database. For that reason, we're extending and enriching behavior in shipitscript to handle that part.

Ship-it v2 => AC (with release information as input) => schedules `release-mark-as-started` task in-tree => shipitscript (talking to Ship-it v1). This way we ensure fluenc in netflows communication as well without touching firewalls rules. 

Tehnically, this job was previously performed by release-runner3 which "started the release" after querying Ship-it v1 and then "marked the release as started" as soon as the graph was scheduled. With Ship-it v2 and this extended behavior, we're basically killing release-runner3 as it's no longer needed. 

Am happy to provide more insight if needed.
Attachment #8983133 - Flags: review?(jlorenzo)
Attachment #8983141 - Attachment is obsolete: true
Attachment #8983219 - Attachment is obsolete: true
Future leftovers when we're ready to switch to production:
* we may want to fix branch for project branches as it's now `maple` instead of `projects/maple`. The config comes directly from parameters.yml so for production branches things will work as expected. I strongly suspect "projects/maple" in Ship-it v1 is prepended anyway in the UI. 
* we may want to fix `Submtted by` field as it's now `shipit-scriptworker-stage`. We can feed that variable by having Ship-it v2 pass it along in the AC and then to `release-mark-as-started` task
* run a general end-to-end config diff to make sure I'm not breaking mark-as-shipped task in other projects/graphs and the existing one is looking good (mark-as-started)
* uplift in-tree changes to central/beta
* land puppet changes to production and unpin the dev worker
* merge shipitscript and bump a production version
* amend the production scopes as well, similarly to what was done for staging[1] but for production this time

[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1461668#c4
Comment on attachment 8983133 [details] [review]
[shipitscript] Enrich shipitscript with new behavior to talk to Ship-it v1

https://github.com/mozilla-releng/shipitscript/pull/8#pullrequestreview-129864299
Attachment #8983133 - Flags: review?(jlorenzo) → review+
Okay, latest status update.

1. The shipitscript PR is ready to be landed (handling both the mark as started behavior and mark-as-shipped)
2. The puppet patch needs to land in puppet
3. The scopes in TC have to be added in production client
4. We need to split the in-tree patch to only land the mark-as-shipped part. 
5. We'll add later on the mark-as-started
All three patches that landed on maple that were successful in testing both behaviors are:
https://hg.mozilla.org/projects/maple/json-rev/1b38998d6310
https://hg.mozilla.org/projects/maple/json-rev/48d85c1f695f
https://hg.mozilla.org/projects/maple/json-rev/e9b94eca6ce5

It's also in this bug as https://bugzilla.mozilla.org/attachment.cgi?id=8983229

For mark-as-started future development when we need to roll this out.
a) branch is now fixed because its taken from head_repo
b) we may want to fix `Submtted by` field as it's now `shipit-scriptworker-stage`. We can feed that variable by having Ship-it v2 pass it along in the AC and then to `release-mark-as-started` task
Comment on attachment 8988550 [details]
[in-tree] tweak mark-as-shipped to new structure.  a=release

https://reviewboard.mozilla.org/r/253798/#review260468
Attachment #8988550 - Flags: review?(rail) → review+
Attachment #8988550 - Attachment description: Bug 1466627 - tweak mark-as-shipped to new structure. a=release → [in-tree] tweak mark-as-shipped to new structure. a=release
Attachment #8983229 - Attachment description: [in-tree] add release-mark-as-started kind → [in-tree] add release-mark-as-started kind and tweak mark-as-shipped
Attachment #8983229 - Attachment is obsolete: true
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cd6a2bf68078
tweak mark-as-shipped to new structure. r=rail a=release
Comment on attachment 8983134 [details] [diff] [review]
[puppet] Adjust puppet to cope with Ship-it v1-v2 sync tasks

https://github.com/mozilla-releng/build-puppet/pull/91
Attachment #8983134 - Attachment is patch: true
Attachment #8983134 - Flags: review?(rail)
Comment on attachment 8983134 [details] [diff] [review]
[puppet] Adjust puppet to cope with Ship-it v1-v2 sync tasks

https://github.com/mozilla-releng/build-puppet/pull/91
Keywords: leave-open
I added the following prod scope:
project:releng:ship-it:server:production

as the other three:
project:releng:ship-it:server:staging
project:releng:ship-it:action:mark-as-started
project:releng:ship-it:action:mark-as-shipped

were already part of moz-tree:level:3:gecko role[1]. 
Not ideal but we can revisit later. More of this under here[2]

[1]: https://tools.taskcluster.net/auth/scopes/project%3Areleng%3Atreescript%3Aaction%3A*/role:moz-tree%3Alevel%3A3%3Agecko
[2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1461668#c4
Comment on attachment 8983134 [details] [diff] [review]
[puppet] Adjust puppet to cope with Ship-it v1-v2 sync tasks

r+ by rail in PR. Landed 
https://github.com/mozilla-releng/build-puppet/commit/f99da301119cffb6785a2ebfa5e9118099fbb3da
Attachment #8983134 - Flags: review?(rail) → review+
Attachment #8983134 - Flags: checked-in+
Okay, latest status update for when we'll roll-out the mark-as-started task to production:

All three patches that landed on maple that were successful in testing both behaviors are:
https://hg.mozilla.org/projects/maple/json-rev/1b38998d6310
https://hg.mozilla.org/projects/maple/json-rev/48d85c1f695f
https://hg.mozilla.org/projects/maple/json-rev/e9b94eca6ce5

On beta the mark-as-shipped tweaks looks like this - https://hg.mozilla.org/releases/mozilla-beta/rev/5ba0247433fe

What we need to ship is maple-patches sans the beta patch ^.

For mark-as-started future development when we need to roll this out.
a) we may want to fix `Submtted by` field as it's now `shipit-scriptworker-stage`. We can feed that variable by having Ship-it v2 pass it along in the AC and then to `release-mark-as-started` task
b) anything else that doesn't look right in Ship-it v1?
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/3ca944d19017
Port bug 1466627 [tweak mark-as-shipped to new structure]. rs=bustage-fix
Want me to file a separate bug for remaining of this bug (mark-as-started logic) to be easier in the future or should I leave things be here?
Flags: needinfo?(rail)
See Also: → 1472667
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #24)
> Want me to file a separate bug for remaining of this bug (mark-as-started
> logic) to be easier in the future or should I leave things be here?

Sure. Would be easier to track it I guess.
Flags: needinfo?(rail)
Blocks: 1475101
Work here is done. Leftover (in-tree mark-as-started task to add) is tracked in bug 1475101.
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.