[in-tree relpro] "mark as shipped" task shouldn't run on Buildbot anymore

RESOLVED FIXED

Status

defect
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jlorenzo, Assigned: jlorenzo)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox59 fixed, firefox60 fixed)

Details

Attachments

(10 attachments)

50 bytes, text/x-github-pull-request
mtabara
: review+
jlorenzo
: checked-in+
Details | Review
93 bytes, text/plain
mtabara
: review+
aki
: review+
Details
60 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
59 bytes, text/x-review-board-request
aki
: review+
jlorenzo
: checked-in+
Details
51 bytes, patch
mtabara
: review+
jlorenzo
: checked-in+
Details | Diff | Splinter Review
55 bytes, text/x-github-pull-request
aki
: review+
jlorenzo
: checked-in+
Details | Review
51 bytes, text/x-github-pull-request
aki
: review+
jlorenzo
: checked-in+
Details | Review
59 bytes, text/x-review-board-request
aki
: review+
jlorenzo
: checked-in+
Details
54 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
51 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
Comment on attachment 8943987 [details] [review]
[shipitapi] Make shipitapi py3-compliant

Lgtm, thanks for this!
Attachment #8943987 - Flags: review?(mtabara) → review+
I created a new repo that I intend to move to [1]. It basically reimplements [2], but stores secrets on a file known the machine. It keeps calling shipitapi. There is no extra-feature, except that I enforce some scopes.

Without enforcement, testing branches would be able to mark releases as ship via the prod shipit_scriptworker.

Apart from that, shipitscript is covered by unit and integration test. Next step is to update taskgraph.

What do you guys think?

[1] https://github.com/mozilla-releng/shipitscript (404 at time of writing)
[2] https://hg.mozilla.org/mozilla-central/file/3d23e6d98a09/testing/mozharness/scripts/release/postrelease_mark_as_shipped.py#l86
Attachment #8944469 - Flags: review?(mtabara)
Attachment #8944469 - Flags: review?(aki)
Comment on attachment 8944469 [details]
[shipitscript] First implementation (7 first commits)

Looks good!

- we may want to have multiple `ship_it_instance` settings. ship-it v1 prod, ship-it v2 prod, and ship-it dev are all potential things we may want to support with the same scriptworker, depending on scopes and branches.
- we probably want to add the appropriate restricted scopes to scriptworker when we're ready to roll this out, so we can only use ship-it prod on release branches.
- ALLOWED_API_ROOT_PER_VALID_SCOPE is a hardcoded check. I'm debating whether we want this hardcoded in the code or in the puppet-built config.
Attachment #8944469 - Flags: review?(aki) → review+
Comment on attachment 8944469 [details]
[shipitscript] First implementation (7 first commits)

++ to what :aki said.

Additional to that, although not urgent, we might want to revisit the scopes structure.

So far we have:
"project:releng:beetmover:bucket:..."
"project:releng:beetmover:action:...."
"project:releng:balrog:channel:..."
"project:releng:googleplay:...."

where
"project:releng:scriptworker:ship-it:production" seems somehow out of sync. We might consider something
"project:releng:ship-it:..." or "project:releng:shipitapi:..." although I'd be in favor of former as we might add further behaviors to this to scriptworker, depending on what future Ship-it v2 will look like.
Attachment #8944469 - Flags: review?(mtabara) → review+
(In reply to Aki Sasaki [:aki] from comment #4)
> - we may want to have multiple `ship_it_instance` settings [...]
I'd be in favor of splitting ship-it-dev from ship-it prod (v1 or v2). This way, we ensure a staging release doesn't update the prod instance.

> - we probably want to add the appropriate restricted scopes to scriptworker [...]
Done in https://github.com/mozilla-releng/scriptworker/pull/181

> - ALLOWED_API_ROOT_PER_VALID_SCOPE is a hardcoded check [...]
I'm not sure either. Let's keep it hardcoded for now, and see if we need to configure it later. I guess "later" will be when Thunderbird moves their ship-it task.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #7)
> "project:releng:scriptworker:ship-it:production" seems somehow out of sync.
Good point. Done in https://github.com/JohanLorenzo/shipitscript/pull/1
Attachment #8945442 - Attachment is patch: true
Attachment #8945442 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #8945442 - Flags: review?(mtabara) → review+
(In reply to Johan Lorenzo [:jlorenzo] from comment #8)
> (In reply to Aki Sasaki [:aki] from comment #4)
> > - we may want to have multiple `ship_it_instance` settings [...]
> I'd be in favor of splitting ship-it-dev from ship-it prod (v1 or v2). This
> way, we ensure a staging release doesn't update the prod instance.

We can do that, but we already have release+nightly+dep signing enabled on the prod signing scriptworkers. Dep is now moved to a separate pool, but I think having a pool of dep ship-it scriptworkers may be a little wasteful given how little work I believe they will do. Right now the scriptworker scope checks + any sanity checks we run are what we use to keep them separate.

Another option is to run multiple ship-it scriptworker daemons on the same EC2 instance... we haven't done that before, but it's possible.
Attachment #8945443 - Flags: review?(aki) → review+
Comment on attachment 8945128 [details] [review]
[cloud-tools] PR: Add shipitworker instance type

Was r+'d by Nick on the PR.
Attachment #8945128 - Flags: review?(nthomas) → review+
(In reply to Aki Sasaki [:aki] from comment #11):
I agree. I open this new PR to support several roots on the same scriptworker instance.
Attachment #8946236 - Flags: review?(aki)
Comment on attachment 8946236 [details] [review]
[shipitscript] PR: Allow shipit_scriptworker to talk to several shipit roots

Noted that setting the instances by config file, via puppet, may be more flexible in the future; scriptworker instances can have the set of allowed instances defined in the config file. Non-allowed ship-it instances won't be in the config file, and therefore will be unreachable by default. Not blocking on this.
Attachment #8946236 - Flags: review?(aki) → review+
Comment on attachment 8946695 [details]
[gecko] Change release_mark_as_shipped worker-type to shipit_scriptworker

https://reviewboard.mozilla.org/r/216664/#review222458

Ship it!
Attachment #8946695 - Flags: review?(aki) → review+
Comment on attachment 8945141 [details]
[puppet] Add shipit_scriptworkers

https://reviewboard.mozilla.org/r/215384/#review222462

::: modules/shipit_scriptworker/manifests/init.pp:44
(Diff revision 6)
> +                'python-gnupg==0.4.1',
> +                'redo==1.6',
> +                'requests==2.18.4',
> +                'scriptworker==8.0.1',
> +                'shipitapi==0.1.0',
> +                'shipitscript==0.1.0',

It makes sense to call this >=1.0.0 as soon as we're using it in production. If you've already versioned and released, we can wait til the next version bump.
Attachment #8945141 - Flags: review?(aki) → review+
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/abc4fb535fef
Change release_mark_as_shipped worker-type to shipit_scriptworker r=aki
https://hg.mozilla.org/mozilla-central/rev/abc4fb535fef
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Comment on attachment 8946695 [details]
[gecko] Change release_mark_as_shipped worker-type to shipit_scriptworker

Landed on:
* central: see comment 34
* beta: https://hg.mozilla.org/releases/mozilla-beta/rev/f7c7cab7877b85a71e3c684e13b489c264026ffd

In case something goes wrong, I left instructions to releaseduty folks at:
* https://github.com/mozilla-releng/releasewarrior-data/commit/8613e9270cd5ee3ba7ce737b0129fe403dbbb89a
* https://github.com/mozilla-releng/releasewarrior-data/commit/31bc4ecb3ff479d969024b368265521813b3ae0c
Attachment #8946695 - Attachment description: Bug 1431764 - Change release_mark_as_shipped worker-type to shipit_scriptworker → [gecko] Change release_mark_as_shipped worker-type to shipit_scriptworker
Attachment #8946695 - Flags: checked-in+
Attachment #8945141 - Attachment description: Bug 1431764 - Add shipit_scriptworkers → [puppet] Add shipit_scriptworkers
You need to log in before you can comment on or make changes to this bug.