Mobile release automation to move candidates to releases/ (push to mirrors)

RESOLVED FIXED

Status

Release Engineering
Release Automation
P2
normal
RESOLVED FIXED
7 years ago
3 years ago

People

(Reporter: aki, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mobile][automation][postrelease])

Attachments

(11 attachments, 3 obsolete attachments)

1007 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
2.05 KB, patch
Details | Diff | Splinter Review
2.28 KB, patch
Details | Diff | Splinter Review
810 bytes, patch
aki
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
26.63 KB, text/plain
Details
13.69 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
61.20 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
17.75 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
906 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
11.77 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
2.88 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
(Reporter)

Description

7 years ago
The layout is the same for builds/source.
The repos, however, aren't 1.0/repos/1.9.2_LOCALE/dists but 1.0/maemo/LOCALE/dists.

We also need install files to point to moff.m.c.
(Reporter)

Updated

7 years ago
Blocks: 535690, 478420
(Reporter)

Updated

7 years ago
Assignee: aki → nobody
Priority: -- → P4
Out of curiosity, is there any reason why the candidates directories can't be exactly the same as the releases ones?
(Reporter)

Comment 2

7 years ago
I'm going to assume you mean "...any reason why the candidates directories can't have the same layout as the releases ones?", and that you're not suggesting we publish candidate builds directly to the releases directory tree.

Couple reasons.
First, we changed the releases/ dir layout at the zero hour of 1.0 GM, and we've had no releases go live since then.  Stuart did ping me about whether we wanted to clean up some of the layout in http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/1.0/ , so the layout isn't fully set yet.  Thinking we might want to wait a release or 2 to make sure the layout's set for the moment.

Also, the deb repos are currently under version-candidates/repos/ rather than version-candidates/buildN/.  This is to facilitate update testing, since people who install build1 for testing will be able to update automatically to build2 via that repo.  For 1.0 I was asked to move the repos/locale/dists directories to releases/1.0/maemo/locale/dists and to not move the .install file (which is needed for candidate repo testing).
(Reporter)

Comment 3

7 years ago
Docs on how to do this as of today are here:

https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Mobile

As mentioned above, the repos are the main reason why we might need automation here. Maybe we need to push to both locations (VERSION-candidates/repos/LOCALE as well as VERSION-candidates/buildN/maemo/LOCALE ?)
(Reporter)

Updated

7 years ago
Whiteboard: [mobile][automation]
Blocks: 627271
No longer blocks: 478420
(Reporter)

Updated

6 years ago
Assignee: nobody → aki
(Reporter)

Updated

6 years ago
Duplicate of this bug: 682787
(Reporter)

Updated

6 years ago
Whiteboard: [mobile][automation] → [mobile][automation][triagefollowup]
(Reporter)

Comment 5

6 years ago
Triage: Marked as triagefollowup since I won't have time in Q4 for this.

Comment 6

6 years ago
(In reply to Aki Sasaki [:aki] from comment #5)
> Triage: Marked as triagefollowup since I won't have time in Q4 for this.

If it's sat for 2 years, I think we're OK with another month. ;)

But is this still required? It's marked as Maemo, which is why I ask.

How do these files make it into releases/ right now? Is there a script?
Whiteboard: [mobile][automation][triagefollowup] → [mobile][automation]
(Reporter)

Comment 7

6 years ago
ffxbld@stage.m.o:push_fennec_{beta,release}.sh
OS: Maemo → All
(Reporter)

Comment 8

6 years ago
https://wiki.mozilla.org/Releases/BuildNotesTemplate#Push_the_files :)
No longer blocks: 627271
Blocks: 714371
No longer blocks: 714371
Mass move of bugs to Release Automation component.
Blocks: 714371
Component: Release Engineering → Release Engineering: Automation (Release Automation)
No longer blocks: 714371
(Reporter)

Updated

5 years ago
Duplicate of this bug: 764970
Aki, this has been assigned to you for over a year without action. I'm going to put it back in the pool to make it more clear that someone else should feel free to pick it up. Things have changed since this was originally filed, and I think this is relatively trivially to do now. I did a diff of the candidates dir vs. releases dir, here's what we _don't_ want synced from candidates -> releases:
- *_info.txt
- logs/
- unsigned/

All of these things are already listed in https://github.com/mozilla/build-tools/blob/946ea8c7ef59d1f51065839d3a7db9323d493089/scripts/release/stage-tasks.py#L31, which means we don't have to tweak this script much. I think we need to make the following changes to make this happen:
* Make the dry run of push to mirrors happen when 'dry_run_push' is in args (instead of inferring it from a different arg): https://github.com/mozilla/build-tools/blob/946ea8c7ef59d1f51065839d3a7db9323d493089/scripts/release/stage-tasks.py#L234
* Add 'dry_run_push' to the args for the check_permissions factory: https://github.com/mozilla/build-buildbotcustom/blob/2a9327ff1b2682d9c47350c9c44c95febc4d690b/process/release.py#L1227
* Drop disablePushToMirrors from Fennec release templates

The only thing to watch out for here is the scheduling -- some of the push to mirrors stuff may be tied into updates at the moment.
Assignee: aki → nobody
(Reporter)

Comment 12

5 years ago
That's cool, the only reason I didn't unassign myself was the old bug rule.
Whiteboard: [mobile][automation] → [mobile][automation][postrelease]
Need to make sure the exclude list includes all of the stuff from https://bugzilla.mozilla.org/show_bug.cgi?id=885013#c0 when this is done.
Is this a bug that Jason could work on?
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #14)
> Is this a bug that Jason could work on?

I think so. I'd like to review comment #11 before though, to make sure it's still accurate.
Comment #11 is still accurate, but incomplete. We need to decide whether or not we want this to run as part of automation, like Firefox's push to mirrors does, or whether it needs to be fully manual.

Aki, I'm leaning towards having it run after all deliverables are ready. Unless there's a reason not to, it seems like having that consistency is better. Any thoughts?
Flags: needinfo?(aki)
(In reply to Ben Hearsum [:bhearsum] from comment #16)
> Comment #11 is still accurate, but incomplete. We need to decide whether or
> not we want this to run as part of automation, like Firefox's push to
> mirrors does, or whether it needs to be fully manual.
> 
> Aki, I'm leaning towards having it run after all deliverables are ready.
> Unless there's a reason not to, it seems like having that consistency is
> better. Any thoughts?

Of course, we still do it by hand for mozilla-release, so I'm only talking about automatic pushing for mozilla-beta.

If we want to run this after deliverables are ready we also need to:
* Remove verifyConfigs as a condition of pushing to mirrors: https://github.com/mozilla/build-buildbotcustom/blob/06795046569f1c6d4b1f425c0fde3f559abe5f32/process/release.py#L1767
* Append push_to_mirrors to the deliverables_builder list, rather than post_update_builders (this is a relic, from when the updates builder still generated files that get pushed).
* Set 'enableAutomaticPushToMirrors' to True in Fennec beta release configs.
Duplicate of this bug: 885013
(Reporter)

Comment 19

4 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> (In reply to Ben Hearsum [:bhearsum] from comment #16)
> > Comment #11 is still accurate, but incomplete. We need to decide whether or
> > not we want this to run as part of automation, like Firefox's push to
> > mirrors does, or whether it needs to be fully manual.
> > 
> > Aki, I'm leaning towards having it run after all deliverables are ready.
> > Unless there's a reason not to, it seems like having that consistency is
> > better. Any thoughts?
> 
> Of course, we still do it by hand for mozilla-release, so I'm only talking
> about automatic pushing for mozilla-beta.

That sounds fine to me.
Flags: needinfo?(aki)
Assignee: nobody → yshun

Comment 20

4 years ago
Created attachment 776404 [details] [diff] [review]
Add dry run option to optionparser
Attachment #776404 - Flags: review?(bhearsum)

Comment 21

4 years ago
Created attachment 776405 [details] [diff] [review]
Add new arg to scriptfactory
Attachment #776405 - Flags: review?(bhearsum)
Comment on attachment 776404 [details] [diff] [review]
Add dry run option to optionparser

Review of attachment 776404 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/stage-tasks.py
@@ +263,5 @@
>                        productName=productName,
>                        version=version,
>                        buildNumber=buildNumber,
>                        extra_excludes=options.extra_excludes,
> +                      dryRun=options.dry_run_push)

This isn't quite what I had in mind for dry_run_push. The "if 'permissions' in args or 'antivirus' in args" should just get replaced by 'if dry_run_push in args'. And the second pushToMirrors call should remain unaltered.

The idea is that a dryRun of pushToMirrors is its own unique thing now. The way this patch is written, it will never run for Fennec, because Fennec doesn't do a permissions or AV check.
Attachment #776404 - Flags: review?(bhearsum) → review-
Comment on attachment 776405 [details] [diff] [review]
Add new arg to scriptfactory

Review of attachment 776405 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1268,5 @@
>                          '--extra-excludes=*.zip',
>                          '--extra-excludes=*.zip.asc',
>                          '--ssh-user', branchConfig['stage_username'],
>                          '--ssh-key', branchConfig['stage_ssh_key'],
> +                        '--dry-run-push'

Per my comment on the other patch, this needs to be adjusted to be an arg instead of an option.
Attachment #776405 - Flags: review?(bhearsum)
You also still need to address the following points:
* Drop disablePushToMirrors from Fennec release templates
* Remove verifyConfigs as a condition of pushing to mirrors: https://github.com/mozilla/build-buildbotcustom/blob/06795046569f1c6d4b1f425c0fde3f559abe5f32/process/release.py#L1767
* Append push_to_mirrors to the deliverables_builder list, rather than post_update_builders (this is a relic, from when the updates builder still generated files that get pushed).
* Set 'enableAutomaticPushToMirrors' to True in Fennec beta release configs.

Comment 25

4 years ago
Created attachment 780540 [details] [diff] [review]
stage_tasks_fennec.diff

check if dry_run_push is in args
Attachment #776404 - Attachment is obsolete: true
Attachment #776405 - Attachment is obsolete: true

Updated

4 years ago
Attachment #780540 - Flags: review?(bhearsum)

Comment 26

4 years ago
Created attachment 780544 [details] [diff] [review]
bbconf_fennec.diff

- Added dry_run_push argument to check_permissions_factory.
- Removed verifyConfigs condition
- Append push_to_mirrors to deliverables_builders instead of post_update_builders
Attachment #780544 - Flags: review?(bhearsum)

Comment 27

4 years ago
Created attachment 780547 [details] [diff] [review]
bbconf_fennec.diff

- Set enableAutomaticPushToMirrors as True in mozilla/release-fennec-mozilla-beta.py.
- Removed disablePushToMirrors from mozilla/release-fennec-mozilla-release.py.template
Attachment #780547 - Flags: review?(bhearsum)
Attachment #780540 - Flags: review?(bhearsum) → review+
Unassigning as there are only 2 more weeks for Jason's internship to be over with and we would like him to focus on the other bugs he already has.
Assignee: yshun → nobody
Thanks for your work here Jason! I'm going to throw some comments about the current patches in, even though this won't be fixed quite yet.
Comment on attachment 780544 [details] [diff] [review]
bbconf_fennec.diff

Review of attachment 780544 [details] [diff] [review]:
-----------------------------------------------------------------

Some of the builders need their names adjusted. Specifically, push_to_mirrors, postrelease both need to include productName, otherwise Fennec and Firefox try to use the same names.

::: process/release.py
@@ +1762,5 @@
>              upstreamBuilders=readyForReleaseUpstreams,
>              builderNames=[builderPrefix('ready_for_release')],
>          ))
>  
> +    if releaseConfig.get('enableAutomaticPushToMirrors')

Syntax error here

@@ +1767,2 @@
>          if releaseConfig.get('disableVirusCheck'):
> +            deliverables_builders.append(builderPrefix('push_to_mirrors'))

I think this actually needs to be post_deliverables_builders...it wasn't triggered automatically when I tried to test this.
Attachment #780544 - Flags: review?(bhearsum)
Comment on attachment 780547 [details] [diff] [review]
bbconf_fennec.diff

Review of attachment 780547 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-beta.py
@@ -111,1 @@
>  # Fennec specific

Need to modify the .template, not the .py. when we run the release, the .py gets overridden with a completely new file based on the .template. The staging templates need updating, too.

::: mozilla/release-fennec-mozilla-release.py.template
@@ -122,5 @@
>  releaseConfig['disableBouncerEntries']    = True
>  releaseConfig['disableStandaloneRepacks'] = True
>  releaseConfig['disablePermissionCheck']   = True
>  releaseConfig['disableVirusCheck']        = True
> -releaseConfig['disablePushToMirrors']     = True

Need to drop disablePushToMirrors from all of the fennec templates, not just this one.
Attachment #780547 - Flags: review?(bhearsum)
Product: mozilla.org → Release Engineering
Created attachment 818058 [details] [diff] [review]
[braindump] Exclude files not relevant to end-users

This is what I suggested in bug 885013 (which was duped over to this bug). I'll attach a log of a test run with this.
Attachment #818058 - Flags: review?(aki)
Created attachment 818059 [details]
Log of push with attachment 818058 [details] [diff] [review]
(Reporter)

Updated

4 years ago
Attachment #818058 - Flags: review?(aki) → review+
Comment on attachment 818058 [details] [diff] [review]
[braindump] Exclude files not relevant to end-users

https://hg.mozilla.org/build/braindump/rev/3da1c5de0c72
Attachment #818058 - Flags: checked-in+

Updated

3 years ago
Summary: Mobile release automation to move candidates to releases/ → Mobile release automation to move candidates to releases/ (push to mirrors)
(Assignee)

Updated

3 years ago
Assignee: nobody → rail
Priority: P4 → P2
(Assignee)

Comment 35

3 years ago
Created attachment 8488338 [details] [diff] [review]
push_to_mirrors-tools.diff

* switch to argparse
* Do not use branch configs at all, everything we needed can be retrieved from the release configs. The only missing key was stage_product. It can be read from branch config, but it's under ["platforms"][platform]. It's much clearer to have it in the release config explicitly.
* no need to pass -c $branchConfig anymore
Attachment #8488338 - Flags: review?(bhearsum)
(Assignee)

Comment 36

3 years ago
Created attachment 8488340 [details] [diff] [review]
push_to_mirrors-buildbot-configs.diff

* add stage_product
* get rid of disablePushToMirrors
* add ftpSymlinkName for fennec (postrelease)
Attachment #8488340 - Flags: review?(bhearsum)
(Assignee)

Comment 37

3 years ago
Created attachment 8488341 [details] [diff] [review]
push_to_mirrors-buildbotcustom.diff

* I would recommend applying the patch and looking at it with -b (without whitespace changes)
* updated extra_args (no branch config)
* changed the builder names to include product name (to avoid collisions)
* no more disablePushToMirrors
Attachment #8488341 - Flags: review?(bhearsum)
Comment on attachment 8488338 [details] [diff] [review]
push_to_mirrors-tools.diff

Review of attachment 8488338 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/stage-tasks.py
@@ +262,3 @@
>      )
> +    parser.add_argument("-r", "--release-config", required=True,
> +                        help="Release coonfig file location relative to "

Typo: coonfig, unless this actually about a racoon.
Attachment #8488338 - Flags: review?(bhearsum) → review+
Attachment #8488340 - Flags: review?(bhearsum) → review+
Attachment #8488341 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 39

3 years ago
Comment on attachment 8488340 [details] [diff] [review]
push_to_mirrors-buildbot-configs.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/c4602ebf9eb4
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/2e7fcdcd13ff
Attachment #8488340 - Flags: checked-in+
(Assignee)

Comment 40

3 years ago
Comment on attachment 8488341 [details] [diff] [review]
push_to_mirrors-buildbotcustom.diff

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/c4ca54bac6fc
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/bb6c243a5303
Attachment #8488341 - Flags: checked-in+
(Assignee)

Comment 41

3 years ago
Comment on attachment 8488338 [details] [diff] [review]
push_to_mirrors-tools.diff

https://hg.mozilla.org/build/tools/rev/c4a74801819d
Attachment #8488338 - Flags: checked-in+
(Assignee)

Comment 42

3 years ago
In production. I'm going to update the docs now.
(Assignee)

Comment 43

3 years ago
https://wiki.mozilla.org/index.php?title=Release:Release_Automation_on_Mercurial:Updates_through_Shipping&diff=0&oldid=1014472

https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=1014959&oldid=1009676
(Assignee)

Comment 44

3 years ago
Created attachment 8489502 [details] [diff] [review]
update_release_templates.diff
Attachment #8489502 - Flags: review?(bhearsum)
Attachment #8489502 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 45

3 years ago
Comment on attachment 8489502 [details] [diff] [review]
update_release_templates.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/5870a6261621
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/f2081be7d41b

No reconfig required for this, just hg up, what I did.
Attachment #8489502 - Flags: checked-in+
(Assignee)

Comment 46

3 years ago
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&diff=1014993&oldid=1014961

https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=1014995&oldid=1014959
(Assignee)

Comment 47

3 years ago
Push to mirrors worked as expected, except it wasn't fired automatically for a beta.

I'm not sure if I can work on making it work this week, so I updated the docs accordingly.
(Assignee)

Comment 48

3 years ago
Created attachment 8490006 [details] [diff] [review]
buildbot-configs_followup.diff

To be tested
(Assignee)

Comment 49

3 years ago
Created attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff
(Assignee)

Comment 50

3 years ago
Just in case :)
https://hg.mozilla.org/build/braindump/rev/ecfc32463222
(Assignee)

Comment 51

3 years ago
Created attachment 8495990 [details] [diff] [review]
push-buildbot-configs.diff

WCPGW? :)
Attachment #8490006 - Attachment is obsolete: true
Attachment #8495990 - Flags: review?(bhearsum)
(Assignee)

Comment 52

3 years ago
Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

checking enableSigningAtBuildTime is redundant now, it's True everywhere and by default.
Attachment #8490007 - Flags: review?(bhearsum)
Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

Review of attachment 8490007 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1762,5 @@
>  
> +    if releaseConfig.get('enableAutomaticPushToMirrors') and \
> +        hasPlatformSubstring(releaseConfig['enUSPlatforms'], 'android'):
> +            post_deliverables_builders.append(builderPrefix('%s_push_to_mirrors' % releaseConfig['productName']))
> +

This is OK, but I feel like we might be able to integrate it with the above block. Maybe even enable av scan for Fennec? Both of these seem like a follow-up.
Attachment #8490007 - Flags: review?(bhearsum) → review+
Comment on attachment 8495990 [details] [diff] [review]
push-buildbot-configs.diff

Review of attachment 8495990 [details] [diff] [review]:
-----------------------------------------------------------------

Make sure these get on production before the next Beta!
Attachment #8495990 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 55

3 years ago
Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/c3a1b00f425d
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/7c50afbfb0aa
Attachment #8490007 - Flags: checked-in+
(Assignee)

Comment 56

3 years ago
Comment on attachment 8495990 [details] [diff] [review]
push-buildbot-configs.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/9eaa48ce8a9c
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/6fa302901916
Attachment #8495990 - Flags: checked-in+
(Assignee)

Comment 57

3 years ago
Worked fin in Fennec 33.0b8 build2

Docs updated:
https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=1019880&oldid=1015294
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&diff=1019881&oldid=1018033
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Merged to production, and deployed.
You need to log in before you can comment on or make changes to this bug.