Closed Bug 543109 Opened 10 years ago Closed 5 years ago

Mobile release automation to move candidates to releases/ (push to mirrors)

Categories

(Release Engineering :: Release Automation: Other, defect, P2)

ARM
All
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: rail)

References

Details

(Whiteboard: [mobile][automation][postrelease])

Attachments

(11 files, 3 obsolete files)

1007 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
2.05 KB, patch
Details | Diff | Splinter Review
2.28 KB, patch
Details | Diff | Splinter Review
810 bytes, patch
aki
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
26.63 KB, text/plain
Details
13.69 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
61.20 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
17.75 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
906 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
11.77 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
2.88 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
The layout is the same for builds/source.
The repos, however, aren't 1.0/repos/1.9.2_LOCALE/dists but 1.0/maemo/LOCALE/dists.

We also need install files to point to moff.m.c.
Blocks: maemo4, 478420
Assignee: aki → nobody
Priority: -- → P4
Out of curiosity, is there any reason why the candidates directories can't be exactly the same as the releases ones?
I'm going to assume you mean "...any reason why the candidates directories can't have the same layout as the releases ones?", and that you're not suggesting we publish candidate builds directly to the releases directory tree.

Couple reasons.
First, we changed the releases/ dir layout at the zero hour of 1.0 GM, and we've had no releases go live since then.  Stuart did ping me about whether we wanted to clean up some of the layout in http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/1.0/ , so the layout isn't fully set yet.  Thinking we might want to wait a release or 2 to make sure the layout's set for the moment.

Also, the deb repos are currently under version-candidates/repos/ rather than version-candidates/buildN/.  This is to facilitate update testing, since people who install build1 for testing will be able to update automatically to build2 via that repo.  For 1.0 I was asked to move the repos/locale/dists directories to releases/1.0/maemo/locale/dists and to not move the .install file (which is needed for candidate repo testing).
Docs on how to do this as of today are here:

https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Mobile

As mentioned above, the repos are the main reason why we might need automation here. Maybe we need to push to both locations (VERSION-candidates/repos/LOCALE as well as VERSION-candidates/buildN/maemo/LOCALE ?)
Whiteboard: [mobile][automation]
Assignee: nobody → aki
Duplicate of this bug: 682787
Whiteboard: [mobile][automation] → [mobile][automation][triagefollowup]
Triage: Marked as triagefollowup since I won't have time in Q4 for this.
(In reply to Aki Sasaki [:aki] from comment #5)
> Triage: Marked as triagefollowup since I won't have time in Q4 for this.

If it's sat for 2 years, I think we're OK with another month. ;)

But is this still required? It's marked as Maemo, which is why I ask.

How do these files make it into releases/ right now? Is there a script?
Whiteboard: [mobile][automation][triagefollowup] → [mobile][automation]
ffxbld@stage.m.o:push_fennec_{beta,release}.sh
OS: Maemo → All
No longer blocks: hg-automation
Mass move of bugs to Release Automation component.
Component: Release Engineering → Release Engineering: Automation (Release Automation)
No longer blocks: hg-automation
Duplicate of this bug: 764970
Aki, this has been assigned to you for over a year without action. I'm going to put it back in the pool to make it more clear that someone else should feel free to pick it up. Things have changed since this was originally filed, and I think this is relatively trivially to do now. I did a diff of the candidates dir vs. releases dir, here's what we _don't_ want synced from candidates -> releases:
- *_info.txt
- logs/
- unsigned/

All of these things are already listed in https://github.com/mozilla/build-tools/blob/946ea8c7ef59d1f51065839d3a7db9323d493089/scripts/release/stage-tasks.py#L31, which means we don't have to tweak this script much. I think we need to make the following changes to make this happen:
* Make the dry run of push to mirrors happen when 'dry_run_push' is in args (instead of inferring it from a different arg): https://github.com/mozilla/build-tools/blob/946ea8c7ef59d1f51065839d3a7db9323d493089/scripts/release/stage-tasks.py#L234
* Add 'dry_run_push' to the args for the check_permissions factory: https://github.com/mozilla/build-buildbotcustom/blob/2a9327ff1b2682d9c47350c9c44c95febc4d690b/process/release.py#L1227
* Drop disablePushToMirrors from Fennec release templates

The only thing to watch out for here is the scheduling -- some of the push to mirrors stuff may be tied into updates at the moment.
Assignee: aki → nobody
That's cool, the only reason I didn't unassign myself was the old bug rule.
Whiteboard: [mobile][automation] → [mobile][automation][postrelease]
Need to make sure the exclude list includes all of the stuff from https://bugzilla.mozilla.org/show_bug.cgi?id=885013#c0 when this is done.
Is this a bug that Jason could work on?
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #14)
> Is this a bug that Jason could work on?

I think so. I'd like to review comment #11 before though, to make sure it's still accurate.
Comment #11 is still accurate, but incomplete. We need to decide whether or not we want this to run as part of automation, like Firefox's push to mirrors does, or whether it needs to be fully manual.

Aki, I'm leaning towards having it run after all deliverables are ready. Unless there's a reason not to, it seems like having that consistency is better. Any thoughts?
Flags: needinfo?(aki)
(In reply to Ben Hearsum [:bhearsum] from comment #16)
> Comment #11 is still accurate, but incomplete. We need to decide whether or
> not we want this to run as part of automation, like Firefox's push to
> mirrors does, or whether it needs to be fully manual.
> 
> Aki, I'm leaning towards having it run after all deliverables are ready.
> Unless there's a reason not to, it seems like having that consistency is
> better. Any thoughts?

Of course, we still do it by hand for mozilla-release, so I'm only talking about automatic pushing for mozilla-beta.

If we want to run this after deliverables are ready we also need to:
* Remove verifyConfigs as a condition of pushing to mirrors: https://github.com/mozilla/build-buildbotcustom/blob/06795046569f1c6d4b1f425c0fde3f559abe5f32/process/release.py#L1767
* Append push_to_mirrors to the deliverables_builder list, rather than post_update_builders (this is a relic, from when the updates builder still generated files that get pushed).
* Set 'enableAutomaticPushToMirrors' to True in Fennec beta release configs.
Duplicate of this bug: 885013
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> (In reply to Ben Hearsum [:bhearsum] from comment #16)
> > Comment #11 is still accurate, but incomplete. We need to decide whether or
> > not we want this to run as part of automation, like Firefox's push to
> > mirrors does, or whether it needs to be fully manual.
> > 
> > Aki, I'm leaning towards having it run after all deliverables are ready.
> > Unless there's a reason not to, it seems like having that consistency is
> > better. Any thoughts?
> 
> Of course, we still do it by hand for mozilla-release, so I'm only talking
> about automatic pushing for mozilla-beta.

That sounds fine to me.
Flags: needinfo?(aki)
Assignee: nobody → yshun
Attachment #776404 - Flags: review?(bhearsum)
Attached patch Add new arg to scriptfactory (obsolete) — Splinter Review
Attachment #776405 - Flags: review?(bhearsum)
Comment on attachment 776404 [details] [diff] [review]
Add dry run option to optionparser

Review of attachment 776404 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/stage-tasks.py
@@ +263,5 @@
>                        productName=productName,
>                        version=version,
>                        buildNumber=buildNumber,
>                        extra_excludes=options.extra_excludes,
> +                      dryRun=options.dry_run_push)

This isn't quite what I had in mind for dry_run_push. The "if 'permissions' in args or 'antivirus' in args" should just get replaced by 'if dry_run_push in args'. And the second pushToMirrors call should remain unaltered.

The idea is that a dryRun of pushToMirrors is its own unique thing now. The way this patch is written, it will never run for Fennec, because Fennec doesn't do a permissions or AV check.
Attachment #776404 - Flags: review?(bhearsum) → review-
Comment on attachment 776405 [details] [diff] [review]
Add new arg to scriptfactory

Review of attachment 776405 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1268,5 @@
>                          '--extra-excludes=*.zip',
>                          '--extra-excludes=*.zip.asc',
>                          '--ssh-user', branchConfig['stage_username'],
>                          '--ssh-key', branchConfig['stage_ssh_key'],
> +                        '--dry-run-push'

Per my comment on the other patch, this needs to be adjusted to be an arg instead of an option.
Attachment #776405 - Flags: review?(bhearsum)
You also still need to address the following points:
* Drop disablePushToMirrors from Fennec release templates
* Remove verifyConfigs as a condition of pushing to mirrors: https://github.com/mozilla/build-buildbotcustom/blob/06795046569f1c6d4b1f425c0fde3f559abe5f32/process/release.py#L1767
* Append push_to_mirrors to the deliverables_builder list, rather than post_update_builders (this is a relic, from when the updates builder still generated files that get pushed).
* Set 'enableAutomaticPushToMirrors' to True in Fennec beta release configs.
check if dry_run_push is in args
Attachment #776404 - Attachment is obsolete: true
Attachment #776405 - Attachment is obsolete: true
Attachment #780540 - Flags: review?(bhearsum)
- Added dry_run_push argument to check_permissions_factory.
- Removed verifyConfigs condition
- Append push_to_mirrors to deliverables_builders instead of post_update_builders
Attachment #780544 - Flags: review?(bhearsum)
- Set enableAutomaticPushToMirrors as True in mozilla/release-fennec-mozilla-beta.py.
- Removed disablePushToMirrors from mozilla/release-fennec-mozilla-release.py.template
Attachment #780547 - Flags: review?(bhearsum)
Attachment #780540 - Flags: review?(bhearsum) → review+
Unassigning as there are only 2 more weeks for Jason's internship to be over with and we would like him to focus on the other bugs he already has.
Assignee: yshun → nobody
Thanks for your work here Jason! I'm going to throw some comments about the current patches in, even though this won't be fixed quite yet.
Comment on attachment 780544 [details] [diff] [review]
bbconf_fennec.diff

Review of attachment 780544 [details] [diff] [review]:
-----------------------------------------------------------------

Some of the builders need their names adjusted. Specifically, push_to_mirrors, postrelease both need to include productName, otherwise Fennec and Firefox try to use the same names.

::: process/release.py
@@ +1762,5 @@
>              upstreamBuilders=readyForReleaseUpstreams,
>              builderNames=[builderPrefix('ready_for_release')],
>          ))
>  
> +    if releaseConfig.get('enableAutomaticPushToMirrors')

Syntax error here

@@ +1767,2 @@
>          if releaseConfig.get('disableVirusCheck'):
> +            deliverables_builders.append(builderPrefix('push_to_mirrors'))

I think this actually needs to be post_deliverables_builders...it wasn't triggered automatically when I tried to test this.
Attachment #780544 - Flags: review?(bhearsum)
Comment on attachment 780547 [details] [diff] [review]
bbconf_fennec.diff

Review of attachment 780547 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-beta.py
@@ -111,1 @@
>  # Fennec specific

Need to modify the .template, not the .py. when we run the release, the .py gets overridden with a completely new file based on the .template. The staging templates need updating, too.

::: mozilla/release-fennec-mozilla-release.py.template
@@ -122,5 @@
>  releaseConfig['disableBouncerEntries']    = True
>  releaseConfig['disableStandaloneRepacks'] = True
>  releaseConfig['disablePermissionCheck']   = True
>  releaseConfig['disableVirusCheck']        = True
> -releaseConfig['disablePushToMirrors']     = True

Need to drop disablePushToMirrors from all of the fennec templates, not just this one.
Attachment #780547 - Flags: review?(bhearsum)
Product: mozilla.org → Release Engineering
This is what I suggested in bug 885013 (which was duped over to this bug). I'll attach a log of a test run with this.
Attachment #818058 - Flags: review?(aki)
Attachment #818058 - Flags: review?(aki) → review+
Comment on attachment 818058 [details] [diff] [review]
[braindump] Exclude files not relevant to end-users

https://hg.mozilla.org/build/braindump/rev/3da1c5de0c72
Attachment #818058 - Flags: checked-in+
Summary: Mobile release automation to move candidates to releases/ → Mobile release automation to move candidates to releases/ (push to mirrors)
Assignee: nobody → rail
Priority: P4 → P2
* switch to argparse
* Do not use branch configs at all, everything we needed can be retrieved from the release configs. The only missing key was stage_product. It can be read from branch config, but it's under ["platforms"][platform]. It's much clearer to have it in the release config explicitly.
* no need to pass -c $branchConfig anymore
Attachment #8488338 - Flags: review?(bhearsum)
* add stage_product
* get rid of disablePushToMirrors
* add ftpSymlinkName for fennec (postrelease)
Attachment #8488340 - Flags: review?(bhearsum)
* I would recommend applying the patch and looking at it with -b (without whitespace changes)
* updated extra_args (no branch config)
* changed the builder names to include product name (to avoid collisions)
* no more disablePushToMirrors
Attachment #8488341 - Flags: review?(bhearsum)
Comment on attachment 8488338 [details] [diff] [review]
push_to_mirrors-tools.diff

Review of attachment 8488338 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/stage-tasks.py
@@ +262,3 @@
>      )
> +    parser.add_argument("-r", "--release-config", required=True,
> +                        help="Release coonfig file location relative to "

Typo: coonfig, unless this actually about a racoon.
Attachment #8488338 - Flags: review?(bhearsum) → review+
Attachment #8488340 - Flags: review?(bhearsum) → review+
Attachment #8488341 - Flags: review?(bhearsum) → review+
In production. I'm going to update the docs now.
Attachment #8489502 - Flags: review?(bhearsum)
Attachment #8489502 - Flags: review?(bhearsum) → review+
Comment on attachment 8489502 [details] [diff] [review]
update_release_templates.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/5870a6261621
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/f2081be7d41b

No reconfig required for this, just hg up, what I did.
Attachment #8489502 - Flags: checked-in+
Push to mirrors worked as expected, except it wasn't fired automatically for a beta.

I'm not sure if I can work on making it work this week, so I updated the docs accordingly.
Attached patch buildbot-configs_followup.diff (obsolete) — Splinter Review
To be tested
WCPGW? :)
Attachment #8490006 - Attachment is obsolete: true
Attachment #8495990 - Flags: review?(bhearsum)
Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

checking enableSigningAtBuildTime is redundant now, it's True everywhere and by default.
Attachment #8490007 - Flags: review?(bhearsum)
Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

Review of attachment 8490007 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1762,5 @@
>  
> +    if releaseConfig.get('enableAutomaticPushToMirrors') and \
> +        hasPlatformSubstring(releaseConfig['enUSPlatforms'], 'android'):
> +            post_deliverables_builders.append(builderPrefix('%s_push_to_mirrors' % releaseConfig['productName']))
> +

This is OK, but I feel like we might be able to integrate it with the above block. Maybe even enable av scan for Fennec? Both of these seem like a follow-up.
Attachment #8490007 - Flags: review?(bhearsum) → review+
Comment on attachment 8495990 [details] [diff] [review]
push-buildbot-configs.diff

Review of attachment 8495990 [details] [diff] [review]:
-----------------------------------------------------------------

Make sure these get on production before the next Beta!
Attachment #8495990 - Flags: review?(bhearsum) → review+
Merged to production, and deployed.
You need to log in before you can comment on or make changes to this bug.