543109 - Mobile release automation to move candidates to releases/ (push to mirrors)

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P2)

Description

[Aki Sasaki (not active)]

The layout is the same for builds/source.
The repos, however, aren't 1.0/repos/1.9.2_LOCALE/dists but 1.0/maemo/LOCALE/dists.

We also need install files to point to moff.m.c.

Comment 1

[[e-mail] (:bhearsum)]

Out of curiosity, is there any reason why the candidates directories can't be exactly the same as the releases ones?

Comment 2

[Aki Sasaki (not active)]

I'm going to assume you mean "...any reason why the candidates directories can't have the same layout as the releases ones?", and that you're not suggesting we publish candidate builds directly to the releases directory tree.

Couple reasons.
First, we changed the releases/ dir layout at the zero hour of 1.0 GM, and we've had no releases go live since then.  Stuart did ping me about whether we wanted to clean up some of the layout in http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/1.0/ , so the layout isn't fully set yet.  Thinking we might want to wait a release or 2 to make sure the layout's set for the moment.

Also, the deb repos are currently under version-candidates/repos/ rather than version-candidates/buildN/.  This is to facilitate update testing, since people who install build1 for testing will be able to update automatically to build2 via that repo.  For 1.0 I was asked to move the repos/locale/dists directories to releases/1.0/maemo/locale/dists and to not move the .install file (which is needed for candidate repo testing).

Comment 3

[Aki Sasaki (not active)]

Docs on how to do this as of today are here:

https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Mobile

As mentioned above, the repos are the main reason why we might need automation here. Maybe we need to push to both locations (VERSION-candidates/repos/LOCALE as well as VERSION-candidates/buildN/maemo/LOCALE ?)

Comment 5

[Aki Sasaki (not active)]

Triage: Marked as triagefollowup since I won't have time in Q4 for this.

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Aki Sasaki [:aki] from comment #5)
> Triage: Marked as triagefollowup since I won't have time in Q4 for this.

If it's sat for 2 years, I think we're OK with another month. ;)

But is this still required? It's marked as Maemo, which is why I ask.

How do these files make it into releases/ right now? Is there a script?

Comment 7

[Aki Sasaki (not active)]

ffxbld@stage.m.o:push_fennec_{beta,release}.sh

Comment 8

[Aki Sasaki (not active)]

https://wiki.mozilla.org/Releases/BuildNotesTemplate#Push_the_files :)

Comment 9

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 11

[[e-mail] (:bhearsum)]

Aki, this has been assigned to you for over a year without action. I'm going to put it back in the pool to make it more clear that someone else should feel free to pick it up. Things have changed since this was originally filed, and I think this is relatively trivially to do now. I did a diff of the candidates dir vs. releases dir, here's what we _don't_ want synced from candidates -> releases:
- *_info.txt
- logs/
- unsigned/

All of these things are already listed in https://github.com/mozilla/build-tools/blob/946ea8c7ef59d1f51065839d3a7db9323d493089/scripts/release/stage-tasks.py#L31, which means we don't have to tweak this script much. I think we need to make the following changes to make this happen:
* Make the dry run of push to mirrors happen when 'dry_run_push' is in args (instead of inferring it from a different arg): https://github.com/mozilla/build-tools/blob/946ea8c7ef59d1f51065839d3a7db9323d493089/scripts/release/stage-tasks.py#L234
* Add 'dry_run_push' to the args for the check_permissions factory: https://github.com/mozilla/build-buildbotcustom/blob/2a9327ff1b2682d9c47350c9c44c95febc4d690b/process/release.py#L1227
* Drop disablePushToMirrors from Fennec release templates

The only thing to watch out for here is the scheduling -- some of the push to mirrors stuff may be tied into updates at the moment.

Comment 12

[Aki Sasaki (not active)]

That's cool, the only reason I didn't unassign myself was the old bug rule.

Comment 13

[[e-mail] (:bhearsum)]

Need to make sure the exclude list includes all of the stuff from https://bugzilla.mozilla.org/show_bug.cgi?id=885013#c0 when this is done.

Comment 14

[Armen [:armenzg]]

Is this a bug that Jason could work on?

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #14)
> Is this a bug that Jason could work on?

I think so. I'd like to review comment #11 before though, to make sure it's still accurate.

Comment 16

[[e-mail] (:bhearsum)]

Comment #11 is still accurate, but incomplete. We need to decide whether or not we want this to run as part of automation, like Firefox's push to mirrors does, or whether it needs to be fully manual.

Aki, I'm leaning towards having it run after all deliverables are ready. Unless there's a reason not to, it seems like having that consistency is better. Any thoughts?

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #16)
> Comment #11 is still accurate, but incomplete. We need to decide whether or
> not we want this to run as part of automation, like Firefox's push to
> mirrors does, or whether it needs to be fully manual.
> 
> Aki, I'm leaning towards having it run after all deliverables are ready.
> Unless there's a reason not to, it seems like having that consistency is
> better. Any thoughts?

Of course, we still do it by hand for mozilla-release, so I'm only talking about automatic pushing for mozilla-beta.

If we want to run this after deliverables are ready we also need to:
* Remove verifyConfigs as a condition of pushing to mirrors: https://github.com/mozilla/build-buildbotcustom/blob/06795046569f1c6d4b1f425c0fde3f559abe5f32/process/release.py#L1767
* Append push_to_mirrors to the deliverables_builder list, rather than post_update_builders (this is a relic, from when the updates builder still generated files that get pushed).
* Set 'enableAutomaticPushToMirrors' to True in Fennec beta release configs.

Comment 19

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #17)
> (In reply to Ben Hearsum [:bhearsum] from comment #16)
> > Comment #11 is still accurate, but incomplete. We need to decide whether or
> > not we want this to run as part of automation, like Firefox's push to
> > mirrors does, or whether it needs to be fully manual.
> > 
> > Aki, I'm leaning towards having it run after all deliverables are ready.
> > Unless there's a reason not to, it seems like having that consistency is
> > better. Any thoughts?
> 
> Of course, we still do it by hand for mozilla-release, so I'm only talking
> about automatic pushing for mozilla-beta.

That sounds fine to me.

Comment 20

[Jason Yeo[:jyeo]]

Attachment: Add dry run option to optionparser

Comment 21

[Jason Yeo[:jyeo]]

Attachment: Add new arg to scriptfactory

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 776404 [details] [diff] [review]
Add dry run option to optionparser

Review of attachment 776404 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/stage-tasks.py
@@ +263,5 @@
>                        productName=productName,
>                        version=version,
>                        buildNumber=buildNumber,
>                        extra_excludes=options.extra_excludes,
> +                      dryRun=options.dry_run_push)

This isn't quite what I had in mind for dry_run_push. The "if 'permissions' in args or 'antivirus' in args" should just get replaced by 'if dry_run_push in args'. And the second pushToMirrors call should remain unaltered.

The idea is that a dryRun of pushToMirrors is its own unique thing now. The way this patch is written, it will never run for Fennec, because Fennec doesn't do a permissions or AV check.

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 776405 [details] [diff] [review]
Add new arg to scriptfactory

Review of attachment 776405 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1268,5 @@
>                          '--extra-excludes=*.zip',
>                          '--extra-excludes=*.zip.asc',
>                          '--ssh-user', branchConfig['stage_username'],
>                          '--ssh-key', branchConfig['stage_ssh_key'],
> +                        '--dry-run-push'

Per my comment on the other patch, this needs to be adjusted to be an arg instead of an option.

Comment 24

[[e-mail] (:bhearsum)]

You also still need to address the following points:
* Drop disablePushToMirrors from Fennec release templates
* Remove verifyConfigs as a condition of pushing to mirrors: https://github.com/mozilla/build-buildbotcustom/blob/06795046569f1c6d4b1f425c0fde3f559abe5f32/process/release.py#L1767
* Append push_to_mirrors to the deliverables_builder list, rather than post_update_builders (this is a relic, from when the updates builder still generated files that get pushed).
* Set 'enableAutomaticPushToMirrors' to True in Fennec beta release configs.

Comment 25

[Jason Yeo[:jyeo]]

Attachment: stage_tasks_fennec.diff

check if dry_run_push is in args

Comment 26

[Jason Yeo[:jyeo]]

Attachment: bbconf_fennec.diff

- Added dry_run_push argument to check_permissions_factory.
- Removed verifyConfigs condition
- Append push_to_mirrors to deliverables_builders instead of post_update_builders

Comment 27

[Jason Yeo[:jyeo]]

Attachment: bbconf_fennec.diff

- Set enableAutomaticPushToMirrors as True in mozilla/release-fennec-mozilla-beta.py.
- Removed disablePushToMirrors from mozilla/release-fennec-mozilla-release.py.template

Comment 28

[Armen [:armenzg]]

Unassigning as there are only 2 more weeks for Jason's internship to be over with and we would like him to focus on the other bugs he already has.

Comment 29

[[e-mail] (:bhearsum)]

Thanks for your work here Jason! I'm going to throw some comments about the current patches in, even though this won't be fixed quite yet.

Comment 30

[[e-mail] (:bhearsum)]

Comment on attachment 780544 [details] [diff] [review]
bbconf_fennec.diff

Review of attachment 780544 [details] [diff] [review]:
-----------------------------------------------------------------

Some of the builders need their names adjusted. Specifically, push_to_mirrors, postrelease both need to include productName, otherwise Fennec and Firefox try to use the same names.

::: process/release.py
@@ +1762,5 @@
>              upstreamBuilders=readyForReleaseUpstreams,
>              builderNames=[builderPrefix('ready_for_release')],
>          ))
>  
> +    if releaseConfig.get('enableAutomaticPushToMirrors')

Syntax error here

@@ +1767,2 @@
>          if releaseConfig.get('disableVirusCheck'):
> +            deliverables_builders.append(builderPrefix('push_to_mirrors'))

I think this actually needs to be post_deliverables_builders...it wasn't triggered automatically when I tried to test this.

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 780547 [details] [diff] [review]
bbconf_fennec.diff

Review of attachment 780547 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-beta.py
@@ -111,1 @@
>  # Fennec specific

Need to modify the .template, not the .py. when we run the release, the .py gets overridden with a completely new file based on the .template. The staging templates need updating, too.

::: mozilla/release-fennec-mozilla-release.py.template
@@ -122,5 @@
>  releaseConfig['disableBouncerEntries']    = True
>  releaseConfig['disableStandaloneRepacks'] = True
>  releaseConfig['disablePermissionCheck']   = True
>  releaseConfig['disableVirusCheck']        = True
> -releaseConfig['disablePushToMirrors']     = True

Need to drop disablePushToMirrors from all of the fennec templates, not just this one.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [braindump] Exclude files not relevant to end-users

This is what I suggested in bug 885013 (which was duped over to this bug). I'll attach a log of a test run with this.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Log of push with attachment 818058

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 818058 [details] [diff] [review]
[braindump] Exclude files not relevant to end-users

https://hg.mozilla.org/build/braindump/rev/3da1c5de0c72

Comment 35

[Rail Aliiev [:rail]]

Attachment: push_to_mirrors-tools.diff

* switch to argparse
* Do not use branch configs at all, everything we needed can be retrieved from the release configs. The only missing key was stage_product. It can be read from branch config, but it's under ["platforms"][platform]. It's much clearer to have it in the release config explicitly.
* no need to pass -c $branchConfig anymore

Comment 36

[Rail Aliiev [:rail]]

Attachment: push_to_mirrors-buildbot-configs.diff

* add stage_product
* get rid of disablePushToMirrors
* add ftpSymlinkName for fennec (postrelease)

Comment 37

[Rail Aliiev [:rail]]

Attachment: push_to_mirrors-buildbotcustom.diff

* I would recommend applying the patch and looking at it with -b (without whitespace changes)
* updated extra_args (no branch config)
* changed the builder names to include product name (to avoid collisions)
* no more disablePushToMirrors

Comment 38

[[e-mail] (:bhearsum)]

Comment on attachment 8488338 [details] [diff] [review]
push_to_mirrors-tools.diff

Review of attachment 8488338 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/stage-tasks.py
@@ +262,3 @@
>      )
> +    parser.add_argument("-r", "--release-config", required=True,
> +                        help="Release coonfig file location relative to "

Typo: coonfig, unless this actually about a racoon.

Comment 39

[Rail Aliiev [:rail]]

Comment on attachment 8488340 [details] [diff] [review]
push_to_mirrors-buildbot-configs.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/c4602ebf9eb4
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/2e7fcdcd13ff

Comment 40

[Rail Aliiev [:rail]]

Comment on attachment 8488341 [details] [diff] [review]
push_to_mirrors-buildbotcustom.diff

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/c4ca54bac6fc
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/bb6c243a5303

Comment 41

[Rail Aliiev [:rail]]

Comment on attachment 8488338 [details] [diff] [review]
push_to_mirrors-tools.diff

https://hg.mozilla.org/build/tools/rev/c4a74801819d

Comment 42

[Rail Aliiev [:rail]]

In production. I'm going to update the docs now.

Comment 43

[Rail Aliiev [:rail]]

https://wiki.mozilla.org/index.php?title=Release:Release_Automation_on_Mercurial:Updates_through_Shipping&diff=0&oldid=1014472

https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=1014959&oldid=1009676

Comment 44

[Rail Aliiev [:rail]]

Attachment: update_release_templates.diff

Comment 45

[Rail Aliiev [:rail]]

Comment on attachment 8489502 [details] [diff] [review]
update_release_templates.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/5870a6261621
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/f2081be7d41b

No reconfig required for this, just hg up, what I did.

Comment 46

[Rail Aliiev [:rail]]

https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&diff=1014993&oldid=1014961

https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=1014995&oldid=1014959

Comment 47

[Rail Aliiev [:rail]]

Push to mirrors worked as expected, except it wasn't fired automatically for a beta.

I'm not sure if I can work on making it work this week, so I updated the docs accordingly.

Comment 48

[Rail Aliiev [:rail]]

Attachment: buildbot-configs_followup.diff

To be tested

Comment 49

[Rail Aliiev [:rail]]

Attachment: buildbotcustom_followup.diff

Comment 50

[Rail Aliiev [:rail]]

Just in case :)
https://hg.mozilla.org/build/braindump/rev/ecfc32463222

Comment 51

[Rail Aliiev [:rail]]

Attachment: push-buildbot-configs.diff

WCPGW? :)

Comment 52

[Rail Aliiev [:rail]]

Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

checking enableSigningAtBuildTime is redundant now, it's True everywhere and by default.

Comment 53

[[e-mail] (:bhearsum)]

Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

Review of attachment 8490007 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1762,5 @@
>  
> +    if releaseConfig.get('enableAutomaticPushToMirrors') and \
> +        hasPlatformSubstring(releaseConfig['enUSPlatforms'], 'android'):
> +            post_deliverables_builders.append(builderPrefix('%s_push_to_mirrors' % releaseConfig['productName']))
> +

This is OK, but I feel like we might be able to integrate it with the above block. Maybe even enable av scan for Fennec? Both of these seem like a follow-up.

Comment 54

[[e-mail] (:bhearsum)]

Comment on attachment 8495990 [details] [diff] [review]
push-buildbot-configs.diff

Review of attachment 8495990 [details] [diff] [review]:
-----------------------------------------------------------------

Make sure these get on production before the next Beta!

Comment 55

[Rail Aliiev [:rail]]

Comment on attachment 8490007 [details] [diff] [review]
buildbotcustom_followup.diff

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/c3a1b00f425d
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/7c50afbfb0aa

Comment 56

[Rail Aliiev [:rail]]

Comment on attachment 8495990 [details] [diff] [review]
push-buildbot-configs.diff

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/9eaa48ce8a9c
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/6fa302901916

Comment 57

[Rail Aliiev [:rail]]

Worked fin in Fennec 33.0b8 build2

Docs updated:
https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&diff=1019880&oldid=1015294
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&diff=1019881&oldid=1018033

Comment 58

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.