1258059 - generify dev configs and environments for release-promotion by removing date strings and prepare for multiple products

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Jordan Lund (:jlund)]

date is simply the name of the gecko repo we use for staging release promotion releases. and currently it represents beta. but we are debating whether it should start representing mozilla-release and what does that mean for fennec..

I think it be better to have two proj repos. why? As we parallelize fennec betas and firefox m-r releases, we will likely need to have two staging environments and therefor two project branches.

regardless, I think it would be moar better if we didn't mention the word date anywhere and instead created environments that mirrored their purpose: s/date/staging_beta/

so instead of balrog channel rules mentioning 'date', we instead used 'dev-beta'. or with mh releases/ configs, we:
`hg mv releases/updates_date.py releases/staging_updates_firefox_beta.py`

what does this gain? I imagine in the futures, we will end up merging our staging env into each respective production repo so we can do staging releases based on real production repos in between real releases.

this bug will track that effort as I start fixing up various repos

Comment 1

[Jordan Lund (:jlund)]

this bug is starting to grow in scope as I dig. While I am updating things like updates/postrelease/bouncer config names, I am also going to make buildbot-configs/releasetasks be more aware that we need to support more than one product

Comment 2

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - bbotcustom - generify dev configs and env for relpro by removing date strings and allow multiple products, r?rail

Review commit: https://reviewboard.mozilla.org/r/41631/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41631/

Comment 3

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - gecko - generify dev configs and env for relpro by removing date strings and allow multiple products, r?rail

Review commit: https://reviewboard.mozilla.org/r/41639/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41639/

Comment 4

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - bbot-cfgs - generify dev configs and env for relpro by removing date strings and allow multiple products, r?rail

Review commit: https://reviewboard.mozilla.org/r/41635/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41635/

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8733192 [details]
MozReview Request: Bug 1258059 - gecko - generify dev configs and env for relpro by removing date strings and allow multiple products, r?rail

https://reviewboard.mozilla.org/r/41639/#review38035

I think you also need to add l10n config, something like http://hg.mozilla.org/projects/date/file/6b69342dcce3/testing/mozharness/configs/single_locale/date.py. Make sure to sync it with beta variables, IIRC we removed some of them after we switched to beta.

::: testing/mozharness/configs/builds/branch_specifics.py:231
(Diff revision 1)
>          'stage_server': 'upload.ffxbld.productdelivery.prod.mozaws.net',
>      },
>      # When build promotion goes live the mozconfig changes are probably better
>      # expressed once in files like configs/builds/releng_base_windows_32_builds.py
>      'date': {
> +        'update_channel': 'beta-dev',

Because we love being cryptic, we build the stub installer only for particular "update channels" (surprise!). You also need to update confvars.sh to something similar to http://hg.mozilla.org/projects/date/file/tip/browser/confvars.sh#l17

::: testing/mozharness/configs/releases/dev_bouncer_firefox_beta.py:1
(Diff revision 1)
> +# lint_ignore=E501

I'm not sure if we need this file at all. We can use the same file we use for betas, until we want to diverge. The only difference should be the API prefox, which is not here. I'm afraid that we won't be keeping it up to date.

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8733194 [details]
MozReview Request: Bug 1258059 - bbot-cfgs - generify dev configs and env for relpro by removing date strings and allow multiple products, r?rail

https://reviewboard.mozilla.org/r/41635/#review38039

::: mozilla/project_branches.py:163
(Diff revision 1)
> -        'bouncer_submitter_config': "releases/bouncer_firefox_date.py",
> +        'bouncer_submitter_config': {
> +            "firefox": "releases/dev_bouncer_firefox_beta.py",
> +            # TODO - fennec
> +            "fennec": "",
> +        },
> +        # TODO - is this used? seems like an invalid value and bouncer_submitter.py doesn't

Feel free to kill it.

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8733191 [details]
MozReview Request: Bug 1258059 - bbotcustom - generify dev configs and env for relpro by removing date strings and allow multiple products, r?rail

https://reviewboard.mozilla.org/r/41631/#review38041

Comment 8

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - bbot-cfgs - use production bouncer_firefox_beta.py in staging, r?rail

Review commit: https://reviewboard.mozilla.org/r/41773/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41773/

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8733437 [details]
MozReview Request: Bug 1258059 - bbot-cfgs - use production bouncer_firefox_beta.py in staging, r?rail

https://reviewboard.mozilla.org/r/41773/#review38157

Comment 10

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - gecko - stub installers with beta-dev, add staging beta single locale, r?rail

Review commit: https://reviewboard.mozilla.org/r/41775/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41775/

Comment 11

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - bbot-cfgs - use custom beta staging single locale config, r?rail

Review commit: https://reviewboard.mozilla.org/r/41777/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41777/

Comment 12

[Jordan Lund (:jlund)]

Attachment: MozReview Request: Bug 1258059 - bbotcustom - allow custom branch single locale config, r?rail

Review commit: https://reviewboard.mozilla.org/r/41779/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41779/

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8733449 [details]
MozReview Request: Bug 1258059 - gecko - stub installers with beta-dev, add staging beta single locale, r?rail

https://reviewboard.mozilla.org/r/41775/#review38169

::: testing/mozharness/configs/single_locale/dev-mozilla-beta.py:11
(Diff revision 1)
> +    # l10n
> +    "hg_l10n_base": "https://hg.mozilla.org/releases/l10n/mozilla-beta",
> +
> +    # mar
> +    # TODO: use different location
> +    "mar_tools_url": "http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/mar-tools/%(platform)s",

Can you kill `mar_tools_url`?

mar_tools_url is passed via props: https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/firefox/l10n.yml.tmpl#L33 and handled by https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#271

::: testing/mozharness/configs/single_locale/dev-mozilla-beta.py:12
(Diff revision 1)
> +    "hg_l10n_base": "https://hg.mozilla.org/releases/l10n/mozilla-beta",
> +
> +    # mar
> +    # TODO: use different location
> +    "mar_tools_url": "http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/mar-tools/%(platform)s",
> +    "latest_mar_dir": "fake_kill_me",

Probably we should kill this in a separate bug...

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 8733451 [details]
MozReview Request: Bug 1258059 - bbotcustom - allow custom branch single locale config, r?rail

https://reviewboard.mozilla.org/r/41779/#review38171

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 8733450 [details]
MozReview Request: Bug 1258059 - bbot-cfgs - use custom beta staging single locale config, r?rail

https://reviewboard.mozilla.org/r/41777/#review38173

::: mozilla/project_branches.py:153
(Diff revision 1)
>          'build_tools_repo_path': 'users/raliiev_mozilla.com/tools',
>          'partners_repo_path': 'build/partner-repacks',
>          'partner_repack_platforms': ('linux', 'linux64', 'win32', 'win64', 'macosx64'),
>          "release_platforms": ("linux", "linux64", "win32", "win64", "macosx64"),
>          "l10n_release_platforms": ("linux", "linux64", "win32", "win64", "macosx64"),
> +        "single_locale_branch_config": "dev-mozilla-beta.py",

the script uses `"single_locale/%s.py"`, make sure to remove `.py` when you land.

Comment 16

[Jordan Lund (:jlund)]

> ::: testing/mozharness/configs/single_locale/dev-mozilla-beta.py:12
> (Diff revision 1)
> > +    "hg_l10n_base": "https://hg.mozilla.org/releases/l10n/mozilla-beta",
> > +
> > +    # mar
> > +    # TODO: use different location
> > +    "mar_tools_url": "http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/mar-tools/%(platform)s",
> > +    "latest_mar_dir": "fake_kill_me",
> 
> Probably we should kill this in a separate bug...

filed 1258798

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5440170659bb

Comment 18

[Jordan Lund (:jlund)]

(In reply to Pulsebot from comment #17)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/5440170659bb

uplifted -> esr45

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5440170659bb

Comment 20

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1258059 - beta-dev should be treated as beta for RCs r=jlund

Review commit: https://reviewboard.mozilla.org/r/45559/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/45559/

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8740096 [details]
MozReview Request: Bug 1258059 - beta-dev should be treated as beta for RCs r=jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/45559/diff/1-2/

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8740096 [details]
MozReview Request: Bug 1258059 - beta-dev should be treated as beta for RCs r=jlund

it worked fine on jamun:

    "beta-dev-cdntest": {
      "partials": {
        "Firefox-46.0b9-build1": "http://download.mozilla.org/?product=firefox-46.0build9-partial-46.0b9build1&os=%OS_BOUNCER%&lang=%LOCALE%",
        "Firefox-44.0-build3": "http://download.mozilla.org/?product=firefox-46.0build9-partial-44.0build3&os=%OS_BOUNCER%&lang=%LOCALE%",
        "Firefox-45.0-build2": "http://download.mozilla.org/?product=firefox-46.0build9-partial-45.0build2&os=%OS_BOUNCER%&lang=%LOCALE%"
      },
      "completes": {
        "*": "http://download.mozilla.org/?product=firefox-46.0build9-complete&os=%OS_BOUNCER%&lang=%LOCALE%"
      }
    },

Comment 23

[Jordan Lund (:jlund)]

Comment on attachment 8740096 [details]
MozReview Request: Bug 1258059 - beta-dev should be treated as beta for RCs r=jlund

https://reviewboard.mozilla.org/r/45559/#review42155

just when I think I'm starting to get familiar :)

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8740096 [details]
MozReview Request: Bug 1258059 - beta-dev should be treated as beta for RCs r=jlund

https://hg.mozilla.org/build/tools/rev/57158b4191c6

Comment 25

[Jordan Lund (:jlund)]

I guess this can be resolved for now

Comment 26

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.