758149 - create-release-repacks.py (l10n repacks) should get mozconfigs from source, not from buildbot-configs

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Mark Banner (:standard8)]

Currently create-release-repacks.py is obtaining the mozconfig to use for l10n repacks from buildbot-configs, rather than from the source tree.

Comment 1

[Mark Banner (:standard8)]

http://hg.mozilla.org/build/tools/annotate/88cf0d0c3f1d/scripts/l10n/create-release-repacks.py#l184

Comment 2

[[e-mail] (:bhearsum)]

We should add these to the source repositories, create a list in the release configs (like https://github.com/mozilla/buildbot-configs/blob/master/mozilla/release-firefox-mozilla-beta.py#L107), and then change the logic to find it based on the location in the release config. Should be pretty easy, not high priority though.

Comment 3

[Mark Banner (:standard8)]

Attachment: Tools fix v1

This patch updates the tools scripts which is where the repacks come from. Specifically wrt mozconfigs it:

- Looks for a new key in releaseConfig called 'l10n-mozconfigs'
- If that is found, then it will attempt to use the source mozconfig specified, falling back to the buildbot-configs one if that is not found
- If the 'l10n-mozconfigs' key isn't present, it will just fall back to the old method of getting mozconfigs from buildbot-configs.

It also adds a 'cat .mozconfig' step so the result can be identified.

I've tested this locally as far as the copying of the mozconfig with a couple of extra patches that I'll attach in a moment, and it seemed to work fine.

Not sure if you guys will want a staging run on it though.

I thought we could start rolling this out with the next Thunderbird beta in a couple of weeks time, and then move onto other products/channels later.

Comment 4

[Mark Banner (:standard8)]

Attachment: buildbot-configs - Thunderbird comm-beta

This patch tells the comm-beta release scripts to use the in source tree version, and removes the old buildbot-configs mozconfigs for comm-beta.

I've also verified according to the last comm-beta release that the 'release' mozconfigs in buildbot-configs are no longer required afaict.

Comment 5

[Mark Banner (:standard8)]

Attachment: Add configs to comm-central and co

This is the patch that adds the mozconfigs to comm-beta. I think the client.py-args changes will work, but I'm just pushed to try to verify (https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=1f5f9f1c992e).

When I land this on branches, I'll make sure to update our scripts to update the l10n path name (for now, I believe there's a separate bug on that somewhere).

Comment 6

[Mark Banner (:standard8)]

Comment on attachment 730168 [details] [diff] [review]
Add configs to comm-central and co

This doesn't quite work on Windows wrt the client.py args. I'll think about that tomorrow.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 730161 [details] [diff] [review]
Tools fix v1

Review of attachment 730161 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/build/l10n.py
@@ +56,5 @@
> +                    cp %(src_mozconfig)s .mozconfig;
> +                else
> +                    echo Downloading mozconfig;
> +                    cp ../%(hg_mozconfig)s .mozconfig;
> +                fi'''.replace("\n", "") % {'src_mozconfig': srcMozconfigPath, 'hg_mozconfig': mozconfigPath }]

Is there any reason this bash to be used? We do it in Buildbot because there's no alternative, but this should be translated to Python if possible. You should be able to get rid of the extra fallback in the "else" block if you do.

Can you file a bug to kill the fallback logic too? We should keep it until we ship a final release/esr with the new mozconfigs (in case we need to chemspill from a relbranch), but I don't want to support it longer than we have to.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 730164 [details] [diff] [review]
buildbot-configs - Thunderbird comm-beta

Review of attachment 730164 [details] [diff] [review]:
-----------------------------------------------------------------

Rolling out on Thunderbird first seems like a good idea as long as that beta is happening in the first half of the cycle.

Comment 9

[Mark Banner (:standard8)]

Attachment: Add configs to comm-central and co v2

I just remembered that client.py picks up MOZILLA_REV and COMM_REV from the environment and buildbot already sets those for l10n builds.

I'd forgotten it with the previous version of this patch, so this will work now.

Comment 10

[Mark Banner (:standard8)]

Attachment: Tools fix v2

Fixes the review comments.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 752172 [details] [diff] [review]
Tools fix v2

Review of attachment 752172 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/l10n/create-release-repacks.py
@@ +239,4 @@
>          l10nBaseRepo=make_hg_url(branchConfig["hghost"],
>                                   releaseConfig["l10nRepoPath"]),
>          mozconfigPath=mozconfig,
> +        srcMozconfigPath=releaseConfig.get('l10n-mozconfigs', {}).get(options.platform),

Can you change this to l10n_mozconfigs on landing, for consistency?

Sorry I didn't get to this before beta 1!

Comment 12

[Mark Banner (:standard8)]

Comment on attachment 752169 [details] [diff] [review]
Add configs to comm-central and co v2

https://hg.mozilla.org/comm-central/rev/5f77a76ab929
https://hg.mozilla.org/releases/comm-aurora/rev/98015487a17c
https://hg.mozilla.org/releases/comm-beta/rev/be6f1766030f

Comment 13

[Mark Banner (:standard8)]

Comment on attachment 752172 [details] [diff] [review]
Tools fix v2

Landed this section:

https://hg.mozilla.org/build/tools/rev/4405017a713d

Comment 14

[Mark Banner (:standard8)]

Comment on attachment 730164 [details] [diff] [review]
buildbot-configs - Thunderbird comm-beta

https://hg.mozilla.org/build/buildbot-configs/rev/94098aa58ab8

Comment 15

[Armen [:armenzg]]

Live in production.

Comment 16

[Aki Sasaki (not active)]

Looks like we need to update the source factory as well?

cp: cannot stat `configs/mozilla2/linux/comm-beta/release/mozconfig': No such file or directory

http://buildbot-master64.srv.releng.usw2.mozilla.com:8001/builders/release-comm-beta-thunderbird_source/builds/0

Comment 17

[Mark Banner (:standard8)]

Attachment: Temporary reland Linux mozconfig for source builds

Rather than try to fix the source issue straight away, lets play it safe and put the mozconfig back that it wants. We can then follow-up later with a full fix for the source factories.

Comment 18

[Aki Sasaki (not active)]

Comment on attachment 730164 [details] [diff] [review]
buildbot-configs - Thunderbird comm-beta

https://hg.mozilla.org/build/buildbot-configs/rev/7f72e2475137

Comment 19

[Mark Banner (:standard8)]

Bug 898099 has fixed some of this for all apps, but there's some tidy up that can be done once we've completed shipping with that in place.

Comment 20

[Mark Banner (:standard8)]

Not working on this atm

Comment 21

[[e-mail] (:bhearsum)]

This script is dying in favour of the new Mozharness one.