Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond)

RESOLVED INCOMPLETE

Status

Release Engineering
Release Automation
P3
normal
RESOLVED INCOMPLETE
3 years ago
a year ago

People

(Reporter: blassey, Assigned: kmoir)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox30-, firefox31+ affected, fennec+)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/649] )

Attachments

(23 attachments, 12 obsolete attachments)

1.12 KB, patch
bhearsum
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
14.58 KB, patch
rail
: review+
bhearsum
: feedback+
Details | Diff | Splinter Review
14.41 KB, patch
rail
: review+
Details | Diff | Splinter Review
14.54 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
9.30 KB, patch
rail
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
10.80 KB, patch
rail
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
14.78 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
1.29 KB, patch
rail
: review+
Details | Diff | Splinter Review
2.77 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
531 bytes, patch
kmoir
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
4.09 KB, patch
nthomas
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
485 bytes, patch
Details | Diff | Splinter Review
1.22 KB, patch
nthomas
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
470 bytes, patch
rail
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
17.99 KB, patch
rail
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
1.58 KB, text/plain
Details
1.38 KB, patch
nthomas
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
6.30 KB, patch
nthomas
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
3.87 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
11.81 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
846 bytes, patch
rail
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
2.32 KB, patch
bhearsum
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
2.35 KB, patch
nthomas
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
As I understand it, we already have this for nightly and aurora. This has been motivated by Android's lack of support for 3 char locale codes and our desire to ship support for Maithili.
Blocks: 987642
OS: Mac OS X → All
Hardware: x86 → All
Summary: update channels for single local Beta and Release builds of Firefox for Android 30 (and beyond) → Update channels for single local Beta and Release builds of Firefox for Android 30 (and beyond)
This is a big project, I doubt this is do-able before we ship. We don't even have another Beta test it with at this point...
Component: Releases → Release Automation
QA Contact: rail → bhearsum

Comment 2

3 years ago
Ben: agreed, it's not without peril on our side.  

In the mobile mtg, I suggested testing on beta first and then going to release, given that we're already in the release process for Fx30 next week. Ben also suggested that beta 1 may not be the best time to turn this on, because there is sometimes random bustage during the first beta cycle after uplift.

I'm going to assume we'd use balrog for this rather than snippets, but call me out if I'm wrong.
As this requires more effort from releng than we have time for between now and next Tuesday's ship date - this isn't a viable FF30 target.  Not even clear that this work needs to be tracked to a particular release, and with Nightly builds now having locale picker built in - do we still need to do this for Beta/Release channels? What's the win there?
tracking-firefox30: ? → -

Comment 4

3 years ago
(In reply to Lukas Blakk [:lsblakk] from comment #3)
> As this requires more effort from releng than we have time for between now
> and next Tuesday's ship date - this isn't a viable FF30 target.  Not even
> clear that this work needs to be tracked to a particular release, and with
> Nightly builds now having locale picker built in - do we still need to do
> this for Beta/Release channels? What's the win there?

blassey: after some more digging, it turns out we're *not* already serving single locale updates on nightly and Aurora, although we thought we were. Sorry for the misinformation, but that makes this a bigger ask.

We discussed this in our weekly post-mortem today and echoed Lukas' questions: is there a use case for single locale updates that will extend beyond when the locale picker hits the Release channel in 13 weeks? Are there benefits to having this in place in the intervening time, enabling a new locale+community notwithstanding?
Flags: needinfo?(blassey.bugs)
We have an indic locale (Maithili in bug 1019726) that won't work with the current Android multi-locale build (and the locale picker). So we were hoping to give the localizers a show of support by creating a single locale Maithili build that could be installed directly from a Web or FTP listing. That build would not be updateable using the Play Store, so we wanted to get Mozilla-based updates working.
(In reply to Lukas Blakk [:lsblakk] from comment #3)
> As this requires more effort from releng than we have time for between now
> and next Tuesday's ship date - this isn't a viable FF30 target.  Not even
> clear that this work needs to be tracked to a particular release, and with
> Nightly builds now having locale picker built in - do we still need to do
> this for Beta/Release channels? What's the win there?

More specifically, the Maithili community is planning on pointing users to single locale builds of Fx30 which will not have any update mechanism, hence the request to track for 30.

Beyond the Maithili community, non-play-store updates will allow us to address non-play-store devices like the Kindle Fires and phones in China.
Flags: needinfo?(blassey.bugs) → needinfo?(lsblakk)
Let's bring Jeff into the conversation here too so we can ensure that the locale(s) we would do this work for are progressing along the channels as per l10n community standards and process.  

A few more questions:

* We have bug 713777 for looking at trying the Amazon app store again (I am currently trying to chase down John O'Duinn for the Mozilla account credentials there in order to progress the investigation)
* Are there other 3 char locales we would need to offer this for in the near future?
* Is there no way to have this locale turned into a 2 char locale name and fit it into the multi-locale build?

Releng (aka Coop) - is there any way to provide an update path for the build (ie: URL) even if that URL is not live *yet* so that the builds on FTP can, at some point a little further down the road, get updates?
Flags: needinfo?(lsblakk)
Flags: needinfo?(jbeatty)
Flags: needinfo?(coop)
On behalf of coop - I took the 29.0.1 es-ES apk and it has the updater enabled, and queries for updates like this
 https://aus4.mozilla.org/update/4/Fennec/29.0.1/20140505221201/Android_arm-eabi-gcc3/es-ES/release/4.3/default/default/29.0.1/update.xml
so it's using a sensible locale. The multi and en-US apks query the same except s/es-ES/en-US/.

So if we had a Maithili build (not currently enabled in beta or release) then we should be able to serve an update to it in the future.

Comment 9

3 years ago
Thanks, Nick.
Flags: needinfo?(coop)
(In reply to Lukas Blakk [:lsblakk] from comment #7)
> Let's bring Jeff into the conversation here too so we can ensure that the
> locale(s) we would do this work for are progressing along the channels as
> per l10n community standards and process.  
> 
> A few more questions:
> 
> * We have bug 713777 for looking at trying the Amazon app store again (I am
> currently trying to chase down John O'Duinn for the Mozilla account
> credentials there in order to progress the investigation)
> * Are there other 3 char locales we would need to offer this for in the near
> future?

There are only two (hsb and dsb), but since bug fix to support ISO 639-2 codes will arrive in Android developer tools in the near future (see bug 1013226 for reference). I'm convinced the best approach is to accomodate these three locales (Maithili) and keep the gate down for other three char locales until the fix is landed.

> * Is there no way to have this locale turned into a 2 char locale name and
> fit it into the multi-locale build?
It's a short term solution that requires significant testing (especially with the inclusion of the locale picker feature) and won't scale once Google's fix lands and we have to fix the locale code remap. The timeframe is too short and the workload too high. I can forward the email discussion between rnewman, nalexander, and I on the topic, if you'd like.
> 
> Releng (aka Coop) - is there any way to provide an update path for the build
> (ie: URL) even if that URL is not live *yet* so that the builds on FTP can,
> at some point a little further down the road, get updates?
Flags: needinfo?(jbeatty)
Sounds like, barring a dot release, the users of the Maithili locale could feasibly download their builds from FTP and we have up to 6 weeks to provide updates for them.

Coop/Nick - can you provide a time estimation for getting single locale updates (esp. if only for up to 3 locales?) on Beta/Release?

Jeff - is Maithili ready to be enabled in Beta/Release for FF30?  If it's not already enabled, perhaps we are better to wait one more cycle and have the update mechanics lined up first?
(In reply to Lukas Blakk [:lsblakk] from comment #11)
> Sounds like, barring a dot release, the users of the Maithili locale could
> feasibly download their builds from FTP and we have up to 6 weeks to provide
> updates for them.
> 
> Coop/Nick - can you provide a time estimation for getting single locale
> updates (esp. if only for up to 3 locales?) on Beta/Release?
> 
> Jeff - is Maithili ready to be enabled in Beta/Release for FF30?  If it's
> not already enabled, perhaps we are better to wait one more cycle and have
> the update mechanics lined up first?
No, Maithili is targeted for FF31 and is ready for FF31.
status-firefox31: --- → affected
tracking-firefox31: --- → +
Kim is going to pick this up, but likely won't be able to start on it for 2 weeks (July). 

She will sync up with bhearsum before he goes on PTO to get the lay of the land.
Assignee: nobody → kmoir
Priority: -- → P3
bug 1019919 is going to be one of the main parts of this, meant to link it up earlier.
Depends on: 1019919
Summary: Update channels for single local Beta and Release builds of Firefox for Android 30 (and beyond) → Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond)
Kim, Ben, Are you still targeting 31? FYI, we release 31 in 3 weeks.
Flags: needinfo?(kmoir)
Flags: needinfo?(bhearsum)
(Assignee)

Comment 16

3 years ago
I haven't had a chance to look at this yet but it is a Q3 goal for me. I plan to look at this bug next week, am dealing with capacity issues right now on our CI farm (bug 1034055)
Flags: needinfo?(kmoir)
I'm just back from PTO. It sounds like Kim is still planning to work on this, and I have other priorities. I'll be supporting her from the sidelines and probably doing reviews, but I'm not the one actively driving this.
Flags: needinfo?(bhearsum)
(Assignee)

Updated

3 years ago
Depends on: 1040205
Kim, any update?
Flags: needinfo?(kmoir)
(Assignee)

Comment 19

3 years ago
The past week and a half I have been working on bug 1034055 (move all 2.3 tests to AWS).  This morning I wrote a patch and tested bug 1030753 (enable more tests on Android debug). Now I am looking at this bug :-)
Flags: needinfo?(kmoir)
(Assignee)

Updated

3 years ago
Depends on: 1042755
(Assignee)

Comment 20

3 years ago
I have patches to enable single locale updates to balrog for m-c and m-a in bug 1019919.  However, I don't see the mai locale enabled anymore here

http://mxr.mozilla.org/mozilla-aurora/source/mobile/android/locales/maemo-locales
http://mxr.mozilla.org/mozilla-beta/source/mobile/android/locales/maemo-locales 

which I understand is how they are enabled.  My single locale builds thus don't include the mai locale to enable Maithili.  Is this plan to be enabled soon or should I open a bug?
`mai` cannot appear in maemo-locales: it breaks the build.

See Bug 1013226 Comment 19.


(If it no longer breaks the build -- I haven't tested since May -- then we have much less urgency for single-locale Maithili builds. But I suspect that breakage is still the case.)
When we get single locale builds working, I believe we'll be pulling the locale list from "all-locales" and not "maemo-locales", which is really used to drive the multi-locale build.

This is kinda orthogonal to your question about the "mai" locale though, which might still break builds.
(Assignee)

Comment 23

3 years ago
Have made good progress on this last night, they only thing that is missing is that the way the balrog credentials are consumed on release channels is different so I have to fix that up. 

Also :bhearsum do I have to add an explicit rule to balrog for each android beta and release, I notice that they are there for firefox
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #23)
> Have made good progress on this last night, they only thing that is missing
> is that the way the balrog credentials are consumed on release channels is
> different so I have to fix that up. 
> 
> Also :bhearsum do I have to add an explicit rule to balrog for each android
> beta and release, I notice that they are there for firefox

Yeah, you'll need rules for Fennec there. There's a few other wrinkles here, too:
1) We need the fileUrls to point at either Bouncer or the "releases" directory on FTP. Eg: http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/32.0b10/. There's no guarantees about how long candidates directories will live, so we can't point updates there. If we want to be consistent with Firefox, Bouncer entries are what we should use. There's also extra work there because we'll need to add bouncer entry automation for Fennec. I think using direct links to FTP is OK, at least for now.

We actually handle Balrog somewhat differently for Releases and Nightlies. For releases, we submit the individual file information as part of each build, but then we submit a bunch of generic information later on. For example, here's a release blob:
https://aus4-admin.mozilla.org/releases/Firefox-32.0-build1/data

You'll note that the fileUrls have moved to the top level of the blob, and have platform/locale substitutions in them. Those are submitted as part of the "updates" builder, which we don't have for Fennec.

The scripts that do these submissions are https://github.com/mozilla/build-tools/blob/master/scripts/updates/balrog-release-pusher.py and https://github.com/mozilla/build-tools/blob/master/scripts/updates/balrog-submitter.py. They would work for Fennec with a few tweaks, I think.

2) We should probably talk to RelMan and/or QA about whether or not we want test channel rules, too. For Firefox, we use test channels to make sure updates work before we push them live.

3) Because we'll be pointing at the "releases" directory, we need to push Fennec files there before we ship. Currently, we push after QA signs off. RelMan and QA should be made aware of this.
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum [:bhearsum] from comment #24)
> (In reply to Kim Moir [:kmoir] from comment #23)
> 2) We should probably talk to RelMan and/or QA about whether or not we want
> test channel rules, too. For Firefox, we use test channels to make sure
> updates work before we push them live.

We can forget about this point. Nick told me yesterday that there's no way change the channel of a Fennec build, so there's no way to run tests on a test channel before shipping.
tracking-fennec: 30+ → ?
tracking-fennec: ? → +
(Assignee)

Comment 26

3 years ago
Created attachment 8487896 [details] [diff] [review]
bug1019724bbcustom.patch
(Assignee)

Comment 27

3 years ago
Created attachment 8487913 [details] [diff] [review]
bug1019724buildbotc.patch

WIP patches.  Not sure of the correct values for testChannelRuleIds and testChannels
(Assignee)

Comment 28

3 years ago
Created attachment 8487940 [details] [diff] [review]
bug1019724mh.patch

this isn't 100% complete but I thought I'd get feedback so far
One thing that is that I couldn't figure out a way to use the existing credentials file in configs/balrog so that's why they are added to the configs explicitly.  The way the releases are constructed are very different from the regular builds
Attachment #8487940 - Flags: feedback?(bhearsum)
(Assignee)

Updated

3 years ago
Attachment #8487913 - Flags: feedback?(bhearsum)
(Assignee)

Comment 29

3 years ago
Comment on attachment 8487896 [details] [diff] [review]
bug1019724bbcustom.patch

I think this could go in at any time
Attachment #8487896 - Flags: review?(bhearsum)
(Assignee)

Comment 30

3 years ago
Forgot link, this is the data in balrog for a staging beta release https://aus4-admin-dev.allizom.org/releases/Fennec-32.0b8-build1/data
Comment on attachment 8487896 [details] [diff] [review]
bug1019724bbcustom.patch

Review of attachment 8487896 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, this should be able to land anytime.
Attachment #8487896 - Flags: review?(bhearsum) → review+
Comment on attachment 8487913 [details] [diff] [review]
bug1019724buildbotc.patch

Review of attachment 8487913 [details] [diff] [review]:
-----------------------------------------------------------------

It looks like these patches cover the data submission part of it. There's a few tweaks, but they look good overall.

I think you still need to take care of the bouncer part. Lucky for you, Rail is already taking care of the pushing to the releases directory automatically as part of https://bugzilla.mozilla.org/show_bug.cgi?id=543109!

::: mozilla/release-fennec-mozilla-beta.py
@@ +142,5 @@
>  releaseConfig['enablePartialMarsAtBuildTime'] = False
>  releaseConfig['autoGenerateChecksums'] = False
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']

This is about to change, so it would be best to use the new style here. We'll just be using a single test channel for beta going forward, "beta-cdntest".

@@ +143,5 @@
>  releaseConfig['autoGenerateChecksums'] = False
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']
> +releaseConfig['partialUpdates']      = {}

Does this need to be set? I don't think it does...

@@ +145,5 @@
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']
> +releaseConfig['partialUpdates']      = {}
> +releaseConfig['bouncerServer']       = 'download.mozilla.org'
> +releaseConfig['testChannelRuleIds']    = []
\ No newline at end of file

Ping me and I'll help you figure out the rule ids.

::: mozilla/release-fennec-mozilla-release.py
@@ +151,5 @@
>  releaseConfig['enablePartialMarsAtBuildTime'] = False
>  releaseConfig['autoGenerateChecksums'] = False
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('android', 'android-x86', 'linux')
> +releaseConfig['testChannels']          = ['releasetest', 'betatest']

You should change these channel names too. For releases, they are "release-localtest" and "release-cdntest". (https://bugzilla.mozilla.org/show_bug.cgi?id=986990 talks about this in more detail if you're curious.)
Attachment #8487913 - Flags: feedback?(bhearsum) → feedback+
Comment on attachment 8487940 [details] [diff] [review]
bug1019724mh.patch

Review of attachment 8487940 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/single_locale/release_mozilla-beta_android.py
@@ +24,5 @@
> +    "balrog_credentials_file": "oauth.txt",
> +    "balrog_api_root": "https://aus4-admin.mozilla.org",
> +    'balrog_usernames': {
> +            'Fennec': 'ffxbld',
> +     },

To use the existing configs you'll need to pass them along to mobile_l10n.py in https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.py#L814. It should be very similar to what you did in https://github.com/mozilla/build-buildbotcustom/commit/adfc9ff7134b74551e88e37354b26b28d98a394e, except "branchConfig" is what will be holding mozharness_configs.

@@ +25,5 @@
> +    "balrog_api_root": "https://aus4-admin.mozilla.org",
> +    'balrog_usernames': {
> +            'Fennec': 'ffxbld',
> +     },
> +    "tools_repo": "https://hg.mozilla.org/build/tools",

tools_repo still belongs here though, I think.

::: scripts/mobile_l10n.py
@@ +569,5 @@
> +     
> +    def query_is_release(self):
> +        
> +        if str(self.buildbot_config['properties']['branch']).startswith("release"):
> +            return True

Hmm, does this work? I guess it may if branch is relative to the root of hg.mozilla.org....

This may also be true if we do nightly l10n jobs on this branch (I have no idea if we do). It might be better to have an explicit flag for this in the mozharness configs rather than make this assumption.

@@ +612,5 @@
> +            else:
> +                self.submit_balrog_updates(release_type="release")
> +                product = self.buildbot_config["properties"]["product"]
> +                cmd = [os.path.join(dirs['abs_tools_dir'], "scripts/updates/balrog-release-pusher.py")]
> +                self.chmod(cmd[0], 0764)

Why do you need to chmod here?

@@ +621,5 @@
> +                cmd.extend(["--credentials-file", self.config.get("balrog_credentials_file")])
> +                cmd.extend(["--username", self.config.get("balrog_usernames")[product]])
> +                env = {'PATH': os.environ.get('PATH')}
> +                status = self.run_command(cmd, dirs['base_work_dir'], env=env, success_codes=(0,255), 
> +                                 return_type="num_errors")

All of this balrog-release-pusher.py stuff should go in BalrogMixin instead of here. You can either explicitly call it in a separate function, or do it as part of submit_balrog_updates if query_is_release() is True - I don't have a preference either way.

@@ +622,5 @@
> +                cmd.extend(["--username", self.config.get("balrog_usernames")[product]])
> +                env = {'PATH': os.environ.get('PATH')}
> +                status = self.run_command(cmd, dirs['base_work_dir'], env=env, success_codes=(0,255), 
> +                                 return_type="num_errors")
> +                if status not in [0, 255]:

Is 255 really a valid exit code?
Attachment #8487940 - Flags: feedback?(bhearsum) → feedback+
(Assignee)

Updated

3 years ago
Attachment #8487896 - Flags: checked-in+
(Assignee)

Comment 34

3 years ago
Created attachment 8494016 [details] [diff] [review]
bug1019724bbcustom-2.patch
Attachment #8494016 - Flags: review?(bhearsum)
(Assignee)

Comment 35

3 years ago
Created attachment 8494026 [details] [diff] [review]
bug1019724mh-2.patch

reworked to address the previous feedback 

I had to chmod the script balrog-release-pusher.py or it wouldn't run because it didn't have x permissions.

For invoking scripts/bouncer_submitter.py, I tried a bunch of different approaches that aren't working right now so that part is yet to come.  Should 
scripts/bouncer_submitter.py just be called in buildbotcustom like scripts/mobile_l10n.py is invoked.  Because calling bouncer_submitter.py from within mobile_l10n.py seems problematic.
Attachment #8494026 - Flags: feedback?(bhearsum)
Comment on attachment 8494026 [details] [diff] [review]
bug1019724mh-2.patch

Review of attachment 8494026 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/balrog/production.py
@@ +4,5 @@
>          'b2g': 'b2gbld',
>          'firefox': 'ffxbld',
>          'thunderbird': 'tbirdbld',
>          'mobile': 'ffxbld',
> +        'Fennec': 'ffxbld',

I'm a bit surprised this is necessary, but it's fine.

::: configs/releases/bouncer_fennec_beta.py
@@ +2,5 @@
> +config = {
> +    "shipped-locales-url": "https://hg.mozilla.org/%(repo)s/raw-file/%(revision)s/mobile/android/locales/all-locales",    
> +    "products": {
> +        "installer": {
> +            "product-name": "Firefox-%(version)s",

Rail tells me that the bouncer submitter code doesn't have any checks to make sure a product already exists before trying to create it, so this will fail right now because Firefox uses the same product. You can either update the Bouncer submitter code to deal with it (https://github.com/mozilla/build-mozharness/blob/master/scripts/bouncer_submitter.py) or use Fennec in the product name. Either is fine IMO.

::: scripts/mobile_l10n.py
@@ +605,5 @@
> +            if self.query_is_nightly():  
> +                self.submit_balrog_updates()                
> +            else:
> +                self.submit_balrog_updates(release_type="release")
> +                self.submit_balrog_release_pusher(dirs)               

I'm not going to r- for it, but it would be better to only have the release pusher run once, after all repacks have completed. Even if you continue to run this here, you'll need a builder like the "updates" one we have for Desktop to announce the updates to the release-drivers list. That is likely going to have to happen in this block of release.py: https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.py#L1150

The simplest thing to do might be to use makeDummyBuilder if the product is Fennec, but that will require some testing. I think the "update shipping" builder will work fine as is, but that probably needs testing too.

(Apologies for not catching this earlier...)
Attachment #8494026 - Flags: feedback?(bhearsum) → feedback+
Attachment #8494016 - Flags: review?(bhearsum)
(Assignee)

Comment 37

3 years ago
bhearsum:

I fixed the issues issues you commented on re my earlier patches. thanks :-)  Regarding, 
http://hg.mozilla.org/build/buildbotcustom/file/a12657766d1d/process/release.py#l1150
I note that 

This seems to apply to Linux only
pf = branchConfig['platforms']['linux']

and fennec doesn't currently have verifyConfigs enabled.  

For firefox they are like this
releaseConfig['verifyConfigs']       = {
    'linux':  'mozBeta-firefox-linux.cfg',
    'linux64':  'mozBeta-firefox-linux64.cfg',
    'macosx64': 'mozBeta-firefox-mac64.cfg',
    'win32':  'mozBeta-firefox-win32.cfg'
}
which refers to, for instance tools/release/updates/mozBeta-firefox-linux.cfg

Looking at the hg history for the above file, it looks like it gets updated automatically. So not sure if I should create the blank files for Fennec.

Is the sole purpose of this builder to announce changes to the release drivers?

I looked for an example of update_shipping on our existing masters but couldn't find a job that had run recently
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #37)
> bhearsum:
> 
> I fixed the issues issues you commented on re my earlier patches. thanks :-)
> Regarding, 
> http://hg.mozilla.org/build/buildbotcustom/file/a12657766d1d/process/release.
> py#l1150
> I note that 
> 
> This seems to apply to Linux only
> pf = branchConfig['platforms']['linux']
> 
> and fennec doesn't currently have verifyConfigs enabled.  
> 
> For firefox they are like this
> releaseConfig['verifyConfigs']       = {
>     'linux':  'mozBeta-firefox-linux.cfg',
>     'linux64':  'mozBeta-firefox-linux64.cfg',
>     'macosx64': 'mozBeta-firefox-mac64.cfg',
>     'win32':  'mozBeta-firefox-win32.cfg'
> }
> which refers to, for instance tools/release/updates/mozBeta-firefox-linux.cfg
> 
> Looking at the hg history for the above file, it looks like it gets updated
> automatically. So not sure if I should create the blank files for Fennec.

update verify isn't something we can run for Fennec, so you definitely don't need update verify configs.

> Is the sole purpose of this builder to announce changes to the release
> drivers?

For Desktop, the updates builder is responsible for running balrog-release-pusher.py as well as creating old style aus2/3 snippets (which is why it's not appropriate to use as-is for this). When it completes, Buildbot sends mail to announce the availability of the updates. It's that part that you're still missing, and you have two options for adding:
1) Stop running balrog release pusher as part of the repacks and create a new "updates" builder for Fennec that runs it.
2) Create a dummy "updates" builder that runs for Fennec.

In either case, it should be triggered after all of the repacks complete. You may need to add the productName to the updates builder because Fennec and Firefox will probably try to use the same one. A lot of the scheduler and builder name changes will probably be similar to what Rail did for push to mirrors in bug 543109 (eg: https://bug543109.bugzilla.mozilla.org/attachment.cgi?id=8488341).

The e-mail announcement should "just work" provided the templates in https://github.com/mozilla/build-buildbot-configs/tree/master/mozilla/release_templates are updated to match any changes in builder names and/or new builders.

I hope this helps clear things up a bit, but I know the builder+scheduler organization is quite complicated, so feel free to ping me for any further clarification.

> I looked for an example of update_shipping on our existing masters but
> couldn't find a job that had run recently

It's a bit easier to find these via the candidates directory, eg: http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/33.0b8-candidates/build1/logs/release-mozilla-beta-update_shipping-bm85-build1-build7.txt.gz.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 39

3 years ago
Had a meeting with Ben and Rail regarding this issue this morning, thanks for the info it was very helpful

These are the steps forward I'll take
Create a dummy "updates" builder that runs for Fennec. This is just send out the email that the updates are available.
Add this dummy builder to the scheduler
add product name to it 
builder needs to be downtream of post_deliverables scheduler
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l1799
updates_success should be renamed to updates_firefox_sucess tb fennec etc
or add symlinks in buildbot-configs_kmoir/mozilla/release_templates/updates_success
strip release branch stuff out of updates_success

Bouncer 
disable bouncer is currently in fennec release configs
delete that from configs
will submit to bouncer automagically
product 
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l182
bouncer configs
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l1592
config from mozharness
http://hg.mozilla.org/build/buildbotcustom/file/56830bc4b6a5/process/release.py#l1593
update configs here to include ones for fennec beta and release
http://hg.mozilla.org/build/mozharness/file/670222dfeb47/configs/releases

rail's patches https://bug543109.bugzilla.mozilla.org/attachment.cgi?id=8488341 are an example of how to specify a product when it wasn't there before 
https://bug543109.bugzilla.mozilla.org/attachment.cgi?id=8488341
(Assignee)

Comment 40

3 years ago
Created attachment 8500436 [details] [diff] [review]
bug1019724dummy.patch

Created attachment 8500432 [details] [diff] [review] [diff] [review]
bug1019724dummy.patch

I added this patch my staging env and my dummy build was triggered after all the l10n repacks completed.  I thought the email would magically get sent but I can't see that it happened here

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-beta-android_repack_complete

So I'm not sure what I'm missing, help appreciated
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #40)
> Created attachment 8500436 [details] [diff] [review]
> bug1019724dummy.patch
> 
> Created attachment 8500432 [details] [diff] [review]
> bug1019724dummy.patch
> 
> I added this patch my staging env and my dummy build was triggered after all
> the l10n repacks completed.  I thought the email would magically get sent
> but I can't see that it happened here
> 
> http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-
> beta-android_repack_complete
> 
> So I'm not sure what I'm missing, help appreciated

I think we resolved this on IRC - Kim didn't realize she needed to add herself to EMAIL_RECIPIENTS to get mail.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 42

3 years ago
Created attachment 8501168 [details] [diff] [review]
bug1019724bbcustom-3.patch
Attachment #8494016 - Attachment is obsolete: true
(Assignee)

Updated

3 years ago
Attachment #8500436 - Attachment is obsolete: true
(Assignee)

Comment 43

3 years ago
Created attachment 8501258 [details] [diff] [review]
bug1019724mh-3.patch
Attachment #8487940 - Attachment is obsolete: true
Attachment #8494026 - Attachment is obsolete: true
(Assignee)

Comment 44

3 years ago
Created attachment 8501278 [details] [diff] [review]
bug1019724bbconfigs-3.patch

buildbot-configs.  Not sure if the bouncer stuff is quite right, not sure how it works to update the db
(Assignee)

Updated

3 years ago
Attachment #8501168 - Flags: feedback?(bhearsum)
(Assignee)

Updated

3 years ago
Attachment #8501258 - Flags: feedback?(bhearsum)
(Assignee)

Updated

3 years ago
Attachment #8501278 - Flags: feedback?(bhearsum)
Comment on attachment 8501168 [details] [diff] [review]
bug1019724bbcustom-3.patch

Review of attachment 8501168 [details] [diff] [review]:
-----------------------------------------------------------------

This looks right to me...
Attachment #8501168 - Flags: feedback?(bhearsum) → feedback+
Comment on attachment 8501258 [details] [diff] [review]
bug1019724mh-3.patch

Review of attachment 8501258 [details] [diff] [review]:
-----------------------------------------------------------------

This looks sensible to me. Rail, can you give the Bouncer parts a second set of eyes?
Attachment #8501258 - Flags: feedback?(rail)
Attachment #8501258 - Flags: feedback?(bhearsum)
Attachment #8501258 - Flags: feedback+
Comment on attachment 8501278 [details] [diff] [review]
bug1019724bbconfigs-3.patch

Review of attachment 8501278 [details] [diff] [review]:
-----------------------------------------------------------------

This is fine, but I think there's two things missing:
1) "releaseChannel", which also means you need to enable the "update_shipping" builder in release.py (this is the one that lets us actually make the release live when we're ready to release).
2) I think you need to adjust e-mail templates still, otherwise we'll get generic success mail rather than "Updates available on xxxxx".
Attachment #8501278 - Flags: feedback?(bhearsum) → feedback+
Comment on attachment 8501258 [details] [diff] [review]
bug1019724mh-3.patch

Review of attachment 8501258 [details] [diff] [review]:
-----------------------------------------------------------------

In overall the direction is correct. The patch just needs some fixes.

::: configs/releases/bouncer_fennec_beta.py
@@ +1,2 @@
> +# lint_ignore=E501
> +config = {

You can use a single file for both beta and release, they are identical. It's the same for Thunderbird. Firefox has different platforms for beta and release, this is why we use different files for them.

@@ +1,3 @@
> +# lint_ignore=E501
> +config = {
> +    "shipped-locales-url": "https://hg.mozilla.org/%(repo)s/raw-file/%(revision)s/mobile/android/locales/all-locales",    

Hmm, do we use this file to generate the list of locales we ship? If one of the locales is missing on FTP bouncer may mark this product as not ready.

::: mozharness/mozilla/updates/balrog.py
@@ +47,5 @@
> +    def submit_balrog_release_pusher(self, dirs): 
> +        dirs = dirs
> +        product = self.buildbot_config["properties"]["product"]
> +        cmd = [os.path.join(dirs['abs_tools_dir'], "scripts/updates/balrog-release-pusher.py")]
> +        self.chmod(cmd[0], 0764)

I would not change the state of the file.

cmd = [self.query_exe("python"), os.path.join(..., "scripts/updates/balrog-release-pusher.py")]

is how it's used in mozharness usually.

@@ +50,5 @@
> +        cmd = [os.path.join(dirs['abs_tools_dir'], "scripts/updates/balrog-release-pusher.py")]
> +        self.chmod(cmd[0], 0764)
> +        cmd.extend(["--build-properties", os.path.join(dirs["base_work_dir"], "balrog_props.json")])
> +        cmd.extend(["--api-root", self.config.get("balrog_api_root")])
> +        cmd.extend(["--buildbot-configs", "https://hg.mozilla.org/users/kmoir_mozilla.com/buildbot-configs"])

I bet this is going to be read from configs in the future. :)

@@ +54,5 @@
> +        cmd.extend(["--buildbot-configs", "https://hg.mozilla.org/users/kmoir_mozilla.com/buildbot-configs"])
> +        cmd.extend(["--release-config", os.path.join(dirs['build_dir'], self.config.get("release_config_file"))])
> +        cmd.extend(["--credentials-file", os.path.join(dirs['base_work_dir'], self.config.get("balrog_credentials_file"))])
> +        cmd.extend(["--username", self.config.get("balrog_usernames")[product]])
> +        env = {'PATH': os.environ.get('PATH')}

Why you need to set PATH here? I think self.query_exe("python") should make the line above redundant.

@@ +60,5 @@
> +        self.info("Calling Balrog release pusher script")
> +        return_code = self.retry(self.run_command,
> +                                 args=(cmd,),
> +                                 kwargs={'cwd': dirs['abs_work_dir'],
> +                                         'env': env})

If you don't need setting PATH, probably you can remove env from kwargs.

@@ +65,5 @@
> +       
> +        if return_code not in [0]:
> +            self.return_code = 1
> +        else:
> +            self.return_code = 0
\ No newline at end of file

Can you add comments why you needed this magic?

::: scripts/mobile_l10n.py
@@ +568,5 @@
> +    def query_is_release(self):
> +        
> +        c = self.config
> +        if str(self.config['is_release']):
> +            return True

I'd just implement the method body as single liner:

    return bool(self.config.get("is_release"))

using self.config["is_release"] may make the script barf if you don't have this variable set in the config. Also "c" is assigned, but never used.

@@ +574,3 @@
>      def submit_to_balrog(self):
> +        
> +        dirs = self.query_abs_dirs()

You can move the line above somewhere below (closer to the place where it's used) so you don't waste time to call this method if the following "if" doesn't pass.

@@ +574,5 @@
>      def submit_to_balrog(self):
> +        
> +        dirs = self.query_abs_dirs()
> +        if not self.query_is_nightly() and not self.query_is_release():
> +            self.info("Not a nightly or release build, skipping balrog submission.")

Hmmm, "return" is missing here I believe.

@@ +608,5 @@
>              self.set_buildbot_property("isOSUpdate", False)
>              self.set_buildbot_property("buildid", self.query_buildid())
>  
> +            if self.query_is_nightly():  
> +                self.submit_balrog_updates()

Can you explicitly pass release_type="nighly" here, just to make it more readable?

@@ +613,5 @@
> +            else:
> +                self.submit_balrog_updates(release_type="release")
> +                self.submit_balrog_release_pusher(dirs)
> +        if not self.query_is_nightly():
> +            self.submit_balrog_release_pusher(dirs)

It looks like you call self.submit_balrog_release_pusher(dirs) N+1 times for release jobs, where N = number of locales. N times in the "else" block and once here. And it looks like you call it twice for the last locale. It'd be great either to fix the logic or add comments here.
Attachment #8501258 - Flags: feedback?(rail) → feedback-
(Assignee)

Comment 49

3 years ago
I have new patches which I've tested and are green.  However, I want to clean up the balrog db for my staging env to make sure it works and there's the bug where you can't delete them from the ui.

How do we remove a release from the balrog dev instance. I tried to do it as mentioned here but it didn't work for me https://bugzilla.mozilla.org/show_bug.cgi?id=1049550#c6
Do we have to open a bug with some IT or db or do we have access to the databases ourselves?
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #49)
> I have new patches which I've tested and are green.  However, I want to
> clean up the balrog db for my staging env to make sure it works and there's
> the bug where you can't delete them from the ui.
> 
> How do we remove a release from the balrog dev instance. I tried to do it as
> mentioned here but it didn't work for me
> https://bugzilla.mozilla.org/show_bug.cgi?id=1049550#c6
> Do we have to open a bug with some IT or db or do we have access to the
> databases ourselves?

The best right now is to have me do it, unfortunately :(. Kim and coordinated on IRC and I deleted Firefox-32.0b8-build1 from the dev db.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 51

3 years ago
Thanks Ben, rerunning builds now.
(Assignee)

Comment 52

3 years ago
Latest data is here

https://aus4-admin-dev.allizom.org/releases/Fennec-32.0b8-build1/data

Looks okay but not sure if 
"completes": {
      "*": "fennec-32.0b8.complete.mar"
    }
should be there?
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #52)
> Latest data is here
> 
> https://aus4-admin-dev.allizom.org/releases/Fennec-32.0b8-build1/data
> 
> Looks okay but not sure if 
> "completes": {
>       "*": "fennec-32.0b8.complete.mar"
>     }
> should be there?

That looks fine. The difference is that we created a new blob schema time mid way through your work on this bug, I think, and this is one of the changes. URLs like https://aus4-dev.allizom.org/update/4/Fennec/27.0/20140123190227/Android_arm-eabi-gcc3/eu/beta-cdntest/4.2.2/default/default/27.0/update.xml are returning updates, so you certainly don't have malformed data or anything.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 54

3 years ago
Created attachment 8507874 [details] [diff] [review]
bug1019724mh-4.patch

new patch with feedback incorporated. Ran green in staging.
Attachment #8501258 - Attachment is obsolete: true
Attachment #8507874 - Flags: feedback?(rail)
Comment on attachment 8507874 [details] [diff] [review]
bug1019724mh-4.patch

Review of attachment 8507874 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

::: mozharness/mozilla/updates/balrog.py
@@ +44,5 @@
> +        
> +        return return_code
> +        
> +    def submit_balrog_release_pusher(self, dirs): 
> +        dirs = dirs

the line above is redundant
Attachment #8507874 - Flags: feedback?(rail) → feedback+
(Assignee)

Comment 56

3 years ago
Created attachment 8507898 [details] [diff] [review]
bug1019724mh-5.patch
Attachment #8507874 - Attachment is obsolete: true
Attachment #8507898 - Flags: review?(rail)
Comment on attachment 8507898 [details] [diff] [review]
bug1019724mh-5.patch

Ship it!
Attachment #8507898 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Attachment #8501278 - Flags: review?(rail)
(Assignee)

Updated

3 years ago
Attachment #8487913 - Attachment is obsolete: true
(Assignee)

Updated

3 years ago
Attachment #8501168 - Flags: review?(rail)
Comment on attachment 8501168 [details] [diff] [review]
bug1019724bbcustom-3.patch

Review of attachment 8501168 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +576,5 @@
>              ))
>              xr_deliverables_builders.append(builderPrefix('xulrunner_source'))
> +    if releaseConfig['productName'] == "fennec":
> +        builders.append(makeDummyBuilder(
> +            name=builderPrefix('%s_updates' % releaseConfig['productName']),

feel free to use "fennec_updates" here (without interpolation) -- easier to grep ;)

@@ +1820,5 @@
> +                '%s_updates_ready' % releaseConfig['productName']),
> +            branch=sourceRepoInfo['path'],
> +            upstreamBuilders=deliverables_builders,
> +            builderNames=post_deliverables_builders,
> +        ))

You are adding a second aggregating scheduler with the same upstream and downstream builders what means that the downstream builders may be triggered twice. The only difference with the previous scheduler is the name. What's the purpose of this scheduler?
Attachment #8501168 - Flags: review?(rail) → review-
Comment on attachment 8501278 [details] [diff] [review]
bug1019724bbconfigs-3.patch

Review of attachment 8501278 [details] [diff] [review]:
-----------------------------------------------------------------

A nit

::: mozilla/release-fennec-mozilla-beta.py
@@ +145,5 @@
>  releaseConfig['ftpSymlinkName'] = 'latest-beta'
> +releaseConfig['enableAutomaticPushToMirrors'] = True
> +# Tuxedo/Bouncer configuration
> +releaseConfig['tuxedoServerUrl']     = 'https://bounceradmin.mozilla.com/api'
> +releaseConfig['bouncer_submitter_config'] = 'releases/bouncer_fennec_beta.py'

I believe this should be releases/bouncer_fennec.py per attachment 8507898 [details] [diff] [review]. The same applies to other hunks as well.

Conditional r+ if you run something like
sed 's,releases/bouncer_fennec_\(beta\|release\).py,releases/bouncer_fennec.py,g' on those files
Attachment #8501278 - Flags: review?(rail) → review+
(Assignee)

Comment 60

3 years ago
Created attachment 8508209 [details] [diff] [review]
bug1019724bbconfigs-4.patch

patch to address feedback in comment #59
(Assignee)

Comment 61

3 years ago
regarding comment #58, the purpose of this builder is to send an email that updates are available for fennec

release-mozilla-beta-fennec_updates and release-mozilla-release-fennec_updates builders are added for this purpose

Is there better way to do this?
Flags: needinfo?(rail)
Using a dummy builder is the easiest way to email. I still wonder why you need 2 schedulers (not builders) with the same up/down stream builders.
Flags: needinfo?(rail)
(Assignee)

Comment 63

3 years ago
Right that's what I did, it adds a dummy builder like this

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-beta-fennec_updates/builds/0
(Assignee)

Comment 64

3 years ago
Rail:

So this is the diff of the builders added, is this what you would expect?

(build1)[kmoir@dev-master1.srv.releng.scl3.mozilla.com build1]$ diff old new
22a23
> release-mozilla-release-fennec_updates DummyFactory
36a38
> release-mozilla-beta-fennec_updates DummyFactory
Flags: needinfo?(rail)
Yup, the diff looks OK to me. My issue is with the schedulers. :)
Flags: needinfo?(rail)
(Assignee)

Comment 66

3 years ago
Created attachment 8509522 [details] [diff] [review]
aggregatingscheduler.patch

I'm testing this patch now.  To ensure that the correct jobs are being triggered, I had to fix my staging environment last night/this morning to run android builds since it was in a weird state but this seems to be working now. (Before I was just retriggering l10n repacks) Once this is complete I'll ask for review on this final patch.
Attachment #8501168 - Attachment is obsolete: true
(Assignee)

Comment 67

3 years ago
Created attachment 8509973 [details] [diff] [review]
bug1019724bbcustom-5.patch

This patch worked in staging and triggered the correct builds and sent emails.  

One thing I didn't do is change all the updates builders to include the product name.  Not sure if this should be included.
Attachment #8509522 - Attachment is obsolete: true
Attachment #8509973 - Flags: review?(rail)
Comment on attachment 8509973 [details] [diff] [review]
bug1019724bbcustom-5.patch

Review of attachment 8509973 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry, have to r- this :(

::: process/release.py
@@ +1266,2 @@
>          builders.append(makeDummyBuilder(
> +            name=builderPrefix('updates_%s' % releaseConfig['productName']),

This won't work:
1) it will break the scheduler graph for non android builds when you set skip_updates to True
2) This dummy builder won't be triggered by the corresponding aggregating scheduler because it's not added to post_signing_builders list. This builder is referenced by name everywhere in the file :/

I'd changed "updates" everywhere in the release.py to include productName.
Attachment #8509973 - Flags: review?(rail) → review-
(Assignee)

Comment 69

3 years ago
Created attachment 8510303 [details] [diff] [review]
bug1019724bbcustom-6.patch

running tests with this patch now
Attachment #8509973 - Attachment is obsolete: true
(Assignee)

Comment 70

3 years ago
Comment on attachment 8510303 [details] [diff] [review]
bug1019724bbcustom-6.patch

This worked in staging
Attachment #8510303 - Flags: review?(rail)
Comment on attachment 8510303 [details] [diff] [review]
bug1019724bbcustom-6.patch

Woot!

When you land the patch, can you also incorporate this change to keep the directory names in sync with the builder names?

https://gist.github.com/rail/41a7261a3376d168ed4f
Attachment #8510303 - Flags: review?(rail) → review+
(Assignee)

Comment 72

3 years ago
Created attachment 8510756 [details] [diff] [review]
bug1019724-6.patch

new patch with rail's other suggestions
Comment on attachment 8510756 [details] [diff] [review]
bug1019724-6.patch

just in case :)
Attachment #8510756 - Flags: review+
(Assignee)

Comment 74

3 years ago
I'll land these changes on Monday, FF33.0.1 build2 is running right now.
(Assignee)

Updated

3 years ago
Attachment #8510303 - Attachment is obsolete: true
(Assignee)

Updated

3 years ago
Blocks: 1058734
(Assignee)

Comment 75

3 years ago
And we still have releases in flight today, this is why I haven't landed the changes.
(Assignee)

Comment 76

3 years ago
Created attachment 8512774 [details] [diff] [review]
bug1019724bbcustom-8.patch

patch to fix failing tests due to duplicate builder names etc

ran test-masters.sh in staging and it was all green
Attachment #8512774 - Flags: review?(rail)
(Assignee)

Comment 77

3 years ago
Created attachment 8512792 [details] [diff] [review]
unbittrotten mh patch that was landed today
Attachment #8512774 - Flags: review?(rail) → review+
(In reply to Kim Moir [:kmoir] from comment #76)
> Created attachment 8512774 [details] [diff] [review]
> bug1019724bbcustom-8.patch
> 
> patch to fix failing tests due to duplicate builder names etc
> 
> ran test-masters.sh in staging and it was all green

For this to work as expected you also need to add new symlinks in buildbot-configs/mozilla/release_templaets
(Assignee)

Comment 79

3 years ago
Created attachment 8512850 [details] [diff] [review]
bug1019724-morebbtemplates.patch

more template files
Attachment #8512850 - Flags: review?(rail)
(Assignee)

Updated

3 years ago
Attachment #8512792 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8512774 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8510756 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8508209 - Flags: checked-in+
Comment on attachment 8512850 [details] [diff] [review]
bug1019724-morebbtemplates.patch

Can you also add thunderbird?
Attachment #8512850 - Flags: review?(rail) → review+
(Assignee)

Comment 81

3 years ago
Created attachment 8512892 [details] [diff] [review]
bug1019724-morebbtemplates-2.patch

added tb
(Assignee)

Updated

3 years ago
Attachment #8512892 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/da3ac7ace7e3
In production: https://hg.mozilla.org/build/buildbot-configs/rev/eab4127639e5
In production: https://hg.mozilla.org/build/buildbotcustom/rev/0151fbedc670
In production: https://hg.mozilla.org/build/buildbotcustom/rev/a2841db14eaf
In production: https://hg.mozilla.org/build/mozharness/rev/282d655f1ab2
Created attachment 8514220 [details] [diff] [review]
add templates for fennec/firefox updates builders

Kim, I noticed we got a plain "[release] Firefox 33.1 build1: completed updates_firefox" instead of "Updates available on the betatest channel" mail for 33.1. This patch should fix that...
Attachment #8514220 - Flags: review?(kmoir)
(Assignee)

Comment 88

3 years ago
Comment on attachment 8514220 [details] [diff] [review]
add templates for fennec/firefox updates builders

thanks Ben!
Attachment #8514220 - Flags: review?(kmoir) → review+
Attachment #8514220 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/d175e19ca8b7
There's a few more templates that need fixing, I think. We got mail like:
31275     Oct 31 release@mozilla (0.1K) ├─>[release] Firefox 34.0b5 build2: completed ready_for_releasetest_testing_firefox
31277     Oct 31 release@mozilla (0.1K) └─>[release] Firefox 34.0b5 build2: completed ready_for_release_firefox

Which should have special subjects:
https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/ready_for_release_success
https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/ready_for_releasetest_testing_success
In Fennec 34.0b6 we got an error trying to submit to bouncer:

scripts/scripts/bouncer_submitter.py -c releases/bouncer_fennec_beta.py --revision FENNEC_34_0b6_RELEASE --repo releases/mozilla-beta --version 34.0b6 --credentials-file oauth.txt --bouncer-api-prefix https://bounceradmin.mozilla.com/api
 in dir /builds/slave/rel-m-beta-bncr_sub-0000000000/. (timeout 1200 secs)
...
IOError: Can't find releases/bouncer_fennec_beta.py in ['.', '/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/scripts/../configs', '/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/scripts/../../configs']!

Full log: http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/34.0b6-candidates/build1/logs/release-mozilla-beta-bouncer_submitter_fennec-bm73-build1-build0.txt.gz

Looks like we just have a mozharness/configs/release/bouncer_fennec.py, with no branch suffix.
(Assignee)

Comment 92

3 years ago
Created attachment 8516375 [details] [diff] [review]
bug1019724bbconfigs-5.patch

patch to address comment #91
Attachment #8516375 - Flags: review?(nthomas)
Comment on attachment 8516375 [details] [diff] [review]
bug1019724bbconfigs-5.patch

Looks good. Would you me to rerun the bouncer submitter for 34.06 with this ?
Attachment #8516375 - Flags: review?(nthomas) → review+
Oops, I meant 34.0b6. The next fennec release will be this time next week, 34.0b8.
(Assignee)

Updated

3 years ago
Attachment #8516375 - Flags: checked-in+
(Assignee)

Comment 95

3 years ago
nthomas: you can rerun the bouncer submitter if it's not too much trouble.  Sorry for the breakage :-)
Comment on attachment 8516375 [details] [diff] [review]
bug1019724bbconfigs-5.patch

Transplanted to production and moved release tags:
http://hg.mozilla.org/build/buildbot-configs/rev/3437ada679c9
I was watching the log as bouncer_submitter_fennec ran, and needed to add an 'android-x86' OS to bouncer. Then it was happy.

There's a problem with the path's though:
 Product         OS              Path
 Fennec-34.0b6   android-x86     /mobile/releases/34.0b6/android-x86/:lang/fennec-34.0b6.android-x86.apk
 Fennec-34.0b6   android         /mobile/releases/34.0b6/android/:lang/fennec-34.0b6.android-arm.apk

There should be another :lang in the filenames, eg fennec-34.0b6.en-US.android-arm.apk. I've tried fixing that up by hand this time, but sentry has issues with it:

[2014-11-03 19:32:56 -0800] HEAD http://72.21.81.253/pub/mobile/releases/34.0b6/android/en-US/fennec-34.0b6.:lang.android-arm.apk ... FAILED. rc=404 CACHE=:miss: TOOK=0.053443
[2014-11-03 19:32:56 -0800] HEAD http://72.21.81.253/pub/mobile/releases/34.0b6/android-x86/en-US/fennec-34.0b6.:lang.android-x86.apk ... FAILED. rc=404 CACHE=:miss: TOOK=0.054189
Finished. Elapsed time: 54.

Only one :lang replacement is being done at https://github.com/mozilla/tuxedo/blob/master/sentry/sentry.pl#L271 because it's not ...@g. We could fix that, and I think serving files would be ok (it uses php's str_replace which replaces all instances, https://github.com/mozilla/tuxedo/blob/master/bouncer/php/index.php#L160).  You might catch the push to prod from bug 1089729 if you're fast.

Alternatively we could switch to pretty naming (MOZ_PKG_PRETTYNAMES) if that exists for Fennec.
Found another glitch - the uptake checker is looking for these:
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6&os=android
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6&os=android-x86
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6-Complete&os=android
 https://bounceradmin.mozilla.com/api/uptake/?product=Fennec-34.0b6-Complete&os=android-x86

The last two must be accidental, but the server returns a '400 BAD REQUEST', which shows up in the exception logs we get emailed by the master.
Sorry, found something else. Wondering why we have new builder names like 
  release-mozilla-beta-bouncer_submitter_firefox
  release-mozilla-beta-updates_firefox
instead of
  release-mozilla-beta-firefox_bouncer_submitter
  release-mozilla-beta-firefox_updates
to fit with the existing style
  release-mozilla-beta-fennec_tag

Might be related to earlier issues with release emails.
(Assignee)

Comment 100

3 years ago
Created attachment 8516722 [details] [diff] [review]
bug1019724sentry.patch

patch for comment #97
Attachment #8516722 - Flags: review?(nthomas)
(Assignee)

Comment 101

3 years ago
Created attachment 8516723 [details] [diff] [review]
bug1019724bouncer.patch

another patch for comment #97
Attachment #8516723 - Flags: review?(nthomas)
(Assignee)

Comment 102

3 years ago
rail: What's the standard for including productName in the builder name?  If you look at release.py, some of the builders include it at the start of their name, some in the middle, and I added ones that include the product name in the end.  As Nick (comment#99) and Ben (comment #90) pointed out, this causing problems with the templates.
Flags: needinfo?(rail)
(Assignee)

Comment 103

3 years ago
:nthomas regarding comment #98, so the complete mar entry should be removed from here https://aus4-admin.mozilla.org/releases/Fennec-34.0b6-build1/data Should it be replaced by something else?
Flags: needinfo?(nthomas)
I don't think we have some formal standards here, just common sense and consistency. We should fix the latter. :)
Flags: needinfo?(rail)
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/649] [kanban:engops:https://kanbanize.com/ctrl_board/6/629]
(Assignee)

Comment 105

3 years ago
Created attachment 8516966 [details] [diff] [review]
bbconfigsnov4.patch

makes builder names more consistent, fixes templates
Attachment #8516966 - Flags: review?(rail)
(Assignee)

Comment 106

3 years ago
Created attachment 8516967 [details] [diff] [review]
bbcustomnov4.patch

makes builder names more consistent, fixes templates
Attachment #8516967 - Flags: review?(rail)
(Assignee)

Comment 107

3 years ago
Created attachment 8516976 [details]
bug1019724.diff

builderdiff from release.py changes
Attachment #8516966 - Flags: review?(rail) → review+
Attachment #8516967 - Flags: review?(rail) → review+

Updated

3 years ago
Attachment #8516723 - Flags: review?(nthomas) → review+
Comment on attachment 8516722 [details] [diff] [review]
bug1019724sentry.patch

Sorry, this will have to go to a bug in Webtools :: Bouncer, and rhelmer can review. Looks fine to me though.
Flags: needinfo?(nthomas)
Attachment #8516722 - Flags: review?(nthomas)
(Assignee)

Updated

3 years ago
Depends on: 1093965
(In reply to Kim Moir [:kmoir] from comment #103)
> :nthomas regarding comment #98, so the complete mar entry should be removed
> from here https://aus4-admin.mozilla.org/releases/Fennec-34.0b6-build1/data
> Should it be replaced by something else?

It's something else. The bouncer check is this code
 http://mxr.mozilla.org/build/source/buildbotcustom/scheduler.py#218
which ends up at
 http://mxr.mozilla.org/build/source/tools/lib/python/util/tuxedo.py#66
checkMars defaults to True, hence the check for Fennec-34.0b6-Complete. It can be forced to False if we set skip_updates=True in the release config, which wouldn't have any other effects for mobile.

There's a few things that need to be consistent though:
* the fileUrls in balrog. Currently Fennec-34.0b6-Complete, which makes a kind of sense over Fennec-34.0b6 (we're serving updates, not installers). [1]
* the products in bouncer submitter, as interpolated from the mozharness config. Currently Fennec-version
* the products checked in TriggerBouncerCheck. I wonder if we should teach this to read the mozharness config instead (we also have bug 1082823 to move it from master to slave)

So there's a wee tangle to sort out there.

[1] Incidentally, the beta-localtest url of 
 http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/34.0b6-candidates/build1/update/%OS_FTP%/%LOCALE%/fennec-34.0b6.complete.mar
needs to lose the update/ part. And we only have en-US for android-x86, perhaps we should be turning on all locales, or not doing anything for it on this bug.
(Assignee)

Updated

3 years ago
Attachment #8516723 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8516966 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8516967 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/4fe5cfc3981b
In production: https://hg.mozilla.org/build/buildbotcustom/rev/cbc087de6234
In production: https://hg.mozilla.org/build/mozharness/rev/4e0acd0c72be
(Assignee)

Comment 113

3 years ago
Okay after reading through a lot of balrog and friends code this morning I have a few questions regarding comment #109

So skip_updates=True should be set in the release configs for beta and release for Fennec, correct?

Regarding this comment
>>>There's a few things that need to be consistent though:
* the fileUrls in balrog. Currently Fennec-34.0b6-Complete, which makes a kind of sense over Fennec-34.0b6 (we're serving updates, not installers). [1]
* the products in bouncer submitter, as interpolated from the mozharness config. Currently Fennec-version
* the products checked in TriggerBouncerCheck. I wonder if we should teach this to read the mozharness config instead (we also have bug 1082823 to move it from master to slave)

I don't understand what needs to change here in the actual data to fix this other than removing updates in the first item
https://aus4-admin.mozilla.org/releases/Fennec-34.0b6-build1/data

The bata-localtest url needs to lose update. So here I would remove the update/ part by 
where ProductName is Fennec and channel is betatest. Because for Firefox it needs to remain
http://hg.mozilla.org/build/tools/file/046e9d5ec108/lib/python/balrog/submitter/cli.py#l183

Should I remove android-x86 from mozharness/configs/releases/bouncer_fennec.py?

If having a short vidyo to discuss would be easier, that would be great with me.
We got an email '[release] Fennec 33.1 build1: Updates available on None', so templates might need a further tweak.
(Assignee)

Comment 115

3 years ago
Hmm, I looked at the templates issue and couldn't seem to see where the problem is.  Will look at it a again a bit later
(Assignee)

Comment 116

3 years ago
So I was talking to Nick about this bug today and he raised two important points

Will there every be anything else than en-US for Android-x86?  If not, it should not be submitted to balrog and removed from the configs that do so.

What sorts of QA will be done on these single locales for Android at the time of release?  If there isn't going to be any formal testing we should remove the localtest channels from these configs.
Flags: needinfo?(blassey.bugs)
(Assignee)

Comment 117

3 years ago
Other discussion with Nick re how to fix the remaining problems

Option 1
Rewrite get_release_update in https://mxr.mozilla.org/build/source/tools/lib/python/util/tuxedo.py#66 to read the to read the mozharness bouncer config files for balrog
http://hg.mozilla.org/build/mozharness/file/2212615c4fa4/configs/releases/bouncer_firefox_release.py#l94
http://hg.mozilla.org/build/mozharness/file/2d78545b5039/configs/releases/bouncer_fennec.py

loop over the keys in the keys and ensure they are correct

This is a change in both Fennec and desktop and will require testing in the staging bouncer + staging master + sentry.

Option 2:
skip_updates=True in the release config
leave bouncer config
change balrog submitter so so it accepts Fennec-Version-complete
wrangle tuxedo.py to parse this
Nice summary, one tweak ...

(In reply to Kim Moir [:kmoir] from comment #117)
> Option 2:
> skip_updates=True in the release config
> leave bouncer config
> change balrog submitter so so it accepts Fennec-Version-complete
> wrangle tuxedo.py to parse this

change balrog submitter so it posts Fennec-<version>. Should be OK in tuxedo.py then, as skip_updates makes checkMars False.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/649] [kanban:engops:https://kanbanize.com/ctrl_board/6/629] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/649]
(In reply to Kim Moir [:kmoir] from comment #116)
> So I was talking to Nick about this bug today and he raised two important
> points
> 
> Will there every be anything else than en-US for Android-x86?  If not, it
> should not be submitted to balrog and removed from the configs that do so.
Yes, Android-x86 should have the same locales as Android-ARM
 
> What sorts of QA will be done on these single locales for Android at the
> time of release?  If there isn't going to be any formal testing we should
> remove the localtest channels from these configs.
That's a question for QA
Flags: needinfo?(blassey.bugs) → needinfo?(kbrosnan)
(Assignee)

Comment 120

3 years ago
> What sorts of QA will be done on these single locales for Android at the
> time of release?  If there isn't going to be any formal testing we should
> remove the localtest channels from these configs.
That's a question for QA

Aaron I'm working on enabling single locale updates for Android on beta and release.  Do you know the extent of testing these repacks will receive by QA?
Flags: needinfo?(aaron.train)
Specifically, do you want to use the desktop model for update testing, which is to check those updates on a test channel prior shipping to users.
(Assignee)

Comment 122

3 years ago
In the mobile meeting today kbrosan said that he would use the desktop model for testing.
Flags: needinfo?(kbrosnan)
Flags: needinfo?(aaron.train)
(In reply to Nick Thomas [:nthomas] from comment #114)
> We got an email '[release] Fennec 33.1 build1: Updates available on None',
> so templates might need a further tweak.

Any idea what's going on here ? Hit it on 34.0 build1 and seems to just be release builds, beta's OK.
(Assignee)

Comment 124

3 years ago
With respect to comment 123, 

the templates for beta have 
releaseConfig['localTestChannel']      = 'betatest'
releaseConfig['cdnTestChannel']        = 'releasetest'
and the message templates such as 
fennec_updates_success reference localTestChannel

So should the templates for fennec releases also reference localtest or should both beta and releases be updated to new channel names for fennec

I read this wiki page but it was not clear to me what the correct channel should be
https://wiki.mozilla.org/Releases/Update_Channels
Flags: needinfo?(nthomas)
(Assignee)

Comment 125

3 years ago
Latest data for the patches to address comment #118 is here

https://aus4-admin-dev.allizom.org/releases#Fennec%2034.0b8

Patches will be attached next.
(Assignee)

Comment 126

3 years ago
Created attachment 8528590 [details] [diff] [review]
bug1019724tools.patch
Attachment #8528590 - Flags: review?(nthomas)
(Assignee)

Comment 127

3 years ago
Created attachment 8528592 [details] [diff] [review]
bug1019724bbconfigs-9.patch

green builds in staging
Attachment #8528592 - Flags: review?(nthomas)

Updated

3 years ago
Attachment #8528592 - Flags: review?(nthomas) → review+

Updated

3 years ago
Attachment #8528590 - Flags: review?(nthomas) → review+
(In reply to Kim Moir [:kmoir] from comment #124)

I think you're right that it's that some of the configs don't have localTestChannel and cdnTestChannel definitions. For now lets copy what firefox does for mozilla-release builds, and use betatest and releasetest. Later, when we swap to Balrog for release we can change the test channel names for both Firefox and Fennec.
Flags: needinfo?(nthomas)
(Assignee)

Updated

3 years ago
Attachment #8528590 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8528592 - Flags: checked-in+
(Assignee)

Comment 129

3 years ago
Created attachment 8529105 [details] [diff] [review]
bug1019724fennecrtemplates.patch

patch to address comment 128
Attachment #8529105 - Flags: review?(nthomas)
(Assignee)

Comment 130

3 years ago
Created attachment 8529136 [details] [diff] [review]
bug1019724fennecrtemplates.patch

noticed looking at the current release jobs that is was also missing releaseChannel and releaseChannelRuleIds.  Not sure what the releaseChannelRuleIds should be
Attachment #8529105 - Attachment is obsolete: true
Attachment #8529105 - Flags: review?(nthomas)
Attachment #8529136 - Flags: review?(nthomas)
In production: https://hg.mozilla.org/build/buildbot-configs/rev/2c26cb775d70
Comment on attachment 8529136 [details] [diff] [review]
bug1019724fennecrtemplates.patch

r+ with a some changes described below.

>diff --git a/mozilla/release-fennec-mozilla-release.py.template b/mozilla/release-fennec-mozilla-release.py.template
>+releaseConfig['releaseChannelRuleIds'] = []

That's fine, will change when we switch to Balrog. Should set up staging configs the same though, and add to all four files for beta.

>diff --git a/mozilla/staging_release-fennec-mozilla-release.py.template b/mozilla/staging_release-fennec-mozilla-release.py.template
>+releaseConfig['testChannels']          = ['release-localtest', 'release-cdntest']

testChannels can be removed complete, deprecated by bug 986990.
Attachment #8529136 - Flags: review?(nthomas) → review+
(Assignee)

Comment 133

3 years ago
Created attachment 8530321 [details] [diff] [review]
bug1019724fennecrtemplates-2.patch

updated incorporating feedback from comment 132
(Assignee)

Updated

3 years ago
Attachment #8530321 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/367a1d157f91
BTW, there are 3 dangling symlinks introduced in http://hg.mozilla.org/build/buildbot-configs/rev/eab4127639e5:

$ find -L . -type l        
./mozilla/release_templates/fennec_ready_for_release
./mozilla/release_templates/firefox_ready_for_release
./mozilla/release_templates/thunderbird_ready_for_release

All of them point to ready_for_release, which doesn't exist. Can you fix them too.
(Assignee)

Comment 136

3 years ago
Created attachment 8533188 [details] [diff] [review]
bug1019724rmtemplates.patch

remove extraneous files mentioned in comment #135
Attachment #8533188 - Flags: review?(rail)
Attachment #8533188 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Attachment #8533188 - Flags: checked-in+
(Assignee)

Comment 137

3 years ago
I think these should be ready to test for the next beta - Kbrosnan could you make sure they are verified by QE?
Flags: needinfo?(kbrosnan)
(Assignee)

Comment 138

3 years ago
Created attachment 8534474 [details] [diff] [review]
bug1019724testchannels.patch

test channels for beta
Attachment #8534474 - Flags: review?(bhearsum)
Attachment #8534474 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

3 years ago
Attachment #8534474 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/d75ad61ca11c
In production: https://hg.mozilla.org/build/buildbot-configs/rev/1bda6639bf51
(Assignee)

Comment 141

3 years ago
It would be good if someone from QE could verify the single locale updates from 35.0b4 on 
http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/35.0b4-candidates/build1/

They are allocated to channels beta-localtest and  beta-cdntest
Unfortunately, bug 792992 means we can't use test channels for Android (not yet anyway, I requested tracking on it). If there aren't any users out there already we can probably test on beta.
(Assignee)

Comment 143

3 years ago
Created attachment 8538033 [details] [diff] [review]
bug1019724rc.patch

okay thanks I didn't realize that nthomas.  I've added a rule for the release channel id in balrog and linked it to 35.0b4. Here's a patch so it can be enabled for future betas automatically.
Attachment #8538033 - Flags: review?(nthomas)

Updated

3 years ago
Attachment #8538033 - Flags: review?(nthomas) → review+
(Assignee)

Updated

3 years ago
Attachment #8538033 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Depends on: 1117831
See 1117831 comment 5 for current status. Blocked on development work or we need to use the beta channel.
Flags: needinfo?(kbrosnan)
(Assignee)

Updated

3 years ago
Depends on: 1122623, 792992

Updated

3 years ago
No longer depends on: 1122623
(In reply to Nick Thomas [:nthomas] from comment #142)
> Unfortunately, bug 792992 means we can't use test channels for Android (not
> yet anyway, I requested tracking on it). If there aren't any users out there
> already we can probably test on beta.

Bug 792992 was fixed. What do we need to do to start some testing?
(Assignee)

Comment 146

2 years ago
Were the patches from  bug 792992 uplifted to beta so it could be tested?  This is what I asked here
https://bugzilla.mozilla.org/show_bug.cgi?id=792992#c34 

We need the patch to be available in beta so QA can test
(Assignee)

Updated

2 years ago
Depends on: 1142049
(Assignee)

Updated

2 years ago
No longer depends on: 1142049
Any update here?
(Assignee)

Comment 148

2 years ago
waiting for a needinfo from mkfinkle in bug 1139567 before we can proceed.  If you read through the comments in bug 1139567, the work required seems to have increased substantially with the introduction of split apks
(In reply to Kim Moir [:kmoir] from comment #148)
> waiting for a needinfo from mkfinkle in bug 1139567 before we can proceed. 
> If you read through the comments in bug 1139567, the work required seems to
> have increased substantially with the introduction of split apks

Karen answered the NI in bug 1139567. In short, this is still wanted.
Duplicate of this bug: 1192279
Blocks: 1192279
I'd like to advocate for a standalone, updateable APK for Indic Marathi (mr). See:

https://github.com/mozilla/participation-org/issues/99
Depends on: 1220773

Updated

2 years ago
Depends on: 710802

Updated

2 years ago
Depends on: 1226474
Depends on: 1237785
Kim, this had a BUNCH of patches on it, most checked in, and most over a year old, can we wrap this up and re-file with the current ask(s) as known, so the current state is easier to detangle, please.
Flags: needinfo?(kmoir)
(Assignee)

Comment 153

a year ago
The patches were implemented to lay the groundwork for supporting single locale builds but it was never fully implemented because other priorities were deemed more important to work on.  

Callek: Looking at the email thread that you started entitled "Fennec Single Locale Repacks (Nightly/Release"
with Barbara, Lawrence, Catlee etc on it, there still doesn't seem to be strong consensus that these are a priority to implement.

From catlee
I spoke with Axel and Flod about the single-locale repacks in London. They mentioned they were useful for localizers working on locales that weren't yet ready to be included in the multi-locale build.

From Lawrence:
Useful but necessary? Is there another option for these localizers?

Until we have strong consensus that these are needed, I'm not sure that this bug should stay open.  Most of the patches that were landed aren't won't be relevant for long anyways, given the move to release promotion and taskcluster.
Flags: needinfo?(kmoir)
(In reply to Kim Moir [:kmoir] from comment #153)
> From catlee
> I spoke with Axel and Flod about the single-locale repacks in London. They
> mentioned they were useful for localizers working on locales that weren't
> yet ready to be included in the multi-locale build.
> 
> From Lawrence:
> Useful but necessary? Is there another option for these localizers?

Is this something that we could offer as opt-in on Try only? 

I know that's not a model we generally support (i.e. builds on Try and not elsewhere), but if these really are only useful for vetting changes, do we really need regular builds elsewhere?
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.