Move shipped-locales decision behind release automation

RESOLVED WONTFIX

Status

Release Engineering
General
P2
normal
RESOLVED WONTFIX
10 years ago
4 years ago

People

(Reporter: Pike, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [l10n])

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

10 years ago
We should make the call on which locales we ship after we created the release bits.

Basically, we don't have enough timely information to make a call on which locales build and are ready to ship without bringing l10n to a complete halt for all projects in our repository.

Recent example was pa-IN on Windows, which was green when we did the shipped-locales magic, and was red by the time we branched, due to the lag of windows builds.
Priority: -- → P3
We should do something about this ASAP if we want this for beta4.

One quick workaround would be to simply add everything in all-locales to shipped-locales, and update/check-in this file and re-run steps as they fail...
Blocks: 418926
Rob, are you aware how the l10n process has changed for b4 ? They're semi-freezing today for testing on nightlies, and there's a good chance shipped-locales will be finalised by the time we come to do the build for real.
So the ramifications to the release team for doing this are:

1) very likely automation will fail out on e.g. l10n verify or update step, due to broken locales
2) 99.999% likely that we will need to remove update and files that do not pass QA

The way to resolve #1 is to check into shipped-locales and remove the broken locales, then redo the failed step. This is a little cumbersome but seems ok to me process-wise.

#2 is the real pain here, we need to do this by hand so we need to get it right. Involves removing update configuration as well as files, just need to make sure we don't miss any.

Maybe for #2 we could check into shipped-locales, re-run update config, regen only the config, remove the private staging area, and re-run the staging step. I think this would be the safest as far as accidentally forgetting to remove some bit, and it'd be clear from CVS history of shipped-locales what we did.
(In reply to comment #2)
> Rob, are you aware how the l10n process has changed for b4 ? They're
> semi-freezing today for testing on nightlies, and there's a good chance
> shipped-locales will be finalised by the time we come to do the build for real.

I was not, thanks for the info. 

We've discussed for a long time changing the process from:

a) guess what locales we think we want to ship

to:

b) try to build everything, remove what doesn't pass QA

I know that Axel and mconnor have both asked for (b) for quite a while now. Does anyone still want to do this, or are is there enough effort going to tighten up (a)?

I think we still need to continue making it easier to add or drop locales after shipped-locales is finalized, but maybe that's a different topic than this bug.

Comment 5

10 years ago
#2 is not blocking for B4 is it?  We have hand tweaked final locales that were
shipped for previous milestones.  So this isn't urgent for B4 is it?  

The QA plan for B4 is to spot check the P1 locales
(http://wiki.mozilla.org/QA/Firefox3/TestResults/Beta4#L10N_P1_Spotchecks)
before release.  And then come back after release to spot check the other
locales and platform.  There is no way to test all locales and all platforms
before B4 release.

So there is a chance that a problem is found during the B4 release testing of locales.

Keep in mind there are several different types of testing going on: localizer tests, locale community test,  3rd party system test, and QA spot tests.  For B4, the QA spot tests are happening on the B4 bits, _not_ the nightlies.
Duplicate of this bug: 368244
(Reporter)

Comment 7

10 years ago
(In reply to comment #2)
> Rob, are you aware how the l10n process has changed for b4 ? They're
> semi-freezing today for testing on nightlies, and there's a good chance
> shipped-locales will be finalised by the time we come to do the build for real.

There has been some confusion about this, and at least I'm not aware that the process changed. It'd be easier for me to catch up on this if there'd be a pointer to that discussion. Otherwise it might just pop up when I make it through my somewhat 2000 unread emails.
No longer blocks: 418926
triaged.
Component: Build & Release → Release Engineering: Projects
QA Contact: build → release
(Reporter)

Updated

9 years ago
Whiteboard: [l10n]

Comment 9

9 years ago
Axel and others, what is the status on this bug?

It has passed more than 12 months since last discussed and since then many chances were done to the hg release automation and to the l10n repackages being run by buildbot.
(Reporter)

Comment 10

9 years ago
This is essentially still something we want.

The whole concept of shipped-locales, all-locales, l10n-changesets and product_details.php is error prone, and doesn't really help us in shipping good software.

Gandalf and I are tentatively thinking about moving that information out of the files themselves. At least for l10n-changesets and shipped-locales, those are already on their way to be created out of a single database on the l10n dashboard.
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Not sure if Axel/Gandalf have a plan for this yet. Perhaps something for us all to discuss at the summit if the decision lingers that long.
(Reporter)

Comment 13

8 years ago
FYI, I'm not going to be at the summit.

Updated

8 years ago
Priority: P3 → P5
Assignee: nobody → rail

Updated

7 years ago
Blocks: 478420
(Assignee)

Comment 14

7 years ago
As a possible solution (maybe intermediate):

1) Releng downloads signed off l10n-changesets manually, reconfigs the master.
2) As a part of tagging step, release automation downloads signed off shipped-locales file and puts it on relbranch.
3) The rest of release automation continues as usual.

Comments are welcome.
(Reporter)

Comment 15

7 years ago
I think we should just get rid of shipped-locales in favor of a l10n-changesets_mobile.json -like solution, tbh.
Duplicate of this bug: 562903
(Assignee)

Comment 17

7 years ago
Created attachment 504460 [details] [diff] [review]
Update shipped-locales during tagging

Attached is a first draft, which may be polished after getting positive feedback.

The main idea here is to download l10n-changesets file (if updateShippedLocales is set in configs) and covert it to shipped-locales (on relbranch). In the future l10n-changesets should be changed with a json file with changesets and platforms (like for mobile).
Attachment #504460 - Flags: feedback?(catlee)
Attachment #504460 - Flags: feedback?(bhearsum)
(Reporter)

Comment 18

7 years ago
Comment on attachment 504460 [details] [diff] [review]
Update shipped-locales during tagging

This seems to not support the case of "not ship 'or' on mac". We don't have that situation right now, AFAICT, but it's something we've had to use in the past. And with more platforms (say, android-based fx), it might come back.
Comment on attachment 504460 [details] [diff] [review]
Update shipped-locales during tagging

Why not download shipped-locales instead of l10n-changesets? We can avoid hardcoding any exceptions that way since we're getting it straight from the authoritative source. No need for special en-US treatment, either.

Eg, https://l10n-stage-sj.mozilla.org/shipping/shipped-locales?ms=fx3.5.17

Also, please don't hardcode "fx" in here, put a mapping somewhere in lib/python and convert the full product name at runtime. That way, other products (fennec, thunderbird, seamonkey) can use it, too.

How do you feel about putting browser/locales/shipped-locales into bumpFiles? You'd have to pull it out before passing it to version-bump.pl, but to me, it seems nicer than having an updateShippedLocales flag.
Attachment #504460 - Flags: feedback?(bhearsum) → feedback-
(Reporter)

Comment 20

7 years ago
Sadly, not true. The platforms need to be manually added at this point, for anything bug ja/ja-JP-mac. As long as the file isn't automated, that works. Storing that information outside of shipped-locales and careful editing that one is bug 535558.

I wonder how to fix that bug, though, would it be the better path to just fix it for the json case and make all apps use that, dropping shipped-locales alltogether?
(In reply to comment #20)
> Sadly, not true. The platforms need to be manually added at this point, for
> anything bug ja/ja-JP-mac. As long as the file isn't automated, that works.
> Storing that information outside of shipped-locales and careful editing that
> one is bug 535558.

To clarify, this means that platform exceptions other than ja-JP-mac/ja are not handled by the l10n dashboard, which means it will serve incorrect shipped-locales files if they exist.

> I wonder how to fix that bug, though, would it be the better path to just fix
> it for the json case and make all apps use that, dropping shipped-locales
> alltogether?

I don't see any reason to conflate this relatively simple automation of a manual process with a big overhaul like that.

In my opinion, our best path here is hardcoding the overrides we need until either:
a) the dashboard correctly serves shipped-locales
b) we switch to an integrated l10n-changesets/shipped-locales

Rail/Axel, how is that for you two?

With that in mind, what you have here is mostly good, except getPlatformLocales belongs in release.l10n, and with a new name, process() can also go there -- converting l10n-changesets -> shipped-locales seems generic enough. The other parts of my previous comment still apply, too.
(Reporter)

Comment 22

7 years ago
We don't have a good track record for keeping hard-coded things up. Like, I think that a good deal of the l10n-sensitive dirs in the scheduler are not right at this point. Is this one gonna be different? Maybe. Maybe just because we're lucky and don't really need anything anyway ;-)
Do you have a better idea that can still be automated?
(Assignee)

Comment 24

7 years ago
I'd be happy to use https://l10n-stage-sj.mozilla.org/shipping/shipped-locales?ms=... URL. In this case the bug turns into one run_cmd(['wget', ...]) maybe with some error handling.

The worst thing I see here is that we introduce another point of failure. It would be impossible to run tagging if the dashboard is not reachable (network, failures, etc).

> How do you feel about putting browser/locales/shipped-locales into bumpFiles?
> You'd have to pull it out before passing it to version-bump.pl, but to me, it
> seems nicer than having an updateShippedLocales flag.

The idea was to have this step configurable and have an ability to skip for some versions (1.9.x?). In the future this step will become mandatory (no shipped-locales on default branch?).
(In reply to comment #24)
> I'd be happy to use
> https://l10n-stage-sj.mozilla.org/shipping/shipped-locales?ms=... URL. In this
> case the bug turns into one run_cmd(['wget', ...]) maybe with some error
> handling.
> 
> The worst thing I see here is that we introduce another point of failure. It
> would be impossible to run tagging if the dashboard is not reachable (network,
> failures, etc).

That's the case with your current patch, isn't it? In any case, we can't trust the shipped-locales we're getting from the dashboard....

> 
> > How do you feel about putting browser/locales/shipped-locales into bumpFiles?
> > You'd have to pull it out before passing it to version-bump.pl, but to me, it
> > seems nicer than having an updateShippedLocales flag.
> 
> The idea was to have this step configurable and have an ability to skip for
> some versions (1.9.x?). In the future this step will become mandatory (no
> shipped-locales on default branch?).

It's still configurable with my suggestion, isn't it? You can read bumpFiles from the config instead of updateShippedLocales.

Comment 26

7 years ago
If the source of revisions ends up being the dashboard could we have some type of jobs that publishes the latest l10n-changeset somewhere else as well?
We could fall back on that if we had a missing dashboard.
Depending on the dashboard in the automation doesn't really change much from the current situation -- where we depend on the dashboard when manually creating the configs. That, combined with an off switch is good enough IMHO.
Blocks: 627271
(Reporter)

Comment 28

7 years ago
I don't think the intention of this bug is to move the dashboard into the critical path twice instead of once. That sounds very much like a countergoal, and is in contrast to previous discussions.
No longer blocks: 478420
(Assignee)

Comment 29

7 years ago
(In reply to comment #28)
> I don't think the intention of this bug is to move the dashboard into the
> critical path twice instead of once. That sounds very much like a countergoal,
> and is in contrast to previous discussions.

Do you mean that it's better to reuse already existing l10n-changests file (in buildbot-configs) instead of fetching it?
(Assignee)

Comment 30

7 years ago
Created attachment 505715 [details] [diff] [review]
Update shipped-locales

Another approach, which reuses already existing l10n-changesets file.

* Not tested at all, still looking for a right approach
* Dashboard isn't used at all, reusing already existing l10n-changesets.

I have another approach in my mind. What if we use l10n-changesets file not for generating shipped-locales from scratch, but for validating it? Something like this:

if locales in both files:
  leave the line as is (including platforms if there)
else:
  if line in shipped-locales only:
    remove the line
  else:
    add that locale (w/o specifying platforms)

In this case we'd have less problems with platforms and exceptions.
Attachment #504460 - Attachment is obsolete: true
Attachment #505715 - Flags: feedback?(l10n)
Attachment #505715 - Flags: feedback?(bhearsum)
Attachment #504460 - Flags: feedback?(catlee)
(Reporter)

Comment 31

7 years ago
Comment on attachment 505715 [details] [diff] [review]
Update shipped-locales

This is closer, but there are some missing parameters to actually be able to hard-code platform exceptions. Should probably be app and branch.
Attachment #505715 - Flags: feedback?(l10n) → feedback+
(Assignee)

Updated

7 years ago
Priority: P5 → P2
(In reply to comment #31)
> Comment on attachment 505715 [details] [diff] [review]
> Update shipped-locales
> 
> This is closer, but there are some missing parameters to actually be able to
> hard-code platform exceptions. Should probably be app and branch.

I think you're saying that you want to be able to hardcode platform exceptions per-app/branch? Is that right?
(Reporter)

Comment 33

7 years ago
app and branch should be the two params giving us the same flexibility we have with a hand-rolled shipped-locales, right?

If so, let's key off of that.
Comment on attachment 505715 [details] [diff] [review]
Update shipped-locales

Can you do the l10nRevisionFile path expansion upfront and pass that to tagRepo rather than configDir?

I'll also echo what Pike said -- we need to maintain platform exceptions per product/branch.
Attachment #505715 - Flags: feedback?(bhearsum) → feedback+
(Assignee)

Comment 35

7 years ago
Created attachment 508328 [details] [diff] [review]
Update shipped-locales

What I've understand from the discussion in this bug and IRC.

1) Sometimes we have a situation, when we need to ship/not ship some locales and this is a last minute decision
2) Some locales could be delivered not for all platforms (platform exceptions)
3) locale/platform exceptions may vary for branches/appName.

Instead of hard coding platform/locale exceptions (per appName/branch) I'd suggest the following approach.

1) l10n drivers maintain shipped-locales file (and platform exceptions) on default branch
2) releng updates shipped-locales file on relbranch according to l10n-revisions file:
  * if a locale is in l10n-revisions file, but not in shipped-locales (last minute addition): add this locale to shipped-locales without specifying platforms
  * if a locale is in shipped-locales, but not in l10n-revisions file (last minute removal): remove the locale from shipped-locales

This is an intermediate solution until we have combined l10n-revisions/shipped-locales file and releng automation doesn't depend on shipped-locales file.
Attachment #505715 - Attachment is obsolete: true
Attachment #508328 - Flags: feedback?(l10n)
Attachment #508328 - Flags: feedback?(bhearsum)
(Reporter)

Comment 36

7 years ago
Comment on attachment 508328 [details] [diff] [review]
Update shipped-locales

I don't think this is worth doing at this point.

l10n-merge on betas and DONTBUILD make shipped-locales much less of a hassle to edit today, and this is the third or forth different process, tuned to implementation details. And those aren't good guides for processes.

I suggest WONTFIX.
Attachment #508328 - Flags: feedback?(l10n) → feedback-
(Assignee)

Updated

7 years ago
Attachment #508328 - Flags: feedback?(bhearsum)
(Assignee)

Comment 37

7 years ago
(In reply to comment #36)
> I suggest WONTFIX.

I agree with you. It would be very hard to make a (automated) decision here, especially when we have locale exceptions per branch/app.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WONTFIX
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.