Closed Bug 1386995 Opened 3 years ago Closed 3 years ago

Fennec 55.0 RC is missing locales

Categories

(Release Engineering :: Release Automation: Other, defect, P1)

defect

Tracking

(firefox55 wontfix, firefox56 fixed, firefox57 affected)

RESOLVED WONTFIX
Tracking Status
firefox55 --- wontfix
firefox56 --- fixed
firefox57 --- affected

People

(Reporter: mtabara, Assigned: jlorenzo)

References

Details

Attachments

(1 file)

So this feels really wrong. I looked at the Fennec taskgraph[1] this morning and noticed it only had 25 tasks instead of usual ~300. Then S3 seemed wrong as no other locales apart from "en-US","multi" are there[2]. I looked up for previous releaes and it sounds like some of them have it, some don't. But certainly 54.0 had them[3]. Could it be a bug we introduced accidentally in beta before the merge yesterday? Or something alike? For example the 54.0 graph looks like this[4]

The logs[5] say "Filter filter_target_tasks pruned 5378 tasks (22 remain)" so this might be a fallback from mack taskgraph changes.

Jlorenzo reproduced this locally.

jlorenzo> eh `SystemError: Parent module 'voluptuous' not loaded, cannot perform relative import`
10:24:46 <jlorenzo> (on mozilla-release) 
10:26:24 <jlorenzo> ./mach bootstrap made it work again 
10:27:31 <jlorenzo> I repro the 22 tasks locally



[1]: https://tools.taskcluster.net/groups/2cSwae3-TiyjszbRzvixwQ
[2]: http://archive.mozilla.org/pub/mobile/candidates/55.0-candidates/build1/android-api-15/
[3]:  http://archive.mozilla.org/pub/mobile/candidates/54.0-candidates/build1/android-api-15/
[4]: https://tools.taskcluster.net/groups/19vsX0HUSX2FdQtq0PzCYg
[5]: https://tools.taskcluster.net/groups/sicHSR4cQxidBN_hXoJLMw/tasks/2cSwae3-TiyjszbRzvixwQ/runs/0/logs/public%2Flogs%2Flive.log#L838
Bug 1352477 possibly related.
See Also: → 1352477
See Also: → 1386776
I confirm that's a fallout of bug 1352477. I missed mozilla-release at [1]

[1] https://hg.mozilla.org/mozilla-central/rev/5533bb014569#l1.15
Depends on: 1352477
See Also: 1352477
Assignee: nobody → jlorenzo
Blocks: 1352477
No longer depends on: 1352477
Comment on attachment 8893292 [details]
Bug 1386995 - Fennec 55.0 RC is missing single locale builds

https://reviewboard.mozilla.org/r/164346/#review169682
Attachment #8893292 - Flags: review?(mtabara) → review+
See Also: → 1352477, 1372911
Downgrading level of importance as we don't ship l10n builds to GP so this is not blocking the release. Will follow-up in a few with details.
Severity: critical → normal
tl;dr:

1. Problem: we're missing the l10n builds from S3[1].
2. Cause: a small fallback from tcmigration
3. Why is this not blocking the release?: 
* because we don't actually ship l10n builds to Google Play Store.
* because re-spinning Fennec 55.0 RC2 would complicate the release calendar timeline
4. Resolution: 
* we ignore for now and we land a fix for central/beta so that we don't hit this again for 56.0 RC in September.
* possibly drop a text file along those missing single-locale folders pointing to this bug and letting them know why they are missing?
5. Lessons learned - more staging releases, especially after tcmigration

More detailed.

1. Problem

Fennec graph this morning did look surprisingly small. 25 tasks as opposed to our usual 300 doesn't sound too good. For example this[3] is how the 54.0 RC graph looked liked and this[4] is where the artifacts were. `en-US` and `multi` locales are present but all single-locale l10n stuff is missing. 

2. Cause

Since we haven't had any other Fennec RC this week, this was likely a fallback from either tcmigration or something that got uplifted to beta just before we did the merge on Monday (beta => release). However, we haven't had the problem on 55.0b14 on Monday which meant this is strictly related to release and not to beta. The logs stated "Filter filter_target_tasks pruned 5378 tasks (22 remain)" which narrows the possibilities. Johan reproduced this locally on top of mozilla-release and noticed this might be related to bug 1386776 where we have l10n builds running across beta and followed-up with patch[7]. We almost wanted to land that patch and uplift to beta/release and respin another Fennec 55 RC2 which has the l10n builds as well but after chatting with RelMan/QE we stopped. 

3. Why is this not blocking the release?

Because we don't actually ship those l10n builds to the Google Play. Google play only cares about the `multi` locale build and *not* about the `single-locale` ones. Single-locales builds are meant for all those users that don't find their locale within the `multi` build. For those users - though it's still unclear to me yet - if they really want that particular locale, they need to set their phones to grab the updates from Balrog. However, if true, this is most likely happening only on Nightly and don't auto update anyway, as we currently don't have any Fennec blobs for beta/release. For beta/release users, what happens in this case is that they actually walk to the S3 buckets[8] themselves from their mobile phones and download the .apk from the l10n single locale folders (I guess there's some security munging involved beforehand 'please accept apk from other sources then Play Store' or something alike).

4. Resolution

* we're gonna land a fix for central/beta so that we don't hit this on next 56.0 RC release in September.
* we are considering dropping a text file along the `en-US` and `multi` locales that explains why these l10n builds are missing (e.g. [8]for 54.0)

5. Lessons learned
* we should increase the bus factor on staging releases and improve docs[5] up to the point where they are copy-paste commands so anyone can do this quick and fast as we need them
* better glancing at the graphs once they are generated

[1]: http://archive.mozilla.org/pub/mobile/candidates/55.0-candidates/build1/android-api-15/
[2]: https://tools.taskcluster.net/groups/2cSwae3-TiyjszbRzvixwQ
[3]: https://tools.taskcluster.net/groups/19vsX0HUSX2FdQtq0PzCYg
[4]: http://archive.mozilla.org/pub/mobile/candidates/54.0-candidates/build1/android-api-15/
[5]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/setup-staging-release.md
[6]: https://tools.taskcluster.net/groups/sicHSR4cQxidBN_hXoJLMw/tasks/2cSwae3-TiyjszbRzvixwQ/runs/0/logs/public%2Flogs%2Flive.log#L838
[7]: https://reviewboard.mozilla.org/r/164346/diff/1#index_header
[8]: http://archive.mozilla.org/pub/mobile/releases/54.0/android-api-15/
Forgot to mention earlier the most important thing, thanks for the help in dealing with this :jlorenzo, :jcristau and :ioanachiorean!
We still haven't confirmed the resolution from https://bugzilla.mozilla.org/show_bug.cgi?id=1386995#c6 but if we do decide to go with some text files dropping in the S3 production buckets, is this something we can do? I was thinking more of this and not sure there's a way to hack a dummy beetmover task to do it because of CoT errors, if run outside the tree. 

So if it turns out we need some dummy files there to let users know of this situations, is that something we can do from AWS S3 console? I know we don't own the S3 buckets so that might be an issue.
Flags: needinfo?(nthomas)
Keywords: leave-open
Comment on attachment 8893292 [details]
Bug 1386995 - Fennec 55.0 RC is missing single locale builds

Jlorenzo landed on autoland + 
https://hg.mozilla.org/releases/mozilla-beta/rev/1ec102ceb2da594ca03d8a7af51e11e4900c38eb

This change will ride the trains to get into release in 56.0
Attachment #8893292 - Flags: checked-in+
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9f50b704b98e
Fennec 55.0 RC is missing single locale builds r=mtabara
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #8)
> We still haven't confirmed the resolution from
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386995#c6 but if we do decide
> to go with some text files dropping in the S3 production buckets, is this
> something we can do? I was thinking more of this and not sure there's a way
> to hack a dummy beetmover task to do it because of CoT errors, if run
> outside the tree. 

We can't use the console but can manually upload a text file using the same credentials as beetmover uses. It's nice to make sure we set the Content-Type to text/plain so it displays in browser rather than downloads.

It looks like the 'push to releases' beetmover job that pushes into releases would copy it over, so long as we avoid the excludes list, so call it README rather than README.txt. eg 54.0 log at http://mozilla-release-logs.s3.amazonaws.com/mozilla-release/fennec-54.0/build2/%5Bbeetmover%5D_fennec_mozilla-release_push_to_releases-all-jYlvYJfbR869jwuaU8ocDA-0
Flags: needinfo?(nthomas)
(In reply to Nick Thomas [:nthomas] from comment #13)
> (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #8)
> > We still haven't confirmed the resolution from
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1386995#c6 but if we do decide
> > to go with some text files dropping in the S3 production buckets, is this
> > something we can do? I was thinking more of this and not sure there's a way
> > to hack a dummy beetmover task to do it because of CoT errors, if run
> > outside the tree. 
> 
> We can't use the console but can manually upload a text file using the same
> credentials as beetmover uses. It's nice to make sure we set the
> Content-Type to text/plain so it displays in browser rather than downloads.
> 
> It looks like the 'push to releases' beetmover job that pushes into releases
> would copy it over, so long as we avoid the excludes list, so call it README
> rather than README.txt. eg 54.0 log at
> http://mozilla-release-logs.s3.amazonaws.com/mozilla-release/fennec-54.0/
> build2/%5Bbeetmover%5D_fennec_mozilla-release_push_to_releases-all-
> jYlvYJfbR869jwuaU8ocDA-0

Sounds good, yeah, wasn't sure if it's okay to use that set of credentials outside the automation but it looks like we have a solution if we need to! Off to jcristau's plate to decide if we really want this to be done for 55.0 Otherwise, we can close this bug as WONTFIX. 
Thanks again :nthomas!
We followed-up with a RC3 and this request hasn't come yet to set that file.
I think we can close this bug as WONTFIX. If RelMan wants it, I'll reopen and proceed with nthomas's solution from https://bugzilla.mozilla.org/show_bug.cgi?id=1386995#c13
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.