1256955 - provide ability to correlate release promotion releases with their respective l10n changeset

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Jordan Lund (:jlund)]

one of changes we have made with release promotion is that we no longer tag gecko or l10n repos. This has a side effect of not clearly expressing which revisions we made our repacks from.

with privs you can see the list of locales to their respective revisions: https://ship-it.mozilla.org/releases/Firefox-46.0b1-build8/l10n

unfortunately, that's not that public or transparent :)

Comment 1

[Jordan Lund (:jlund)]

rail mentioned maybe using the l10n dashboard: https://l10n.mozilla.org/shipping/dashboard

Though I'm not too familiar how you can verify sign-offs/revisions to what ends up in the l10n-cset list in ship-it. maybe lizzard, you could shed some light to interested people as to how to how to know which release uses which l10n cset for a given locale?

if all else fails, we could also create a task that digests the l10n-cset list from release-runner and puts an artifact in the releases/candidates location via beetmover.. ?

Comment 2

[Chris AtLee [:catlee]]

generate a signed artifact listing the l10n changesets used, and upload it?

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

Maybe we could put it into the note that gets emailed to r-d when we start a build from ship-it.    
For now, we could put it by hand, ugh.    

The source of the l10n changesets is here: 
https://l10n.mozilla.org/shipping/milestones

We generate a new one for every build (not ideal but it's what we've got)  by clicking a button and then clicking around a bit to get to something like this:
https://l10n.mozilla.org/shipping/about-milestone/fx46_beta_b2

Comment 4

[Rail Aliiev [:rail]]

plain text:

https://l10n.mozilla.org/shipping/l10n-changesets?ms=fx46_beta_b1
https://l10n.mozilla.org/shipping/l10n-changesets?ms=fennec46_beta_b1

There was a JSON endpoint IIRC

Comment 5

[Chris Coulson]

Parsing a publicly available list of changesets when we generate our tarballs wouldn't really be a problem for us.

Putting the list in the note that gets emailed to r-d isn't ideal - whilst I can see this, none of my colleagues or other contributors can, which means I end up being the only person who can generate builds in Ubuntu.

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Ah, I see. Sylvestre has some more details here that should help, I think he will comment to share it when he gets a moment free this week. 

Here is a sample json endpoint, https://l10n.mozilla.org/shipping/json-changesets?ms=fennec46_beta_b2&platforms=android&multi_android-multilocale_repo=releases%2Fmozilla-beta&multi_android-multilocale_rev=default&multi_android-multilocale_path=mobile%2Fandroid%2Flocales%2Fmaemo-locales

Comment 7

[Mike Hommey [:glandium]]

Why not take this from a different angle, and generate "source" tarballs for l10n, like we generate source tarballs for Firefox?

Comment 8

[Chris Coulson]

That would probably work for us.

Is the plan to use this process for the 46.0 release as well?

Comment 9

[Rail Aliiev [:rail]]

We can also publish l10n-chagesets.txt as a part of release.

Comment 10

[Francesco Lodolo [:flod]]

I know it's an edge case, but what about people building localized versions of Firefox on their own? How are they supposed to know which l10n changeset to pick? I confess that I sometimes check l10n repos to see which changeset is shipping, especially to double check if a sign-off was accepted and in which build it ended up.

I've received an explicit question about it (why is there no tag for fx46b*?), and the first info I got is that we tag only after the build has shipped, later I was pointed to this bug.

Comment 11

[Axel Hecht [:Pike]]

I think that not tagging l10n repos is a good thing, though I had liked that to be explicitly communicated.

For now, using the l10n milestones from the dashboard is a good idea, but in the end we should all use Sylvestre's public api. Backing up the existing needinfo, Sylvestre, do you have a bug tracking your work there?

Comment 12

[Chris Coulson]

(In reply to Axel Hecht [:Pike] from comment #11)
> I think that not tagging l10n repos is a good thing, though I had liked that
> to be explicitly communicated.
> 

Yeah, this came as quite a surprise to us.

Also, it's not clear to me where we get the mozilla-beta changeset from either - I see there are 46.0b4 candidates in https://ftp.mozilla.org/pub/firefox/candidates/46.0b4-candidates/build1/, but no tag in    http://hg.mozilla.org/releases/mozilla-beta.

And does anyone know the answer to my question in comment 8?

Comment 13

[Rail Aliiev [:rail]]

(In reply to Chris Coulson from comment #8)
> Is the plan to use this process for the 46.0 release as well?

Yes.

(In reply to Chris Coulson from comment #12)
> Also, it's not clear to me where we get the mozilla-beta changeset from
> either - I see there are 46.0b4 candidates in
> https://ftp.mozilla.org/pub/firefox/candidates/46.0b4-candidates/build1/,
> but no tag in    http://hg.mozilla.org/releases/mozilla-beta.

It will be tagged after we ship it.

I'm going to look at this bug and publish l10n changesets to the candidates directory.

Comment 14

[Chris Coulson]

(In reply to Rail Aliiev [:rail] from comment #13)
> (In reply to Chris Coulson from comment #8)
> > Is the plan to use this process for the 46.0 release as well?
> 
> Yes.
> 
> (In reply to Chris Coulson from comment #12)
> > Also, it's not clear to me where we get the mozilla-beta changeset from
> > either - I see there are 46.0b4 candidates in
> > https://ftp.mozilla.org/pub/firefox/candidates/46.0b4-candidates/build1/,
> > but no tag in    http://hg.mozilla.org/releases/mozilla-beta.
> 
> It will be tagged after we ship it.
> 
> I'm going to look at this bug and publish l10n changesets to the candidates
> directory.

Thanks.

We normally produce Ubuntu builds early enough so that we can have them tested and ready to ship when you do. If mozilla-release isn't tagged until after you ship and there's no other way to discover the changeset used to create the build until this point, then I'm not sure how we'll be able to do this.

Comment 15

[Mike Hommey [:glandium]]

(In reply to Chris Coulson from comment #14)
> We normally produce Ubuntu builds early enough so that we can have them
> tested and ready to ship when you do. If mozilla-release isn't tagged until
> after you ship and there's no other way to discover the changeset used to
> create the build until this point, then I'm not sure how we'll be able to do
> this.

There is a .txt file in the candidates directory telling you exactly what was used.
https://ftp.mozilla.org/pub/firefox/candidates/46.0b4-candidates/build1/linux-x86_64/en-US/firefox-46.0b4.txt

In fact, that file exists everywhere there is a Firefox release (nightly, aurora, beta, release, esr, candidates, etc.)

The problem is there is no equivalent for the l10n repo.

Comment 16

[Chris Coulson]

(In reply to Mike Hommey [:glandium] from comment #15)
> (In reply to Chris Coulson from comment #14)
> > We normally produce Ubuntu builds early enough so that we can have them
> > tested and ready to ship when you do. If mozilla-release isn't tagged until
> > after you ship and there's no other way to discover the changeset used to
> > create the build until this point, then I'm not sure how we'll be able to do
> > this.
> 
> There is a .txt file in the candidates directory telling you exactly what
> was used.
> https://ftp.mozilla.org/pub/firefox/candidates/46.0b4-candidates/build1/
> linux-x86_64/en-US/firefox-46.0b4.txt
> 
> In fact, that file exists everywhere there is a Firefox release (nightly,
> aurora, beta, release, esr, candidates, etc.)
> 
> The problem is there is no equivalent for the l10n repo.

Ah, thanks. I didn't see that

Comment 17

[Rail Aliiev [:rail]]

Or JSON: https://ftp.mozilla.org/pub/firefox/candidates/46.0b4-candidates/build1/linux-x86_64/en-US/firefox-46.0b4.json

Comment 18

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1256955 - provide ability to correlate release promotion releases with their respective l10n changeset r=jlund a=release DONTBUILD

Review commit: https://reviewboard.mozilla.org/r/44385/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/44385/

Comment 19

[Rail Aliiev [:rail]]

Attachment: PR for releasetasks

Comment 20

[Jordan Lund (:jlund)]

Comment on attachment 8738247 [details]
MozReview Request: Bug 1256955 - provide ability to correlate release promotion releases with their respective l10n changeset r=jlund a=release DONTBUILD

https://reviewboard.mozilla.org/r/44385/#review41149

r+ with typo fix: s/l10n_changesets.tmpl/l10n_changesets.yml.tmpl

Comment 21

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d924e3a25a7d

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8738247 [details]
MozReview Request: Bug 1256955 - provide ability to correlate release promotion releases with their respective l10n changeset r=jlund a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/d924e3a25a7d
https://hg.mozilla.org/releases/mozilla-aurora/rev/8f15683d56c2
https://hg.mozilla.org/releases/mozilla-beta/rev/9d5dacb658d2
https://hg.mozilla.org/releases/mozilla-esr45/rev/f16e4b7561f9

Comment 23

[Rail Aliiev [:rail]]

I think we are on track here.

Comment 24

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1256955 - tools: provide ability to correlate release promotion releases with their respective l10n changeset r=jlund

Review commit: https://reviewboard.mozilla.org/r/44699/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/44699/

Comment 25

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d924e3a25a7d

Comment 26

[Rail Aliiev [:rail]]

not ready yet

Comment 27

[Jordan Lund (:jlund)]

Comment on attachment 8738806 [details]
MozReview Request: Bug 1256955 - tools: provide ability to correlate release promotion releases with their respective l10n changeset r=jlund

https://reviewboard.mozilla.org/r/44699/#review41547

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8738806 [details]
MozReview Request: Bug 1256955 - tools: provide ability to correlate release promotion releases with their respective l10n changeset r=jlund

https://hg.mozilla.org/build/tools/rev/1aaed49b263a

Comment 29

[Rail Aliiev [:rail]]

Let's call this done: http://ftp.mozilla.org/pub/firefox/candidates/46.0b10-candidates/build1/l10n_changesets.txt


FTR, this file is not going to be published to the releases directory, similar to other "info" files.

Comment 30

[Mike Hommey [:glandium]]

(In reply to Rail Aliiev [:rail] from comment #29)
> Let's call this done:
> http://ftp.mozilla.org/pub/firefox/candidates/46.0b10-candidates/build1/
> l10n_changesets.txt
> 
> 
> FTR, this file is not going to be published to the releases directory,
> similar to other "info" files.

FWIW, I already didn't think it was very convenient that the .txt file was not in the releases/ directory, but didn't really care because we have the tarballs, and thus the info is not /very/ useful, but for the l10n changesets, it means I'll need to find the right buildn directory corresponding to a tarball to then download the right l10n sources.

Comment 31

[matteo.bernardini]

sorry if I'm missing something but does this bug apply also to thunderbird?
I'm asking because it seems since 45.1b1 l10n aren't tagged for it anymore too and there's no related l10n_changesets.txt in the related candidates directory...

Comment 32

[Rail Aliiev [:rail]]

They should be tagged, see https://hg.mozilla.org/releases/l10n/mozilla-beta/ast/rev/a4dc899c6d18 and the log of the l10n builder http://ftp.mozilla.org/pub/thunderbird/candidates/45.1b1-candidates/build1/logs/release-comm-beta-thunderbird_tag_l10n-unknown-bm72-build1-build3.txt.gz

Comment 33

[matteo.bernardini]

thanks! it seems that all beta/release tags that once are available, for example, here https://hg.mozilla.org/releases/l10n/mozilla-release/it/tags with this beta aren't anymore in one place as for 45.1b1 the tag is available here https://hg.mozilla.org/releases/l10n/mozilla-beta/it/tags
will also thunderbird follow the same standard as firefox and have changesets available in a similar path for versions >= 46.x? I was thinking if something like the following will apply... http://ftp.mozilla.org/pub/thunderbird/candidates/46.0b1-candidates/build1/l10n_changesets.txt