1297519 - Generate multilocale Firefox snaps in beta

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Snaps should include all shipped locales. We can probably schedule only nightly CI and release tasks for these (because it would require to have all repacks and XPIs ready).

Comment 1

[Rail Aliiev [:rail]]

Attachment: snapcraft.diff

The idea is have a docker image and a script to generate snap images.

I put all related files into a single directory. We can either generate an image and publish it to dockerhub, or generate it runtime. 

There will be a task generating a snap and its checksum, a task signing the checksum, and a beetmover task to copy these to the candidates directory.

Snaps will be enabled per branch (another branch config variable passed down to releasetasks).

Something like this.

Comment 2

[Axel Hecht [:Pike]]

Do we need to remove the firefox-l10n.js pref file that sets en-US? IIRC, the matchOS pref and the general.useragent.locale need both to be tweaked for matchOS to work.

Comment 3

[Rail Aliiev [:rail]]

I tested it and it works without any further customization, LANG=xx_YY /snap/bin/firefox works just fine.

Comment 4

[Rail Aliiev [:rail]]

The easiest way to test it is to run the following command and then install the generated snap file:

docker run -e VERSION=49.0b7 -e BUILD_NUMBER=1 \
  -e CANDIDATES_DIR=https://archive.mozilla.org/pub/firefox/candidates \
  -v $(pwd):/snapcraft -v $(pwd)/home:/home \
  -ti rail/snap-dev sh -c "cd /snapcraft && bash runme.sh"

This will generate ./home/worker/artifacts/firexo*.snap, which can be installed by running sudo snap install ./home/worker/artifacts/firexo*.snap

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8786073 [details] [diff] [review]
snapcraft.diff

from an overall approach perspective, this looks sane to me. ++ to having what you have defined in tree.

we discussed over video of using hg.m.o subdir archive url locations for retrieving this since we only need to grab this once per release.

have you tested this either through tc task creator yet? Locally maybe? Be cool to see inputs/outputs/logs for this.

Comment 6

[Rail Aliiev [:rail]]

Attachment: Bug 1297519 - Generate multilocale Firefox snaps

Review commit: https://reviewboard.mozilla.org/r/79090/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/79090/

Comment 7

[Rail Aliiev [:rail]]

Attachment: Bug 1297519 - Generate multilocale Firefox snaps

Review commit: https://reviewboard.mozilla.org/r/79094/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/79094/

Comment 8

[Rail Aliiev [:rail]]

There is nothing using this flags yet, but soon will be. Just need to land these before I can use them in releasetasks.

Comment 9

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8791825 [details]
Bug 1297519 - Generate multilocale Firefox snaps

The changes in release-runner make sense to me.

Comment 10

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8791824 [details]
Bug 1297519 - Generate multilocale Firefox snaps

So do the variable initialization in buildbot-configs

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8791825 [details]
Bug 1297519 - Generate multilocale Firefox snaps

https://hg.mozilla.org/build/tools/rev/46ece28c68a5

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8791824 [details]
Bug 1297519 - Generate multilocale Firefox snaps

https://hg.mozilla.org/build/buildbot-configs/rev/4354ce6042f5

Comment 13

[Rail Aliiev [:rail]]

Attachment: Bug 1297519 - Generate multilocale Firefox snaps , a=release DONTBUILD

Review commit: https://reviewboard.mozilla.org/r/79290/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/79290/

Comment 14

[Rail Aliiev [:rail]]

The patch requires https://github.com/mozilla/releasetasks/compare/master...rail:snap?expand=1 to work.

Tested on jamun, see https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=c7f77d4cd7da9f8ee19eca47a67b565d0a9f6041&filter-searchStr=snap and the corresponding graphs.

Comment 15

[Justin Wood (:Callek)]

Comment on attachment 8792039 [details]
Bug 1297519 - Generate multilocale Firefox snaps , a=release DONTBUILD

https://reviewboard.mozilla.org/r/79290/#review77838

I'm not too familair with snap/snapcraft, but the overall logic, while fragile, looks sane [and no less fragile than many of our other tooling in this area].

::: release/docker/firefox-snap/Dockerfile:5
(Diff revision 1)
> +FROM ubuntu:16.04
> +
> +RUN apt-get update -q &&  apt-get install -qy software-properties-common && apt-get clean
> +RUN add-apt-repository ppa:snappy-dev/snapcraft-daily
> +RUN apt-get update && apt-get install -qy snapcraft bzip2 curl && apt-get clean

not as enticed by a ppa for -dev/-daily and then installing that package without any sort of version pinning.

Feels like we'd be asking for trouble on docker regens (since an unrelated change could take a newer unstable snapshot that breaks us).

I won't block on this since we're still getting our feet wet here. :-)

::: release/docker/firefox-snap/Makefile:9
(Diff revision 1)
> +
> +build:
> +	docker build -t $(FULL_IMAGE_NAME) --no-cache --rm .
> +
> +push:
> +	docker push $(FULL_IMAGE_NAME):latest

can we tag each push with some sort of buildID as well, so rollbacks are easy?

Comment 16

[Rail Aliiev [:rail]]

(In reply to Justin Wood (:Callek) from comment #15)
> not as enticed by a ppa for -dev/-daily and then installing that package
> without any sort of version pinning.

It needed this at some point to get latest snap/snapcraft. I can check if Ubuntu got the updates in the main repo, so we can get rid of the PPA.

> can we tag each push with some sort of buildID as well, so rollbacks are
> easy?

We won't be using tags in any case (we use sha256). Pushing :latest is just faster (no need to push everything).

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8792039 [details]
Bug 1297519 - Generate multilocale Firefox snaps , a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79290/diff/1-2/

Comment 18

[Rail Aliiev [:rail]]

https://reviewboard.mozilla.org/r/79290/diff/1-2/ addresses the PPA comment

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4606672d8efe
Generate multilocale Firefox snaps r=Callek, a=release DONTBUILD

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 8792039 [details]
Bug 1297519 - Generate multilocale Firefox snaps , a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/4606672d8efe
https://hg.mozilla.org/releases/mozilla-aurora/rev/79adb317eca4

Comment 21

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/4606672d8efe

Comment 22

[Rail Aliiev [:rail]]

Forgot the magic words

Comment 23

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/4354ce6042f5

Comment 24

[Rail Aliiev [:rail]]

Attachment: PR for releasetasks

Comment 25

[Rail Aliiev [:rail]]

Comment on attachment 8794390 [details] [review]
PR for releasetasks

r+ed in the PR