1379773 - Funsize should download mar tools from current branch

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Rail Aliiev [:rail]]

https://dxr.mozilla.org/mozilla-central/rev/91c943f7373722ad4e122d98a2ddd6c79708b732/taskcluster/docker/funsize-update-generator/scripts/funsize.py#139-145 uses mozilla-central based mar tools to generate partials. We should probably use mar tools from the same branch/revision.

Also at some point we are going to drop bzip2 compression, so we may end up with bzip2-compressed "from" complete mar and lzma-compressed "to" mar.


In releasetasks we can pass the location of mar tools similar to
https://github.com/mozilla-releng/releasetasks/blob/23d4da604e72aaff98e545711bf9494f46be7c60/releasetasks/templates/desktop/l10n.yml.tmpl#L33 and modify funsize.py to use the env variable if it's set.

Comment 2

[Simon Fraser [:sfraser] ⌚️GMT]

This caused some confusion today, so will revisit it.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I had a quick go at fixing this while we worked on 59.0b1 issues, independently coming to the same solution as Rail. See
  https://gist.github.com/nthomas-mozilla/2671a1a855537ff0e158ef0d29d116a1
  https://gist.github.com/nthomas-mozilla/c1daabfa2eba2113fa17f3aab9d0f2e6
They're untested but might be helpful.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Plus we'd need something in-tree for 59.0 and higher.

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

Do releasetasks use a docker container built from that repo, or the one from central? We've seen that a new version of the container landing in inbound/autoland gets built, and then used in central before it's merged in, so I'm unsure how it's build in other repos.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #5)
> Do releasetasks use a docker container built from that repo, or the one from
> central? We've seen that a new version of the container landing in
> inbound/autoland gets built, and then used in central before it's merged in,
> so I'm unsure how it's build in other repos.

Releasetasks is only used by release runner, and not from any docker container. We'll need to release a new version and bump https://github.com/mozilla/build-puppet/blob/50e69b4d199dc3105e06e152ecd5dacca89a1ec2/modules/releaserunner/manifests/init.pp#L52 once a fix for it is landed.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #6)
> (In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #5)
> > Do releasetasks use a docker container built from that repo, or the one from
> > central? We've seen that a new version of the container landing in
> > inbound/autoland gets built, and then used in central before it's merged in,
> > so I'm unsure how it's build in other repos.
> 
> Releasetasks is only used by release runner, and not from any docker
> container. We'll need to release a new version and bump
> https://github.com/mozilla/build-puppet/blob/
> 50e69b4d199dc3105e06e152ecd5dacca89a1ec2/modules/releaserunner/manifests/
> init.pp#L52 once a fix for it is landed.

Ergh, sorry - I just realized you were talking about a Funsize container, not a releasetasks one. It looks like releasetasks generates its own funsize image as part of the graph (per https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/funsize_image.yml.tmpl (and as seen in https://tools.taskcluster.net/groups/J8pxHMI6Sd2iHxrFSiONoQ/tasks/fgHdTIhxQOGALRLr3aeVjg/details and https://tools.taskcluster.net/groups/J8pxHMI6Sd2iHxrFSiONoQ/tasks/FG_ftFVgQCuxTLoP3akmUA/details).

Comment 8

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: 1379773.diff

A draft patch to allow for setting the download urls for mar/mbsdiff in the task definition. This will let us use the ones from the current build when generating a task graph, and default to the latest-mozilla-central versions if not specified.

Does this seem like a reasonable approach? We'd end up with a partials payload like this:

    {
        "to_mar": "https://queue.taskcluster.net/v1/task/QLhs5IOMR3agwLgLgIgVhA/artifacts/public/build/target.complete.mar",
        "dest_mar": "target.partial-1.mar",
        "locale": "en-US",
        "from_mar": "https://archive.mozilla.org/pub/firefox/nightly/2018/03/2018-03-07-22-01-00-mozilla-central/firefox-60.0a1.en-US.linux-x86_64.complete.mar",
        "update_number": 1,
        "platform": "linux64",
        "branch": "mozilla-central",
        "mar_binary": "https://queue.taskcluster.net/v1/task/aDVCyooBT9O2z8ComppTBA/runs/0/artifacts/public/build/host/bin/mar",
        "mbsdiff_binary": "https://queue.taskcluster.net/v1/task/aDVCyooBT9O2z8ComppTBA/runs/0/artifacts/public/build/host/bin/mbsdiff"
      },

So it's bigger, but not by much. It allows us to keep a default, and add different versions to each partial generated if needed, for odd release situations.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 8957174 [details] [diff] [review]
1379773.diff

Review of attachment 8957174 [details] [diff] [review]:
-----------------------------------------------------------------

This looks reasonable to me. Hopefully we can remove the default eventually, to avoid accidentally using the wrong tools? I know it's much easier to have it now, until everything is always setting it.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b42a6edd2645
Update funsize to allow mar/mbsdiff definitions r=bhearsum

Comment 11

[Natalia Csoregi [:nataliaCs]]

Backed out for flake failure on scripts/funsize.py:229

backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/41889754fc469678e9becaa993e47424300fa782

push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b42a6edd264592d613b01b0dca11393013741ce4

failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=170251233&repo=mozilla-inbound&lineNumber=242

[task 2018-03-26T10:45:04.981Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenv/bin/python2.7
[task 2018-03-26T10:45:04.981Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenv/bin/python
[task 2018-03-26T10:45:06.545Z] Installing setuptools, pip, wheel...done.
[task 2018-03-26T10:45:07.620Z] running build_ext
[task 2018-03-26T10:45:07.620Z] building 'psutil._psutil_linux' extension
[task 2018-03-26T10:45:07.620Z] creating build
[task 2018-03-26T10:45:07.620Z] creating build/temp.linux-x86_64-2.7
[task 2018-03-26T10:45:07.620Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2018-03-26T10:45:07.621Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-03-26T10:45:07.621Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-03-26T10:45:07.621Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2018-03-26T10:45:07.621Z] creating build/lib.linux-x86_64-2.7
[task 2018-03-26T10:45:07.621Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2018-03-26T10:45:07.622Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2018-03-26T10:45:07.622Z] building 'psutil._psutil_posix' extension
[task 2018-03-26T10:45:07.622Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-03-26T10:45:07.623Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-03-26T10:45:07.623Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2018-03-26T10:45:07.623Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2018-03-26T10:45:07.623Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2018-03-26T10:45:07.623Z] 
[task 2018-03-26T10:45:07.623Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-03-26T10:45:37.125Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/taskcluster/docker/funsize-update-generator/scripts/funsize.py:229:100 | line too long (140 > 99 characters) (E501)
[task 2018-03-26T10:45:37.125Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/taskcluster/docker/funsize-update-generator/scripts/funsize.py:230:100 | line too long (122 > 99 characters) (E501)
[task 2018-03-26T10:45:37.125Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/taskcluster/docker/funsize-update-generator/scripts/funsize.py:231:100 | line too long (129 > 99 characters) (E501)
[taskcluster 2018-03-26 10:45:37.526Z] === Task Finished ===
[taskcluster 2018-03-26 10:45:37.526Z] Unsuccessful task run with exit code: 1 completed in 277.659 seconds

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/00bfa0099015
Update funsize to allow mar/mbsdiff definitions r=bhearsum

Comment 13

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/00bfa0099015