Closed Bug 1447263 Opened 6 years ago Closed 6 years ago

Snap: Split build and upload task

Categories

(Release Engineering :: Release Automation: Other, enhancement)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox61 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox61 --- fixed

People

(Reporter: jlorenzo, Assigned: jlorenzo)

References

Details

Attachments

(6 files)

[cloud-tools] PR: Add pushsnapworker instance type
60 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
[build/puppet] Add pushsnap_scriptworker instances
59 bytes, text/x-review-board-request
rail
: review+
jlorenzo
: checked-in+
Details
Bug 1447263 - Snap: Split build and upload task
59 bytes, text/x-review-board-request
rail
: review+
Details
[scriptworker] PR: Support push-snap tasks
55 bytes, text/x-github-pull-request
mozilla
: review+
jlorenzo
: checked-in+
Details | Review
[cot-gpg-keys] Add pushsnapworker-1 key
54 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
[mozilla-releng/pushsnapscript] 14 first commits
97 bytes, text/plain
mozilla
: review+
rail
: review+
jlorenzo
: checked-in+
Details 
Let's use scriptworker to do so.
I added to [1]:
* queue:create-task:low:scriptworker-prov-v1/dep-pushsnap
* project:releng:snapcraft:firefox:mock

I also added to [2]:
* queue:create-task:highest:scriptworker-prov-v1/pushsnap-v1
* project:releng:snapcraft:firefox:beta
* project:releng:snapcraft:firefox:candidate

[1] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A1%3Agecko
[2] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko
Attachment #8962368 - Flags: review?(aki)
Comment on attachment 8962368 [details] [review]
[scriptworker] PR: Support push-snap tasks

r+ with comments (on the pr).
Attachment #8962368 - Flags: review?(aki) → review+
TL;DR: Uploads are done via a git submodule. Credentials are copied to specific locations to make that submodule happy. I looked at other solution. I couldn't find better in a reasonable amount of time.

First of all, I'm sorry I didn't make a PR like Callek smartly did in [1]. I'll do it next time.

Here is an example of the worker passing: [2]. This is the production one against maple because I had to test prod keys and credentials. 



GENERAL FLOW

The worker is very similar to pushapk:
1. It fetches snaps from the snap-repackage task[3] thanks to scriptworker
2. In some future, it will make some sanity checks like signatures (more details in bug 1417960).
3. It delegates the publication to a 3rd party library: snapcraft[4].



FIRST HACK: SNAPCRAFT

snapcraft is a tool to make and upload snaps. It is built in python, but unlike Google's APIs[5], there is no distribution-independent python library. For instance, a specific apt library is needed. I didn't manage to install snapcraft on my Linux distro (Arch Linux). I haven't dared on CentOS 6.
snapcraft doesn't exist as a RPM package, yet. There is some ongoing work[7] for Fedora, but I don't think it contains the tools to make and upload snaps, I think it only contains what's necessary to *run* snaps.

Therefore, I worked around these problems by making snapcraft a submobdule[8] of pushsnapscript. This submodule is linked to the latest stable version: 2.39.2 [9]. To make this submodule work, I installed a subset of the dependencies from [10]. This subset is what's needed to push and release (nothing more) a snap. Imports are done this way[11].

This the first major hack.



SNAPCRAFT: WHAT WERE THE ALTERNATIVES?

A. Implement a client that talks to the REST APIs directly.
However, there are some features in snapcraft that may be useful in the long run. For instance, the client is in charge of creating "delta updates"[12] (aka partials). It also deals with credentials (more details later)

B. Package snapcraft for CentOS 6.
Like said above, there are few attempts to port it on recent RedHat-like systems. Based on the number of dependencies, I didn't dare to try to package it myself. Moreover, I don't have a lot of experience in packaging RedHat from scratch. That solution would have been time consuming

C. Use snapcraft's docker image. 
Snapcraft is shipped as a docker image[13]. We now officially inherit from it at the repackage step (bug 1444898). Using it in the context of scriptworker introduces more complexity which may make the worker fragile. I didn't explore that route yet. This may become easier in the glorious future, if scriptworker are based on docker images.

D. Use an Ubuntu distro
Snapcraft is packaged for Ubuntu (but still may cause issues like the one described in bug 1444898). Aki suggested this solution if we had enough time to set up the whole scriptworker infrastructure under Ubuntu. That seemed like a huge task to me. I preferred not to try it, for now.

E. Use snapcraft embedded in a snap.
This solution was suggested by Canonical. It would be easy to update. However, snapd isn't officially supported on CentOS[14] and like said above, things are just being experimented on RedHat-like systems.



SECOND HACK: CREDENTIALS LOCATION

Snapcraft is getting better[15] at exporting credentials, called macaroons. Sadly, imports are still not there yet. We had to make a file named ~/.config/snapcraft/snacraft.cfg in the initial docker script[16]. The only other option is to copy a file in $CWD/.snapcraft/snacraft.cfg. That's what I did[17].

Apart from this location, original "snacraft.cfg" files are stored on disk, just like it's done on pushapk.



THE MOCK CHANNEL

Just like dep-pushapk, dep-pushsnap will do everything but talk to snap store[18]. This is represented by the mock channel, passed as a scope.



CODE HEALTH

The project has 100% code coverage and has been tested end-to-end like said at the beginning[2]



What do you all think? 



[1] https://github.com/mozilla-releng/addonscript/pull/1
[2] https://tools.taskcluster.net/groups/L0DVme6iTxarFSvLFs7Khg/tasks/Ek9nxppWS3iYGb_4ie15qg/runs/4/logs/public%2Flogs%2Flive_backing.log
[3] For example: https://tools.taskcluster.net/groups/XoR2_FOyTD-C7uRgWMPg4Q/tasks/Mx8bU_JiQ-u-NHP2wWHuOg/runs/0/logs/public%2Flogs%2Flive.log
[4] https://github.com/snapcore/snapcraft
[5] https://github.com/google/google-api-python-client
[6] https://github.com/snapcore/snapcraft/blob/2.39.2/requirements.txt#L10
[7] https://copr.fedorainfracloud.org/coprs/ngompa/snapcore-el7/
[8] https://github.com/mozilla-releng/pushsnapscript/commit/44fcf6c2606417c56ca6ab6f2ed55e6432797faa and https://github.com/mozilla-releng/pushsnapscript/commit/7ec365436f136eb2cd569c2a34aa745b11f9cb83
[9] https://github.com/snapcore/snapcraft/releases/tag/2.39.2
[10] https://github.com/mozilla-releng/pushsnapscript/commit/44fcf6c2606417c56ca6ab6f2ed55e6432797faa#diff-b4ef698db8ca845e5845c4618278f29aR3
[11] https://github.com/mozilla-releng/pushsnapscript/commit/7ec365436f136eb2cd569c2a34aa745b11f9cb83#diff-62d14b5f427f0c76b5db0e636464a08cR16
[12] https://github.com/snapcore/snapcraft/blob/2.39.2/snapcraft/_store.py#L491
[13] https://store.docker.com/community/images/snapcore/snapcraft
[14] https://docs.snapcraft.io/core/install
[15] For instance, we can now use this command for exports https://forum.snapcraft.io/t/better-support-for-other-ci-systems/2968/9
[16] https://searchfox.org/mozilla-central/rev/7e663b9fa578d425684ce2560e5fa2464f504b34/taskcluster/docker/firefox-snap/fetch_macaroons.sh#7
[17] https://github.com/mozilla-releng/pushsnapscript/commit/669c2e19c69bdc43ddc3a9ea22628038a5a5b8a0#diff-62d14b5f427f0c76b5db0e636464a08cR34
[18] https://github.com/mozilla-releng/pushsnapscript/commit/3a8eb4c3de92403b0b5e202a550efe69c79f8d01#diff-62d14b5f427f0c76b5db0e636464a08cR22
Attachment #8963577 - Flags: review?(rail)
Attachment #8963577 - Flags: review?(aki)
Ah I forgot about 1 last hack: I had to install unsquashfs 4.3 (instead of 4.0 in CentOS 6). snapcraft calls the binary directly to get some metadata out of the snap. unsquashfs 4.0 doesn't support xz compression[1] and that's what snapcraft uses[2].

[1] https://superuser.com/questions/919025/what-was-the-squashfs-compression-method/919026
[2] Tested against https://queue.taskcluster.net/v1/task/Mx8bU_JiQ-u-NHP2wWHuOg/runs/0/artifacts/public/build/target.snap, via command given in [1]
We may have some cleanup to do (puppet email)

Thu Mar 29 15:58:15 -0700 2018 Puppet (err): Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install mozilla-squashfs-tools-4.3-0.21.gitaae0aff4.el6' returned 1: Transaction Check Error:
  file /usr/sbin/unsquashfs from install of mozilla-squashfs-tools-4.3-0.21.gitaae0aff4.el6.x86_64 conflicts with file from package squashfs-tools-4.0-5.el6.x86_64

Error Summary
-------------
Wrapped exception:
Execution of '/usr/bin/yum -d 0 -e 0 -y install mozilla-squashfs-tools-4.3-0.21.gitaae0aff4.el6' returned 1: Transaction Check Error:
  file /usr/sbin/unsquashfs from install of mozilla-squashfs-tools-4.3-0.21.gitaae0aff4.el6.x86_64 conflicts with file from package squashfs-tools-4.0-5.el6.x86_64

Error Summary
-------------
Thu Mar 29 15:58:15 -0700 2018 /Stage[main]/Packages::Mozilla::Squashfs_tools/Package[mozilla-squashfs-tools]/ensure (err): change from absent to 4.3-0.21.gitaae0aff4.el6 failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install mozilla-squashfs-tools-4.3-0.21.gitaae0aff4.el6' returned 1: Transaction Check Error:
Comment on attachment 8963577 [details]
[mozilla-releng/pushsnapscript] 14 first commits

(In reply to Johan Lorenzo [:jlorenzo] from comment #15)
> TL;DR: Uploads are done via a git submodule. Credentials are copied to
> specific locations to make that submodule happy. I looked at other solution.
> I couldn't find better in a reasonable amount of time.

I'm not entirely sure why we did a submodule here, rather than install snapcraft
in the virtualenv as a dependency?

> First of all, I'm sorry I didn't make a PR like Callek smartly did in [1].
> I'll do it next time.

I cloned the repo, nuked everything but .git/, then did a `git diff -R > comment`.
Let's see how this goes.

> Here is an example of the worker passing: [2]. This is the production one
> against maple because I had to test prod keys and credentials. 

Cool! We may be able to silence the progress bar somehow, but I'm not too
concerned about it atm.

> GENERAL FLOW
> 
> The worker is very similar to pushapk:
> 1. It fetches snaps from the snap-repackage task[3] thanks to scriptworker
> 2. In some future, it will make some sanity checks like signatures (more
> details in bug 1417960).
> 3. It delegates the publication to a 3rd party library: snapcraft[4].

> FIRST HACK: SNAPCRAFT
> 
> snapcraft is a tool to make and upload snaps. It is built in python, but
> unlike Google's APIs[5], there is no distribution-independent python
> library. For instance, a specific apt library is needed. I didn't manage to
> install snapcraft on my Linux distro (Arch Linux). I haven't dared on CentOS
> 6.
> snapcraft doesn't exist as a RPM package, yet. There is some ongoing work[7]
> for Fedora, but I don't think it contains the tools to make and upload
> snaps, I think it only contains what's necessary to *run* snaps.
> 
> Therefore, I worked around these problems by making snapcraft a
> submobdule[8] of pushsnapscript. This submodule is linked to the latest
> stable version: 2.39.2 [9]. To make this submodule work, I installed a
> subset of the dependencies from [10]. This subset is what's needed to push
> and release (nothing more) a snap. Imports are done this way[11].
> 
> This the first major hack.

Can you use the source python module? https://pypi.python.org/pypi/snapcraft/2.40.1
This would be similar to cloning the release revision, except we would use the
standard python pinning methods that we currently use for all other python modules
in scriptworker and puppet. If the code we're using in the clone is in the python
source module, I'm not entirely sure what the difference would be.

... unless we're unable to install certain dependencies of the source module for
some reason. I'd also be curious if we'd be able to generate a platform-specific
wheel if needed.

If the above is not possible, I'd be curious as to why, but a git submodule
may not be the worst option.

> SNAPCRAFT: WHAT WERE THE ALTERNATIVES?

I'm not thrilled with these alternatives, but the python module doesn't appear
to be listed.

> SECOND HACK: CREDENTIALS LOCATION
> 
> Snapcraft is getting better[15] at exporting credentials, called macaroons.
> Sadly, imports are still not there yet. We had to make a file named
> ~/.config/snapcraft/snacraft.cfg in the initial docker script[16]. The only
> other option is to copy a file in $CWD/.snapcraft/snacraft.cfg. That's what
> I did[17].
> 
> Apart from this location, original "snacraft.cfg" files are stored on disk,
> just like it's done on pushapk.

I think a snapcraft.cfg is fine; it mirrors other scriptworker configs. It's only
used for publishing, and never leaves the scriptworker, right?

> THE MOCK CHANNEL
> 
> Just like dep-pushapk, dep-pushsnap will do everything but talk to snap
> store[18]. This is represented by the mock channel, passed as a scope.

Can we talk to the snap store in a read-only fashion here, or in integration testing?
Doing so may help us make sure we can still access the store.

It's possible we could add an `integration` channel or something?

> CODE HEALTH
> 
> The project has 100% code coverage and has been tested end-to-end like said
> at the beginning[2]

Nice, thank you!

> diff --git b/.gitmodules a/.gitmodules
> new file mode 100644
> index 0000000..ad7404e
> --- /dev/null
> +++ a/.gitmodules
> @@ -0,0 +1,3 @@
> +[submodule "snapcraft"]
> +	path = pushsnapscript/snapcraft
> +	url = https://github.com/snapcore/snapcraft.git

Can we pin this? It looks like the submodule can take a branch.

I don't think we should be taking external unpinned tip-of-master; we should be
explicit when we update our deps. This is especially important because we're
using a non-public API (_store has a single initial underscore, meaning it's
intended to be private only). We have no guarantees against future bustage.

> diff --git b/README.md a/README.md
> new file mode 100644
> index 0000000..e69de29
> diff --git b/examples/config.example.json a/examples/config.example.json
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ a/examples/config.example.json
> @@ -0,0 +1 @@
> +{}
> diff --git b/examples/task.example.json a/examples/task.example.json
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ a/examples/task.example.json
> @@ -0,0 +1 @@
> +{}

Let's populate these .md and .json files, as well as scopes.md :)
If things are still in flux, this can wait, but ideally we should do this
before initial deployment. Shortly afterwards is ok.

> diff --git b/pushsnapscript/snap_store.py a/pushsnapscript/snap_store.py
> new file mode 100644
> index 0000000..e9cb8cb
> --- /dev/null
> +++ a/pushsnapscript/snap_store.py
> @@ -0,0 +1,44 @@
> +import logging
> +import os
> +import shutil
> +import tempfile
> +
> +from scriptworker.utils import makedirs
> +
> +from pushsnapscript import task
> +from pushsnapscript.utils import cwd
> +
> +# XXX Hack to only import a subset of snapcraft. Otherwise snapcraft can't be built on any other
> +# distribution than Ubuntu. The prod instance runs CentOS 6. There isn't a package version of
> +# snapcraft on that platform either.
> +import sys
> +dir_path = os.path.dirname(os.path.realpath(__file__))
> +sys.path.append(os.path.join(dir_path, 'snapcraft'))
> +from snapcraft import _store as snapcraft_store_client  # noqa
> +
> +log = logging.getLogger(__name__)
> +
> +
> +def push(context, snap_file_path, channel):
> +    if not task.is_allowed_to_push_to_snap_store(channel=channel):
> +        log.warn('Not allowed to push to Snap store. Skipping push...')
> +        # We don't raise an error because we still want green tasks on dev instances
> +        return
> +
> +    # Snapcraft requires credentials to be stored at $CWD/.snapcraft/snapcraft.cfg. Let's store them
> +    # in a folder that gets purged at the end of the run.
> +    with tempfile.TemporaryDirectory() as temp_dir:
> +        _craft_credentials_file(context, channel, temp_dir)
> +
> +        log.debug('Calling snapcraft push with these args: {}, {}'.format(snap_file_path, channel))
> +        with cwd(temp_dir):
> +            snapcraft_store_client.push(snap_file_path, channel)

Does this raise any exceptions? retry? have a timeout?
With network i/o, we should have error detection and retries with exponential backoff
when we hit errors that could be fixed with a retry.

> +def _craft_credentials_file(context, channel, temp_dir):
> +    macaroon_original_location = context.config['macaroons_locations'][channel]
> +
> +    snapcraft_dir = os.path.join(temp_dir, '.snapcraft')
> +    makedirs(snapcraft_dir)
> +    macaroon_target_location = os.path.join(snapcraft_dir, 'snapcraft.cfg')
> +    shutil.copyfile(macaroon_original_location, macaroon_target_location)

We may want to log this (copying location1 to location2) and wrap the copyfile
in a try/except so we can exit with the appropriate exit code. It's probably not
as important as if we were copying files in a less restricted environment, but
we may catch some issues this way.

> diff --git b/pushsnapscript/snapcraft a/pushsnapscript/snapcraft
> new file mode 160000
> index 0000000..e7e331a
> --- /dev/null
> +++ a/pushsnapscript/snapcraft
> @@ -0,0 +1 @@
> +Subproject commit e7e331ab1d85b7d48d848a2b7b1ad3bcaacdb2cf

Ah, is this a pin?
If so, awesome to see.
Do we have a sane path forward if we want to bump this revision in the future
and make sure things still work? Roll out to production and be ready to back
out the pushsnapscript version bump in puppet?

> diff --git b/pushsnapscript/test/integration/test_integration_script.py a/pushsnapscript/test/integration/test_integration_script.py

Glad to see integration tests :) ... these don't actually touch the service, right?
I wonder how sane we can make read-only tests that actually hit the service for
this and pushapk. I think it's doable. It's also possible I'm very wrong.

> diff --git b/requirements.txt a/requirements.txt
> new file mode 100644
> index 0000000..3b891dd
> --- /dev/null
> +++ a/requirements.txt
> @@ -0,0 +1,16 @@
> +scriptworker
> +
> +# XXX Dependencies required by snapcraft._store.push
> +click
> +progressbar33
> +pyelftools
> +pymacaroons
> +pysha3
> +python-debian
> +pyxdg
> +PyYAML
> +requests
> +requests-toolbelt
> +requests-unixsocket
> +simplejson
> +tabulate

With the git submodule, I think it's even more important to use something
like dephash or pip-tools to expand and pin dep versions. Even if we don't
directly use the pinned file, having it in history will be useful when
things break.

> diff --git b/setup.py a/setup.py
> new file mode 100644
> index 0000000..d3a29e9
> --- /dev/null
> +++ a/setup.py
> @@ -0,0 +1,35 @@

<snip>

> +setup(
> +    name='pushsnapscript',
> +    version=version,
> +    description='TaskCluster Ship-It Worker',
> +    author='Mozilla Release Engineering',
> +    author_email='release+python@mozilla.com',
> +    url='https://github.com/mozilla-releng/pushsnapscript',
> +    packages=find_packages(),

Pretty sure your MANIFEST.in will work with the snapcraft source, but have
you tested a `python setup.py sdist` yet? Tox may have done that for you;
just checking.

> +    include_package_data=True,
> +    zip_safe=False,
> +    entry_points={
> +        'console_scripts': [
> +            'pushsnapscript = pushsnapscript.script:main',
> +        ],
> +    },
> +    license='MPL2',
> +    install_requires=requirements,
> +    classifiers=(
> +        'Programming Language :: Python :: 3.5',

Let's add 3.6.

> diff --git b/tox.ini a/tox.ini
> new file mode 100644
> index 0000000..fbb42b8
> --- /dev/null
> +++ a/tox.ini
> @@ -0,0 +1,42 @@
> +[tox]
> +envlist = py35

Let's test py36 too.
Attachment #8963577 - Flags: review?(aki) → review+
Thank you for your fast and very detailed review :D 


(In reply to Aki Sasaki [:aki] from comment #17)
> We may have some cleanup to do (puppet email)
Thank you for spotting this. It was on dep-pushsnap which I had forgotten about. It's now fixed.


(In reply to Aki Sasaki [:aki] from comment #18)
> ... unless we're unable to install certain dependencies of the source module
> for some reason. I'd also be curious if we'd be able to generate a platform-specific
> wheel if needed.
That's the issue. I didn't manage to install the snapcraft module because of these 2 deps: https://github.com/snapcore/snapcraft/blob/2.39.2/requirements.txt#L10-L11. I'm unsure how we could install the source package without pip trying to install the deps. I mean, I see this can work in production (thanks to puppet), but I don't see how local development can work. I'll take any suggestion you have.


> I think a snapcraft.cfg is fine; it mirrors other scriptworker configs. It's
> only used for publishing, and never leaves the scriptworker, right?
That's right. I made sure it doesn't get inadvertently copied as an artifact by storing the copy in a temporary dir: https://github.com/mozilla-releng/pushsnapscript/blob/5afcdce50e0592ddb73be5193bfd3d3a337ecd45/pushsnapscript/snap_store.py#L30. No matter if `snapcraft_store_client.push()` passes or fails, this folder will be nuked before scriptworker handles artifacts uploads.



> Can we talk to the snap store in a read-only fashion here, or in integration
> testing?
As far as I know, there is no commit option passed. Therefore, if we start a push transaction, we have no way to revert it. We could also just log onto the snap store, but that doesn't pass through the same code. 


> It's possible we could add an `integration` channel or something?
That's a good idea! We can create as many channels as we'd like. The only problem is that these channels are visible to Snap users. We may end up having people on this channel. We can try setting up a different account and keep the snap private. I'll look into this. 


> Let's populate these .md and .json files, as well as scopes.md :)
> If things are still in flux, this can wait, but ideally we should do this
> before initial deployment. Shortly afterwards is ok.
Good catch. I had forgotten about these. I added them in https://github.com/mozilla-releng/pushsnapscript/pull/1


> Does this raise any exceptions? retry? have a timeout?
> With network i/o, we should have error detection and retries with
> exponential backoff when we hit errors that could be fixed with a retry.
Great questions! It does raise exceptions. For example: https://tools.taskcluster.net/groups/GjlzpT7rRpCPuAmTTIQplw/tasks/IRqsdu3LTDyOffLVM0i3_A/runs/2/logs/public%2Flogs%2Flive_backing.log#L362. I hadn't checked about retries. It turns out they do use urllib3's retry mechanism with these args: https://github.com/snapcore/snapcraft/blob/2.39.2/snapcraft/storeapi/_client.py#L29-L31. They filter out what exception need a retry (for instance bad creds don't). They have 5 retries with an exponential backoff of 2. That looks correct to our needs, I think.


> We may want to log this (copying location1 to location2) and wrap the
> copyfile in a try/except so we can exit with the appropriate exit code.
> It's probably not as important as if we were copying files in a less
> restricted environment, but we may catch some issues this way.
I can do it. I'm not sure what kind of issues this would catch, though. Could you give an example?


> Ah, is this a pin? If so, awesome to see.
> Do we have a sane path forward if we want to bump this revision in the future
> and make sure things still work? Roll out to production and be ready to back
> out the pushsnapscript version bump in puppet?
Yeah, like said in (massive) comment 15, it's pinned to the latest stable version: 2.39.2. More precisely, it's a git tag. Regarding bumps, snapcraft is part of the pushsnapscript-X.Y.Z.tar.gz. Bumping snapcraft implies to bump pushsnapscript and other regular deps. I don't think this is too different from other scriptworker types.


> Glad to see integration tests :) ... these don't actually touch the service,
> right? I wonder how sane we can make read-only tests that actually hit
> the service for this and pushapk. I think it's doable. It's also possible I'm very wrong.
I think you're right. We can probably do what you suggested in https://github.com/mozilla-releng/pushapkscript/issues/34, if we have dummy accounts. Let's continue the discussion there.


> With the git submodule, I think it's even more important to use something
> like dephash or pip-tools to expand and pin dep versions. Even if we don't
> directly use the pinned file, having it in history will be useful when
> things break.
Right. Done in: https://github.com/mozilla-releng/pushsnapscript/pull/3


> Pretty sure your MANIFEST.in will work with the snapcraft source, but have
> you tested a `python setup.py sdist` yet? Tox may have done that for you;
> just checking.
Yeah, the workers currently rely on an sdist package which embed snapcraft. Because of that, the package is about 1+ MB.


> Let's add 3.6.
Good idea. Done in: https://github.com/mozilla-releng/pushsnapscript/pull/2
Comment on attachment 8960656 [details]
Bug 1447263 - Snap: Split build and upload task

https://reviewboard.mozilla.org/r/229430/#review238858

LGTM
Attachment #8960656 - Flags: review?(rail) → review+
Comment on attachment 8963577 [details]
[mozilla-releng/pushsnapscript] 14 first commits

Ship it!
Attachment #8963577 - Flags: review?(rail) → review+
Comment on attachment 8960588 [details]
[build/puppet] Add pushsnap_scriptworker instances

https://reviewboard.mozilla.org/r/229340/#review238892
Attachment #8960588 - Flags: review?(rail) → review+
Comment on attachment 8960588 [details]
[build/puppet] Add pushsnap_scriptworker instances

default: https://hg.mozilla.org/build/puppet/rev/8ac23cefde440f7a825640ce1e4360b8bcd3f575
production: https://hg.mozilla.org/build/puppet/rev/ab734fa6e369f9a207a727aa2a9f0b69d31aa776
Attachment #8960588 - Attachment description: Bug 1447263 - Add pushsnap_scriptworker instances → [build/puppet] Add pushsnap_scriptworker instances
Attachment #8960588 - Flags: checked-in+
Comment on attachment 8963577 [details]
[mozilla-releng/pushsnapscript] 14 first commits

Released as 0.1.0: https://github.com/mozilla-releng/pushsnapscript/releases/tag/0.1.0
Attachment #8963577 - Flags: checked-in+
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c830b2ccadd
Snap: Split build and upload task r=rail
Depends on: 1443336
https://hg.mozilla.org/mozilla-central/rev/9c830b2ccadd
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox61: --- → fixed
Resolution: --- → FIXED
The whole chain worked in beta: https://tools.taskcluster.net/groups/GYT_nZNyQzGn7wjcwBTajw/tasks/fFIauoJaQQCFAYJVeQqheg/runs/0/logs/public%2Flogs%2Flive_backing.log#L861. The Snap was correctly pushed, but wasn't published to a channel. This seems to be a regression on the Snap Store side. Canonical is on the hook already. Let's track this in bug 1451667.
Blocks: 1451667
See Also: → 1451694
Blocks: 1533563
Blocks: 1590267
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: