1600787 - snap push to releases task on beta is broken

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Snap, defect)

Description

[Jordan Lund (:jlund)]

first task seemed end with snap/ubuntu determining this can't be automated and we need a human review:

Pushing target.snap [==================================================  ]  97%
Pushing target.snap [=================================================== ]  98%
Pushing target.snap [=================================================== ]  99%
Pushing target.snap [====================================================] 100%
Will need manual review...
exit code: 1

rerun failed because it seems run0 already pushed bits and the task is not idempotent

Processing...|                                                                 
2019-12-02 20:40:42,856 - pushsnapscript.snap_store - WARNING - The store was unable to accept this snap.
  - binary_sha3_384: A file with this exact same content has already been uploaded
2019-12-02 20:40:49,177 - pushsnapscript.snap_store - DEBUG - Logging off Snap store...
2019-12-02 20:40:49,178 - pushsnapscript.snap_store - INFO - Logged off Snap store
Error while processing...
Traceback (most recent call last):
  File "/app/bin/pushsnapscript", line 11, in <module>
    load_entry_point('pushsnapscript==2.0.1', 'console_scripts', 'pushsnapscript')()
  File "/app/lib/python3.7/site-packages/pushsnapscript/script.py", line 42, in main
    return client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
  File "/app/lib/python3.7/site-packages/scriptworker/client.py", line 164, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/app/lib/python3.7/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.7/site-packages/pushsnapscript/script.py", line 23, in async_main
    snap_store.push(context, snap_file_path, channel)
  File "/app/lib/python3.7/site-packages/pushsnapscript/snap_store.py", line 73, in push
    _release_if_needed(store, channel, snap_file_path)
  File "/app/lib/python3.7/site-packages/pushsnapscript/snap_store.py", line 102, in _release_if_needed
    metadata_per_revision = _populate_sha3_384(store, metadata_per_revision)
  File "/app/lib/python3.7/site-packages/pushsnapscript/snap_store.py", line 162, in _populate_sha3_384
    for revision, revision_metadata in metadata_per_revision.items()
  File "/app/lib/python3.7/site-packages/pushsnapscript/snap_store.py", line 163, in <dictcomp>
    if revision > 1    # First revision doesn't have sha3_384
  File "/app/lib/python3.7/site-packages/pushsnapscript/snap_store.py", line 180, in _get_from_sha3_384_from_revision
    return resp.json()['download_sha3_384']
KeyError: 'download_sha3_384'
exit code: 1

Then we got an email:

Manual review requested for v72.0b1-1 (288) upload of firefox.

To check review details, go to https://dashboard.snapcraft.io/snaps/firefox/revisions/288/review/


Upload status: Manual review pending


Visit the developer portal at https://dashboard.snapcraft.io/stores/snaps/ to find out more.
To view the snap, go to https://dashboard.snapcraft.io/snaps/firefox/

Best,

The Snap Store team

Comment 1

[Jordan Lund (:jlund)]

logging into snapcraft I see:

human review required due to 'deny-connection' constraint (interface attributes) declaration-snap-v2_slots_connection (dbus-daemon, dbus) 

under the failed section

@jlorenzo, any ideas?

I don't think we should block on this for b1

Comment 2

[Johan Lorenzo [:jlorenzo]]

Yeah, let's not block b1 because of this bug.

I think the manual review is needed because we merge bug 1591751. It uses some new APIs, and I think the people at Canonical have to double-check if we're doing this right. I don't think we can act on anything at the moment. I'll monitor the progress.

Comment 3

[Jordan Lund (:jlund)]

hit this on b2 too https://firefoxci.taskcluster-artifacts.net/Z_zU0Y8RSk6CnNjovve8Zw/0/public/logs/live_backing.log

Comment 4

[Krish De Souza]

I have submitted this problem to forum.snapcraft as that seems to be the quickest way to get feedback.

https://forum.snapcraft.io/t/manual-review-firefox/14460

Comment 5

[Krish De Souza]

Looks like the snap has been manually approved and should now work for automation.

https://forum.snapcraft.io/t/manual-review-firefox/14460/2

Comment 6

[Johan Lorenzo [:jlorenzo]]

Attachment: Screenshot_2019-12-04 Manage version 72 0b1-1 of Firefox .png

Thank you very much for your help, Krish!

I confirm b1 has been manually approved (see screenshot) and b2 now passes the automated checks.

Comment 7

[Johan Lorenzo [:jlorenzo]]

I confirm people are now able to install firefox snap:

$ snap refresh --beta firefox 
firefox (beta) 72.0b2-1 from Mozilla✓ refreshed

Comment 8

[Krish De Souza]

Hi Johan,

Would it be possible to know how many people are currently using the Firefox snap? This is just for personal interest and to measure the impact of my change.

Kind regards,
Krish

Comment 9

[Jordan Lund (:jlund)]

new error in the script run. Unrelated to Krish's fix but I'm re-opening this bug since it starting tracking failing push snap tasks.

https://firefoxci.taskcluster-artifacts.net/NoGkPPU2S3eHHEd4oG06Pw/0/public/logs/live_backing.log

pkg_resources.ContextualVersionConflict: (PyYAML 5.1.2 (/app/lib/python3.7/site-packages), Requirement.parse('PyYAML==3.13'), {'pushsnapscript'})

3.13 is not a new pinned version in requirements. https://github.com/mozilla-releng/scriptworker-scripts/blame/81b5f3bebd7b958b1bce94f94f764a1a62e4335c/pushsnapscript/requirements/base.in#L13

I'm not sure why we have 5.1.2 installed and it's ignoring 3.13

I also see that we don't have 3.13 in our pypi mirror so I'm confused how this worked before: https://pypi.pub.build.mozilla.org/pub/

Comment 10

[Johan Lorenzo [:jlorenzo]]

(In reply to Krish De Souza from comment #8)

Would it be possible to know how many people are currently using the Firefox snap? This is just for personal interest and to measure the impact of my change.

Hey Krish! I can say 92.8% of our beta population got 72.0b3! The rest still has to update :)

(In reply to Jordan Lund (:jlund) from comment #9)

I'm not sure why we have 5.1.2 installed and it's ignoring 3.13

I see what happened. It's a regression of [1]. Here's what their build logs[2] read:

Step 6 : RUN python -m venv /app  && ./bin/pip install --upgrade pip  && cd pushsnapscript  && /app/bin/pip install -r requirements/base.txt  && /app/bin/pip install .  && cd /app/configloader  && /app/bin/pip install -r requirements/base.txt  && /app/bin/pip install .  && cd /app
 ---> Running in 4bed7f2efb98
[...]
Collecting pyyaml==3.13
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
[...]
Successfully installed aiohttp-3.6.2 aiomemoizettl-0.0.3 arrow-0.15.4 async-timeout-3.0.1 attrs-19.3.0 certifi-2019.9.11 cffi-1.13.2 chardet-3.0.4 click-7.0 cryptography-2.8 dictdiffer-0.8.0 frozendict-1.2 future-0.18.2 github3.py-1.3.0 idna-2.8 importlib-metadata-0.23 json-e-3.0.0 jsonschema-3.2.0 jwcrypto-0.6.0 libnacl-1.3.6 mohawk-1.1.0 more-itertools-7.2.0 mozilla-version-0.4.1 multidict-4.5.2 progressbar33-2.4 pycparser-2.19 pyelftools-0.24 pymacaroons-0.9.2 pyrsistent-0.15.5 pysha3-1.0.2 python-dateutil-2.8.1 python-debian-0.1.28 pyxdg-0.25 pyyaml-3.13 requests-2.22.0 requests-toolbelt-0.6.0 requests-unixsocket-0.1.5 scriptworker-29.0.2 simplejson-3.8.2 six-1.13.0 slugid-2.0.0 snapcraft-2.43.1 tabulate-0.7.5 taskcluster-23.0.0 taskcluster-urls-12.1.0 uritemplate-3.0.0 urllib3-1.25.7 yarl-1.3.0 zipp-0.6.0
Processing /app/pushsnapscript
[...]
Requirement already satisfied: PyYAML==3.13 in /app/lib/python3.7/site-packages (from pushsnapscript==2.0.1) (3.13)
[...]
Installing collected packages: pushsnapscript
    Running setup.py install for pushsnapscript: started
    Running setup.py install for pushsnapscript: finished with status 'done'
Successfully installed pushsnapscript-2.0.1
Requirement already satisfied: click==7.0 in /app/lib/python3.7/site-packages (from -r requirements/base.txt (line 8)) (7.0)
Requirement already satisfied: json-e==3.0.0 in /app/lib/python3.7/site-packages (from -r requirements/base.txt (line 11)) (3.0.0)
Collecting pyyaml==5.1.2
  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
Requirement already satisfied: slugid==2.0.0 in /app/lib/python3.7/site-packages (from -r requirements/base.txt (line 27)) (2.0.0)
ERROR: pushsnapscript 2.0.1 has requirement PyYAML==3.13, but you'll have pyyaml 5.1.2 which is incompatible.
Installing collected packages: pyyaml
  Found existing installation: PyYAML 3.13
    Uninstalling PyYAML-3.13:
      Successfully uninstalled PyYAML-3.13
    Running setup.py install for pyyaml: started
    Running setup.py install for pyyaml: finished with status 'done'
Successfully installed pyyaml-5.1.2
Processing /app/configloader
Requirement already satisfied: click in /app/lib/python3.7/site-packages (from configloader==0.0.1) (7.0)
Requirement already satisfied: json-e in /app/lib/python3.7/site-packages (from configloader==0.0.1) (3.0.0)
Requirement already satisfied: pyyaml in /app/lib/python3.7/site-packages (from configloader==0.0.1) (5.1.2)
Requirement already satisfied: slugid in /app/lib/python3.7/site-packages (from configloader==0.0.1) (2.0.0)
Installing collected packages: configloader
    Running setup.py install for configloader: started
    Running setup.py install for configloader: finished with status 'done'
Successfully installed configloader-0.0.1

You can see several things:

1. /app/bin/pip install -r requirements/base.txt in the pushsnapscript folder does install PyYAML 3.13
2. /app/bin/pip install . in the same folder tries to reinstall PyYAML 3.13, but there's nothing more to do because it was already installed in the previous step
3. /app/bin/pip install -r requirements/base.txt in the configloader folder installs json-e[3], which depends on PyYAML[4] with no version specified. Therefore pip installs the latest available version (which is 5.1.2) ignoring the pinned one 2 steps ago.

I know for a fact [1] did change this behavior because the build before that one doesn't override PyYAML[5].

I see 2 potential ways of solving this error:
a. Either we bump snapcraft hoping they don't pin PyYAML to 3.13 anymore.
b. Or we don't install the dependencies or configloader after the main project anymore.

After checking, a. cannot be done, they still pin it[6]. So we have to do something about b. What do you think, Rail?

[1] https://github.com/mozilla-releng/scriptworker-scripts/pull/64
[2] https://firefox-ci-tc.services.mozilla.com/tasks/XY9rF-1TQSyERq1wyWmyPg/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FXY9rF-1TQSyERq1wyWmyPg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L4771
[3] https://github.com/rail/configloader/blob/b07566347e781e8e0e93b07fb6d0bd0ea400026a/requirements/base.in#L2
[4] https://github.com/taskcluster/json-e/blob/v3.0.0/setup.py#L36
[5] https://firefox-ci-tc.services.mozilla.com/tasks/euHOFf2gS4OuyuL7u4oUiQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FeuHOFf2gS4OuyuL7u4oUiQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log
[6] https://github.com/snapcore/snapcraft/blob/3.9.3/requirements.txt#L6

Comment 11

[Rail Aliiev [:rail]]

Sorry, just realized that I didn't respond here.

We ended up with https://github.com/mozilla-releng/scriptworker-scripts/pull/114. Rerunning the task worked fine.

Comment 12

[Johan Lorenzo [:jlorenzo]]

Closing bug per comment 11.