1488396 - Intermittent: Snap gets uploaded onto Snap store but hangs when releasing it

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Release Automation: Snap, defect)

Description

[Johan Lorenzo [:jlorenzo]]

It happened today in 62.0rc2[1] and in 61.0.2[2]. We don't have much more logs than:

> Pushing target.snap [==================================================  ]  97%
> Pushing target.snap [==================================================  ]  98%
> Pushing target.snap [=================================================== ]  99%
> Pushing target.snap [====================================================] 100%
> Automation Error: python exited with signal -15

The next run (with the same snap) reads it was already uploaded[3]


[1] https://tools.taskcluster.net/groups/UQd6GS1uRxqTZDpbG8e43Q/tasks/U4KfD70JSGSfd4z5rEfWrw/runs/0/logs/public%2Flogs%2Flive_backing.log#L112
[2] https://tools.taskcluster.net/groups/faHC2a7tRU23Z09Fpy35Iw/tasks/TCocuj7LR_StwVblH4VwGg/runs/0/logs/public%2Flogs%2Flive_backing.log#L113
[3] https://tools.taskcluster.net/groups/UQd6GS1uRxqTZDpbG8e43Q/tasks/U4KfD70JSGSfd4z5rEfWrw/runs/1/logs/public%2Flogs%2Flive_backing.log#L553

Comment 1

[Johan Lorenzo [:jlorenzo]]

Happened in 63.0b3 too https://tools.taskcluster.net/groups/Et2Kn47nS5WefXfzw3S3kw/tasks/WXxPmmAnSj2bydd2B_kDpw/runs/0/logs/public%2Flogs%2Flive_backing.log#L113

Comment 2

[Johan Lorenzo [:jlorenzo]]

63.0b4 went fine, though https://tools.taskcluster.net/groups/BnAI_o7FSzqPEC40fR2CHw/tasks/AAML0YWSQO-e6KAN-gWu1Q/details

Comment 3

[Intermittent Failures Robot]

3 failures in 2667 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 3

Platform breakdown:
* linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1488396&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 4

[Johan Lorenzo [:jlorenzo]]

Note: The 3 reported failures are actually just reruns of the same job.

Comment 5

[Johan Lorenzo [:jlorenzo]]

63.0b5 failed too https://tools.taskcluster.net/groups/f36SGmzCQGG5Y9VLg5vYuA/tasks/HuIP3SnxQXOt2vHoJONClQ/runs/1/logs/public%2Flogs%2Flive_backing.log#L555 :(

Comment 6

[Intermittent Failures Robot]

2 failures in 2684 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1488396&startday=2018-09-10&endday=2018-09-16&tree=all

Comment 8

[Johan Lorenzo [:jlorenzo]]

Attachment: [pushsnapscript] PR: Bump snapcraft to 2.43.1

Comment 9

[Johan Lorenzo [:jlorenzo]]

Attachment: [build/puppet] Bug 1488396 - Update pushsnap_scriptworker to latest snapcraft

Was tested in https://tools.taskcluster.net/groups/CTU6CA-PTGyDBu4IExIgoA/tasks/USJ2lRdtQ16waOO0kzT-zw/runs/2/logs/public%2Flogs%2Flive_backing.log

Comment 10

[Intermittent Failures Robot]

2 failures in 2902 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1488396&startday=2018-09-17&endday=2018-09-23&tree=all

Comment 11

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9010946 [details] [review]
[pushsnapscript] PR: Bump snapcraft to 2.43.1

Landed on master at https://github.com/mozilla-releng/pushsnapscript/commit/3e919626fd98abb226babbdb3b71bd3dbf71ac5d
Released as part of pushsnapscript 0.2.2: https://pypi.org/project/pushsnapscript/0.2.2/ https://github.com/mozilla-releng/pushsnapscript/releases/tag/0.2.2

Comment 12

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9010947 [details] [review]
[build/puppet] Bug 1488396 - Update pushsnap_scriptworker to latest snapcraft

Landed on master https://github.com/mozilla-releng/build-puppet/commit/82552c49b85324a882601f8a7db58a4a7ad3c93c

Comment 13

[Johan Lorenzo [:jlorenzo]]

The last job was done against comment 11 and comment 12 live in production: https://tools.taskcluster.net/groups/GWhlex9mT4Or6laQ0YLn2A/tasks/d-n3ms05SUm_ho2ONMWpHA/runs/0/logs/public%2Flogs%2Flive_backing.log

This is then fixed.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Looks like this happened again today with 63.0b11 :(
https://treeherder.mozilla.org/logviewer.html#?job_id=202730072&repo=mozilla-beta

Comment 15

[Johan Lorenzo [:jlorenzo]]

Oh noes! Thanks for letting me know. I made some good progress on bug 1491262. This bug should mitigate the impact of this error. Let's track the fix there.

Comment 16

[Intermittent Failures Robot]

2 failures in 2708 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 1
* mozilla-esr60: 1

Platform breakdown:
* linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1488396&startday=2018-10-01&endday=2018-10-07&tree=all