1604486 - Intermittent [taskcluster:error] Uploading error artifact public/build from file artifacts/ with message "Could not read directory '/Users/task_1576583400/artifacts'", after mercurial.error.SignalInterrupt

Status: NEW

(Developer Services :: Mercurial: hg.mozilla.org, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=281534568&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YPjhDkXGSJ2Q5UORpYXlZg/runs/0/artifacts/public/logs/live_backing.log

---

[vcs 2019-12-17T12:06:13.416Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/worker.py", line 243, in _posixworker
[vcs 2019-12-17T12:06:13.416Z] res = util.pickle.load(key.fileobj)
[vcs 2019-12-17T12:06:13.416Z] _pickle.UnpicklingError: pickle data was truncated
[vcs 2019-12-17T12:06:13.417Z] ** unknown exception encountered, please report by visiting
[vcs 2019-12-17T12:06:13.417Z] ** https://mercurial-scm.org/wiki/BugTracker
[vcs 2019-12-17T12:06:13.417Z] ** Python 3.7.5 (default, Nov 1 2019, 02:16:32) [Clang 11.0.0 (clang-1100.0.33.8)]
[vcs 2019-12-17T12:06:13.417Z] ** Mercurial Distributed SCM (version 5.2)
[vcs 2019-12-17T12:06:13.417Z] ** Extensions loaded: share, rebase, strip, mq, purge, robustcheckout, sparse
[vcs 2019-12-17T12:06:13.417Z] Traceback (most recent call last):
[vcs 2019-12-17T12:06:13.417Z] File "/usr/local/bin/hg", line 36, in <module>
[vcs 2019-12-17T12:06:13.419Z] dispatch.run()
[vcs 2019-12-17T12:06:13.419Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 111, in run
[vcs 2019-12-17T12:06:13.419Z] status = dispatch(req)
[vcs 2019-12-17T12:06:13.419Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 250, in dispatch
[vcs 2019-12-17T12:06:13.419Z] ret = _runcatch(req) or 0
[vcs 2019-12-17T12:06:13.419Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 424, in _runcatch
[vcs 2019-12-17T12:06:13.420Z] return _callcatch(ui, _runcatchfunc)
[vcs 2019-12-17T12:06:13.420Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 433, in _callcatch
[vcs 2019-12-17T12:06:13.420Z] return scmutil.callcatch(ui, func)
[vcs 2019-12-17T12:06:13.420Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/scmutil.py", line 177, in callcatch
[vcs 2019-12-17T12:06:13.420Z] return func()
[vcs 2019-12-17T12:06:13.420Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 414, in _runcatchfunc
[vcs 2019-12-17T12:06:13.420Z] return _dispatch(req)
[vcs 2019-12-17T12:06:13.420Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 1174, in _dispatch
[vcs 2019-12-17T12:06:13.421Z] lui, repo, cmd, fullargs, ui, options, d, cmdpats, cmdoptions
[vcs 2019-12-17T12:06:13.421Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 862, in runcommand
[vcs 2019-12-17T12:06:13.421Z] ret = _runcommand(ui, options, cmd, d)
[vcs 2019-12-17T12:06:13.421Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 1185, in _runcommand
[vcs 2019-12-17T12:06:13.421Z] return cmdfunc()
[vcs 2019-12-17T12:06:13.421Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 1171, in <lambda>
[vcs 2019-12-17T12:06:13.422Z] d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
[vcs 2019-12-17T12:06:13.422Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/util.py", line 1843, in check
[vcs 2019-12-17T12:06:13.422Z] return func(*args, **kwargs)
[vcs 2019-12-17T12:06:13.422Z] File "/usr/local/lib/hgext/robustcheckout.py", line 261, in robustcheckout
[vcs 2019-12-17T12:06:13.422Z] sparse_profile=sparseprofile)
[vcs 2019-12-17T12:06:13.422Z] File "/usr/local/lib/hgext/robustcheckout.py", line 777, in _docheckout
[vcs 2019-12-17T12:06:13.423Z] if commands.update(ui, repo, rev=checkoutrevision, clean=True):
[vcs 2019-12-17T12:06:13.423Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/commands.py", line 7694, in update
[vcs 2019-12-17T12:06:13.425Z] ui, repo, rev, brev, clean=clean, updatecheck=updatecheck
[vcs 2019-12-17T12:06:13.425Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/hg.py", line 1106, in updatetotally
[vcs 2019-12-17T12:06:13.425Z] ret = _clean(repo, checkout)
[vcs 2019-12-17T12:06:13.425Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/hg.py", line 1043, in clean
[vcs 2019-12-17T12:06:13.426Z] stats = updaterepo(repo, node, True)
[vcs 2019-12-17T12:06:13.426Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/hg.py", line 1024, in updaterepo
[vcs 2019-12-17T12:06:13.426Z] updatecheck=updatecheck,
[vcs 2019-12-17T12:06:13.426Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/merge.py", line 2558, in update
[vcs 2019-12-17T12:06:13.427Z] repo, actions, wc, p2, overwrite, wantfiledata, labels=labels
[vcs 2019-12-17T12:06:13.427Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/merge.py", line 1927, in applyupdates
[vcs 2019-12-17T12:06:13.427Z] for final, res in prog:
[vcs 2019-12-17T12:06:13.427Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/worker.py", line 243, in _posixworker
[vcs 2019-12-17T12:06:13.427Z] res = util.pickle.load(key.fileobj)
[vcs 2019-12-17T12:06:13.427Z] _pickle.UnpicklingError: pickle data was truncated
[taskcluster 2019-12-17T12:06:13.613Z] Exit Code: 1
[taskcluster 2019-12-17T12:06:13.613Z] User Time: 4m25.074255s
[taskcluster 2019-12-17T12:06:13.613Z] Kernel Time: 2m43.841782s
[taskcluster 2019-12-17T12:06:13.613Z] Wall Time: 2m31.335821108s
[taskcluster 2019-12-17T12:06:13.613Z] Result: FAILED
[taskcluster 2019-12-17T12:06:13.613Z] === Task Finished ===
[taskcluster 2019-12-17T12:06:13.613Z] Task Duration: 2m31.340631289s
[taskcluster:error] Uploading error artifact public/build from file artifacts/ with message "Could not read directory '/Users/task_1576583400/artifacts'", reason "file-missing-on-worker" and expiry 2020-12-16T11:17:38.610Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/Users/task_1576583400/artifacts'
[taskcluster 2019-12-17T12:06:13.841Z] [mounts] Preserving cache: Moving "/Users/task_1576583400/checkouts" to "/var/local/generic-worker/caches/H8eFo-Y3RFCwfLo1o1h9qg"
[taskcluster 2019-12-17T12:06:13.843Z] [mounts] Denying task_1576583400 access to '/var/local/generic-worker/caches/H8eFo-Y3RFCwfLo1o1h9qg'
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read directory '/Users/task_1576583400/artifacts'

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

[vcs 2019-12-17T12:06:11.942Z] updating [================================================> ] 281700/285287 02s
[vcs 2019-12-17T12:06:13.317Z] updating [================================================> ] 284400/285287 01sTraceback (most recent call last):
[vcs 2019-12-17T12:06:13.317Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/scmutil.py", line 177, in callcatch
[vcs 2019-12-17T12:06:13.317Z] return func()
[vcs 2019-12-17T12:06:13.317Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/worker.py", line 207, in workerfunc
[vcs 2019-12-17T12:06:13.317Z] os.write(wfd, util.pickle.dumps(result))
[vcs 2019-12-17T12:06:13.317Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 303, in catchterm
[vcs 2019-12-17T12:06:13.317Z] raise error.SignalInterrupt
[vcs 2019-12-17T12:06:13.317Z] mercurial.error.SignalInterrupt

Comment 2

[Chris Manchester (limited bugmail, email directly)]

Hmm, this only seems to be happening on pgo run tasks. Connor, does this stack look familiar? Did we do something wrong configuring these workers?

Comment 3

[Connor Sheehan [:sheehan]]

This doesn't look familiar to me. From the source code it looks like Mercurial uses signal.signal to turn SIGTERM and SIGHUP (possibly more, didn't dig much deeper) into that exception. Hope that helps.

Comment 4

[Andreea Pavel [:apavel]]

There are 37 total failures in the last 5 days, all on macosx64-shippable opt and macosx64 pgo

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=282008592&repo=autoland&lineNumber=334

Connor can you assign someone to take a look?

Comment 5

[Intermittent Failures Robot]

17 failures in 933 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17

Platform breakdown:

• macosx64-shippable: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2019-12-19&endday=2019-12-19&tree=all

Comment 6

[Intermittent Failures Robot]

54 failures in 4582 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #27 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 51
• mozilla-central: 3

Platform breakdown:

• macosx64: 21
• macosx64-shippable: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 7

[Intermittent Failures Robot]

19 failures in 1365 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #41 most frequent failure this week.

Repository breakdown:

• autoland: 19

Platform breakdown:

• macosx64-shippable: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 8

[Andreea Pavel [:apavel]]

There are 28 total failures in the last 7 days on macosx64-shippable opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283371878&repo=autoland&lineNumber=292

Connor have you had a chance to look?

Comment 9

[Intermittent Failures Robot]

33 failures in 1862 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 32
• mozilla-central: 1

Platform breakdown:

• macosx64-shippable: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2019-12-30&endday=2020-01-05&tree=all

Comment 10

[Intermittent Failures Robot]

32 failures in 794 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• mozilla-beta: 5
• try: 1

Platform breakdown:

• macosx64-shippable: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-06&endday=2020-01-06&tree=all

Comment 11

[Intermittent Failures Robot]

29 failures in 795 pushes (0.036 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 26
• mozilla-central: 2
• mozilla-beta: 1

Platform breakdown:

• macosx64-shippable: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-07&endday=2020-01-07&tree=all

Comment 12

[Andreea Pavel [:apavel]]

This has reached the disable-recommended queue. Connor is this something to worry about?

Comment 13

[Connor Sheehan [:sheehan]]

(In reply to Andreea Pavel [:apavel] from comment #12)

This has reached the disable-recommended queue. Connor is this something to worry about?

I'm not sure to be honest. This is only happening on macos64-shippable/opt/generate-profile-macosx64-shippable, which leads me to believe it's a task configuration issue.

However these tasks are also some of the first to use Python 3 Mercurial for checkouts in automation. It's possible there's a Python 3 bug in robustcheckout or core Mercurial that we need to address. Until we start seeing failures on other platforms I don't think it's worth investigating as a Py3 problem. I don't have any bright ideas for possible causes at the moment.

Comment 14

[Andreea Pavel [:apavel]]

Thank you!

Joel does this qualify for stockwell infra?

Comment 15

[Intermittent Failures Robot]

17 failures in 897 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 1

Platform breakdown:

• macosx64-shippable: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-08&endday=2020-01-08&tree=all

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

this is an infra issue, not related to product or tests cases specifically. Given the frequency of this we should be making this a higher priority.

Comment 17

[Chris Manchester (limited bugmail, email directly)]

I'm going to attempt to enable sparse checkouts on these workers, which will either make this less frequent/not happen or at least give us more information about where the issue is.

Currently this needs the pr at https://github.com/mozilla-platform-ops/ronin_puppet/pull/169 to be merged.

Comment 18

[Intermittent Failures Robot]

34 failures in 819 pushes (0.042 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 32
• mozilla-central: 1
• mozilla-beta: 1

Platform breakdown:

• macosx64-shippable: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-09&endday=2020-01-09&tree=all

Comment 19

[Intermittent Failures Robot]

16 failures in 831 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16

Platform breakdown:

• macosx64-shippable: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-10&endday=2020-01-10&tree=all

Comment 20

[Intermittent Failures Robot]

134 failures in 4472 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 121
• mozilla-beta: 7
• try: 1
• mozilla-central: 5

Platform breakdown:

• macosx64-shippable: 134

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 21

[Andreea Pavel [:apavel]]

Chris, based on Joel's comment, should the priority of this bug be changed from P5? Is that something you can do?

Comment 22

[Chris Manchester (limited bugmail, email directly)]

I'm not sure of an appropriate priority for this bug, but I intend to follow through on the idea in comment 17, which Connor and I agree should help the situation.

Comment 23

[Intermittent Failures Robot]

22 failures in 693 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 22

Platform breakdown:

• macosx64-shippable: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-13&endday=2020-01-13&tree=all

Comment 24

[Intermittent Failures Robot]

37 failures in 819 pushes (0.045 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• mozilla-beta: 1

Platform breakdown:

• macosx64-shippable: 37

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-14&endday=2020-01-14&tree=all

Comment 25

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1604486 - Use sparse checkout for the macOS profile run. r?sheehan

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/17e35065ec03
Use sparse checkout for the macOS profile run. r=sheehan

Comment 27

[Chris Manchester (limited bugmail, email directly)]

This seems to have morphed into something that looks more like bug 1304942. Connor, does the stack from this failure look familiar: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=285117960&repo=autoland&lineNumber=155 ?

Comment 28

[Chris Manchester (limited bugmail, email directly)]

That seems to have been a temporary outage of some sort. Clearing ni? for now.

Comment 29

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/17e35065ec03

Comment 30

[Intermittent Failures Robot]

81 failures in 3773 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 77
• mozilla-beta: 3
• mozilla-central: 1

Platform breakdown:

• macosx64-shippable: 81

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 31

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #26)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/17e35065ec03
Use sparse checkout for the macOS profile run. r=sheehan

Chris, makes sense that this patch caused the below regression? There were 2 suspect patches, while the other one was backed out - without fixing the regression.
== Change summary for alert #24703 (as of Thu, 16 Jan 2020 15:48:53 GMT) ==

Regressions:

14% perf_reftest_singletons external-string-pass.html macosx1014-64-shippable opt e10s stylo 887.66 -> 1,012.21
6% perf_reftest_singletons bidi-resolution-1.html macosx1014-64-shippable opt e10s stylo 160.76 -> 170.36
6% perf_reftest_singletons parent-basic-singleton.html macosx1014-64-shippable opt e10s stylo 168.08 -> 177.47

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24703

Comment 32

[Chris Manchester (limited bugmail, email directly)]

Unfortunately this does seem to be the fault of this patch, I will post a fix.

Comment 33

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1604486 - Update profile generate sparse profile to include talos tests.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/feea4bfad848
Update profile generate sparse profile to include talos tests. r=sheehan

Comment 35

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/feea4bfad848

Comment 36

[Marian Raiciof [:marauder]]

Hey Chris,

There is a single regression from the changeset above, is that expected ?
Thanks!

== Change summary for alert #24738 (as of Wed, 22 Jan 2020 07:16:20 GMT) ==

Regressions:

9% perf_reftest_singletons bloom-basic.html windows7-32-shippable opt e10s stylo 44.77 -> 48.74

Improvements:

18% perf_reftest_singletons external-string-pass.html windows7-32-shippable opt e10s stylo 1,105.06 -> 903.19
18% perf_reftest_singletons external-string-pass.html windows10-64-shippable opt e10s stylo 1,008.88 -> 830.73
17% perf_reftest_singletons external-string-pass.html windows10-64-shippable-qr opt e10s stylo 1,009.56 -> 836.34
7% perf_reftest_singletons tiny-traversal-singleton.html windows7-32-shippable opt e10s stylo 786.12 -> 733.19
7% perf_reftest_singletons scrollbar-styles-1.html windows7-32-shippable opt e10s stylo 595.62 -> 556.04
6% perf_reftest_singletons tiny-traversal-singleton.html windows10-64-shippable-qr opt e10s stylo 832.34 -> 778.62
6% perf_reftest_singletons id-getter-2.html windows10-64-shippable-qr opt e10s stylo 918.48 -> 864.88
6% perf_reftest_singletons scrollbar-styles-1.html windows10-64-shippable opt e10s stylo 603.84 -> 570.65
5% perf_reftest_singletons scrollbar-styles-1.html windows10-64-shippable-qr opt e10s stylo 609.87 -> 578.58
5% perf_reftest_singletons bidi-resolution-1.html windows7-32-shippable opt e10s stylo 151.84 -> 144.10

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24738

Comment 37

[Chris Manchester (limited bugmail, email directly)]

This does seem to be related to this change. I don't know these benchmarks well enough to know whether to be concerned about this, but it seems fine given the other improvements here. :dmajor, do you have any input here?

Comment 38

[(Away)]

The perf singleton tests really seem to have a mind of their own. I had hoped that getting them into PGO would stabilize the numbers but they still jump around a lot.

Comment 39

[Intermittent Failures Robot]

25 failures in 4004 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 22
• mozilla-beta: 1
• mozilla-central: 2

Platform breakdown:

• macosx64-shippable: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 40

[Jan Alexander Steffens [:heftig]]

I've also been running into this exception building Nightly for Arch Linux, when our build system does a hg clone across filesystems. It only started happening after Mercurial was moved to Python 3. So this seems very much like a Mercurial core bug.

Comment 41

[Jan Alexander Steffens [:heftig]]

This should fix the problem: https://phab.mercurial-scm.org/D8051

Comment 42

[Connor Sheehan [:sheehan]]

Thanks for debugging this, :heftig!!

:glob and I were just looking at this yesterday, but we were having a hard time reproducing it.

Comment 43

[Intermittent Failures Robot]

10 failures in 2190 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 1
• try: 1
• mozilla-beta: 1

Platform breakdown:

• macosx64-shippable: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-01-27&endday=2020-02-02&tree=all

Comment 44

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #24832 (as of Sun, 02 Feb 2020 22:06:35 GMT) ==

Improvements:

5% perf_reftest_singletons bidi-resolution-1.html macosx1014-64-shippable opt e10s stylo 169.55 -> 160.95

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24832

Comment 45

[Intermittent Failures Robot]

9 failures in 3364 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9

Platform breakdown:

• macosx64-shippable: 8
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 46

[Intermittent Failures Robot]

18 failures in 4690 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2
• autoland: 15
• try: 1

Platform breakdown:

• macosx64-shippable: 2
• macosx1014-64-shippable: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-02-10&endday=2020-02-16&tree=all

Comment 47

[Chris Manchester (limited bugmail, email directly)]

The mitigation landed and was partly effective, but with the actual fix upstream I'm un-assigning myself.

Comment 48

[Intermittent Failures Robot]

24 failures in 4045 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

This is the #43 most frequent failure this week.

Repository breakdown:

• autoland: 21
• mozilla-central: 2
• mozilla-release: 1

Platform breakdown:

• macosx1014-64-shippable: 23
• macosx64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-02-17&endday=2020-02-23&tree=all

Comment 49

[Intermittent Failures Robot]

7 failures in 5233 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• try: 1

Platform breakdown:

• macosx1014-64-shippable: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-02-24&endday=2020-03-01&tree=all

Comment 51

[Connor Sheehan [:sheehan]]

There is a fix for this bug that has landed in upstream Mercurial 5.3. Upgrading the docker image to use the latest hg should fix this problem.

Comment 52

[Intermittent Failures Robot]

18 failures in 5483 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18

Platform breakdown:

• macosx1014-64-shippable: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-03-02&endday=2020-03-08&tree=all

Comment 53

[Andrei Ciure[:aciure]]

There are 23 failures associated to this bug in the last 7 days. These are occurring on macosx1014-64-shippable opt.

recent log: https://treeherder.mozilla.org/logviewer.html#?job_id=292692665&repo=autoland
[vcs 2020-03-11T15:31:28.548Z] Traceback (most recent call last):
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/worker.py", line 210, in _posixworker
[vcs 2020-03-11T15:31:28.548Z] ret = scmutil.callcatch(ui, workerfunc)
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/scmutil.py", line 177, in callcatch
[vcs 2020-03-11T15:31:28.548Z] return func()
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/worker.py", line 206, in workerfunc
[vcs 2020-03-11T15:31:28.548Z] for result in func(*(staticargs + (pargs,))):
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/merge.py", line 1724, in batchget
[vcs 2020-03-11T15:31:28.548Z] fctx(f).data(),
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/context.py", line 1248, in data
[vcs 2020-03-11T15:31:28.548Z] return self._filelog.read(self._filenode)
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/filelog.py", line 169, in read
[vcs 2020-03-11T15:31:28.548Z] return storageutil.filtermetadata(self.revision(node))
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/filelog.py", line 93, in revision
[vcs 2020-03-11T15:31:28.548Z] return self._revlog.revision(node, _df=_df, raw=raw)
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/revlog.py", line 1715, in revision
[vcs 2020-03-11T15:31:28.548Z] return self._revisiondata(nodeorrev, _df, raw=raw)[0]
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/revlog.py", line 1743, in _revisiondata
[vcs 2020-03-11T15:31:28.548Z] rev, rawtext, validated = self._rawtext(node, rev, _df=_df)
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/revlog.py", line 1812, in _rawtext
[vcs 2020-03-11T15:31:28.548Z] bins = self._chunks(chain, df=_df, targetsize=targetsize)
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/revlog.py", line 1631, in _chunks
[vcs 2020-03-11T15:31:28.548Z] ladd(decomp(buffer(data, chunkstart - offset, chunklength)))
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/revlog.py", line 2095, in decompress
[vcs 2020-03-11T15:31:28.548Z] return _zlibdecompress(data)
[vcs 2020-03-11T15:31:28.548Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/dispatch.py", line 303, in catchterm
[vcs 2020-03-11T15:31:28.548Z] raise error.SignalInterrupt
[vcs 2020-03-11T15:31:28.548Z] mercurial.error.SignalInterrupt
...............
[vcs 2020-03-11T15:31:28.569Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/merge.py", line 1927, in applyupdates
[vcs 2020-03-11T15:31:28.570Z] for final, res in prog:
[vcs 2020-03-11T15:31:28.570Z] File "/usr/local/Cellar/mercurial/5.2_1/lib/python3.7/site-packages/mercurial/worker.py", line 243, in _posixworker
[vcs 2020-03-11T15:31:28.570Z] res = util.pickle.load(key.fileobj)
[vcs 2020-03-11T15:31:28.570Z] _pickle.UnpicklingError: pickle data was truncated
[taskcluster 2020-03-11T15:31:28.649Z] Exit Code: 1
[taskcluster 2020-03-11T15:31:28.649Z] User Time: 13.267878s
[taskcluster 2020-03-11T15:31:28.649Z] Kernel Time: 5.528787s
[taskcluster 2020-03-11T15:31:28.649Z] Wall Time: 14.12946886s
[taskcluster 2020-03-11T15:31:28.649Z] Result: FAILED
[taskcluster 2020-03-11T15:31:28.649Z] === Task Finished ===
[taskcluster 2020-03-11T15:31:28.649Z] Task Duration: 14.134413269s
[taskcluster:error] Uploading error artifact public/build from file artifacts/ with message "Could not read directory '/Users/task_1583938311/artifacts'", reason "file-missing-on-worker" and expiry 2021-03-11T14:45:45.761Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/Users/task_1583938311/artifacts'
[taskcluster 2020-03-11T15:31:28.871Z] [mounts] Preserving cache: Moving "/Users/task_1583938311/checkouts" to "/var/local/generic-worker/caches/T0rSF8cZR8WKXVOU7pqBQg"
[taskcluster 2020-03-11T15:31:28.871Z] [mounts] Denying task_1583938311 access to '/var/local/generic-worker/caches/T0rSF8cZR8WKXVOU7pqBQg'
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read directory '/Users/task_1583938311/artifacts'

Comment 54

[:glob ✱]

The UnpicklingError error has been fixed in Mercurial 5.3 however the log in comment 53 indicates it's still using 5.2.1.
Who is responsible for updating the docker image to use the latest hg version?

Comment 55

[Intermittent Failures Robot]

15 failures in 5336 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 13
• mozilla-release: 1

Platform breakdown:

• macosx1014-64-shippable: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 56

[Intermittent Failures Robot]

9 failures in 4955 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 2

Platform breakdown:

• macosx1014-64-shippable: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 57

[Intermittent Failures Robot]

8 failures in 4773 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 1

Platform breakdown:

• macosx1014-64-shippable: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-03-23&endday=2020-03-29&tree=all

Comment 58

[Connor Sheehan [:sheehan]]

Dave, this bug can be fixed at any time by upgrading the Mercurial version from 5.2 to 5.3+ (5.3.1 is the latest version ATM). Are you able to upgrade this? Is a patch to ronin_puppet the correct place to do such an upgrade?

Comment 59

[:dhouse]

Connor, thanks! That is the right repo. I need to do some other updates on the pgo branch in the coming month. So I'll do this also.

Comment 60

[Intermittent Failures Robot]

6 failures in 5015 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• try: 1

Platform breakdown:

• macosx1014-64-shippable: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-03-30&endday=2020-04-05&tree=all

Comment 61

[Intermittent Failures Robot]

5 failures in 4823 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5

Platform breakdown:

• macosx1014-64-shippable: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 62

[Intermittent Failures Robot]

8 failures in 4528 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 1

Platform breakdown:

• macosx1014-64-shippable: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 63

[Intermittent Failures Robot]

8 failures in 5582 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform breakdown:

• macosx1014-64-shippable: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 64

[Intermittent Failures Robot]

1 failures in 5423 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows2012-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 65

[Intermittent Failures Robot]

1 failures in 5276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• macosx1014-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 66

[Intermittent Failures Robot]

1 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• windows2012-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 67

[Intermittent Failures Robot]

1 failures in 4618 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• macosx1014-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 68

[Intermittent Failures Robot]

1 failures in 3760 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• macosx1014-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-09-14&endday=2020-09-20&tree=all

Comment 69

[Intermittent Failures Robot]

1 failures in 4019 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• macosx1014-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 70

[Intermittent Failures Robot]

13 failures in 4275 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 7
• mozilla-beta: 2

Platform breakdown:

• macosx1014-64-shippable: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 71

[Intermittent Failures Robot]

3 failures in 4128 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-release: 1
• try: 1

Platform breakdown:

• macosx1014-64-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604486&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 72

[Intermittent Failures Robot]

2 failures in 4060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2

Platform breakdown:

• macosx1014-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1604486&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 73

[Intermittent Failures Robot]

1 failures in 4257 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1604486&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 74

[Intermittent Failures Robot]

1 failures in 3706 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• macosx1014-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1604486&startday=2021-04-26&endday=2021-05-02&tree=all