1455534 - Intermittent IOError: [Errno 13] y:\hg-shared\8ba995b74e18334ab3707f27e9eb8f4e37ba3d29\.hg/store\data/modules/libpref/init/_static_pref_list.h.i: Access is denied

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=174707743&repo=autoland

https://queue.taskcluster.net/v1/task/LvLYufiiRIaAK7IvyIwKMA/runs/0/artifacts/public/logs/live_backing.log

Traceback (most recent call last):
  File "mercurial\scmutil.pyc", line 150, in callcatch
  File "mercurial\dispatch.pyc", line 292, in _runcatchfunc
  File "mercurial\dispatch.pyc", line 896, in _dispatch
  File "mercurial\dispatch.pyc", line 658, in runcommand
  File "mercurial\dispatch.pyc", line 904, in _runcommand
  File "mercurial\dispatch.pyc", line 893, in <lambda>
  File "mercurial\util.pyc", line 1077, in check
  File "C:/mozilla-build/robustcheckout.py", line 239, in robustcheckout
  File "C:/mozilla-build/robustcheckout.py", line 516, in _docheckout
  File "mercurial\exchange.pyc", line 1251, in pull
  File "mercurial\exchange.pyc", line 1393, in _pullbundle2
  File "mercurial\bundle2.pyc", line 364, in processbundle
  File "mercurial\bundle2.pyc", line 464, in _processpart
  File "mercurial\bundle2.pyc", line 1552, in handlechangegroup
  File "mercurial\bundle2.pyc", line 405, in _processchangegroup
  File "mercurial\changegroup.pyc", line 327, in apply
  File "mercurial\changegroup.pyc", line 986, in _addchangegroupfiles
  File "mercurial\revlog.pyc", line 1886, in addgroup
  File "mercurial\store.pyc", line 488, in __call__
  File "mercurial\vfs.pyc", line 405, in __call__
  File "mercurial\windows.pyc", line 140, in posixfile
IOError: [Errno 13] y:\hg-shared\8ba995b74e18334ab3707f27e9eb8f4e37ba3d29\.hg/store\data/modules/libpref/init/_static_pref_list.h.i: Access is denied
abort: y:\hg-shared\8ba995b74e18334ab3707f27e9eb8f4e37ba3d29\.hg/store\data/modules/libpref/init/_static_pref_list.h.i: Access is denied
[taskcluster:error]    Exit Code: 255
[taskcluster:error]    User Time: 15.625ms
[taskcluster:error]  Kernel Time: 0s
[taskcluster:error]    Wall Time: 11.918s
[taskcluster:error]       Result: FAILED
[taskcluster 2018-04-20T06:32:16.368Z] === Task Finished ===
[taskcluster 2018-04-20T06:32:16.368Z] Task Duration: 12.1849674s
[taskcluster:error] Uploading error artifact public/build from file public/build with message "Could not read directory 'Z:\\task_1524202998\\public\\build'", reason "file-missing-on-worker" and expiry 2019-04-20T06:30:54.782Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'Z:\task_1524202998\public\build'
[taskcluster 2018-04-20T06:32:17.074Z] Uploading artifact public/logs/certified.log from file generic-worker\certified.log with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-04-20T06:30:54.782Z
[taskcluster 2018-04-20T06:32:17.751Z] Uploading artifact public/chainOfTrust.json.asc from file generic-worker\chainOfTrust.json.asc with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-04-20T06:30:54.782Z
[taskcluster 2018-04-20T06:32:18.331Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/LvLYufiiRIaAK7IvyIwKMA/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-04-20T06:30:54.782Z
[taskcluster:error] Task not successful due to following exception(s):
[taskcluster:error] Exception 1)
[taskcluster:error] exit status 255
[taskcluster:error] Exception 2)
[taskcluster:error] file-missing-on-worker: Could not read directory 'Z:\task_1524202998\public\build'
[taskcluster:error]

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Pete cleared the gecko-3-b-win2012, this shall resolve the issue.

Comment 2

[Pete Moore [:pmoore][:pete]]

This would be solved with bug 1350956.

Comment 3

[Intermittent Failures Robot]

75 failures in 130 pushes (0.577 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 52
* try: 19
* autoland: 4

Platform breakdown:
* windows2012-32: 38
* windows2012-64: 33
* windows2012-64-noopt: 2
* windows2012-32-noopt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1455534&startday=2018-04-20&endday=2018-04-20&tree=trunk

Comment 4

[Andreea Pavel [:apavel]]

This continued to occurred on today's nightlys: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=dd0e54d786743974a50a338059bcd68a09b6d5b2&selectedJob=174941784

All failures ran on worker i-0b15cd926915cb97d, i terminated the machine.

Comment 5

[Intermittent Failures Robot]

18 failures in 44 pushes (0.409 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-central: 12
* try: 4
* mozilla-inbound: 2

Platform breakdown:
* windows2012-32: 14
* windows2012-64: 3
* windows2012-64-noopt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1455534&startday=2018-04-21&endday=2018-04-21&tree=trunk

Comment 6

[Intermittent Failures Robot]

127 failures in 782 pushes (0.162 failures/push) were associated with this bug in the last 7 days. 

This is the #7 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 65
* mozilla-central: 29
* try: 25
* autoland: 8

Platform breakdown:
* windows2012-32: 76
* windows2012-64: 44
* windows2012-64-noopt: 4
* windows2012-32-noopt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1455534&startday=2018-04-16&endday=2018-04-22&tree=trunk

Comment 7

[Pete Moore [:pmoore][:pete]]

It looks like https://github.com/mozilla-releng/OpenCloudConfig/blob/7eccd4a9085f3cb4e2fbde1d8cac81b260258d00/userdata/Configuration/GenericWorker/run-generic-worker-format-and-reboot.bat#L58 is creating the flag file C:\dsc\task-claim-state.valid before rebooting.

This is that flag file that OCC uses to delay the launch of generic-worker.exe in https://github.com/mozilla-releng/OpenCloudConfig/blob/master/userdata/Configuration/GenericWorker/run-generic-worker-format-and-reboot.bat#L22

This file should only be created when OCC run has completed. I believe that is done here: https://github.com/mozilla-releng/OpenCloudConfig/blob/dbd82051b22f3cd336153d257f5509bf838652d0/userdata/rundsc.ps1#L1204

If that rundsc.ps1 line runs at the end of every OCC run, the solution is simply to delete line 58 of run-generic-worker-format-and-reboot.bat (the first link above).

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for OpenCloudConfig

Comment 9

[Pete Moore [:pmoore][:pete]]

Kendall, is someone on your team available to deploy this? It is 7pm here, and it takes a few hours to completely roll out.

Comment 10

[Kendall Libby [:fubar] (he/him)]

I'm *very* wary about rolling out changes to this outside of EU hours, to be honest, especially given how long it takes to completely roll out. I would like it if grenade can deploy, and the two of you keep an eye on the rollout.

Comment 11

[Intermittent Failures Robot]

128 failures in 105 pushes (1.219 failures/push) were associated with this bug yesterday.    

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* autoland: 54
* mozilla-central: 50
* mozilla-beta: 15
* mozilla-inbound: 8
* try: 1

Platform breakdown:
* windows2012-64: 60
* windows2012-32: 49
* windows2012-64-noopt: 9
* windows2012-32-noopt: 3
* windows2012-32-devedition: 3
* windows2012-64-devedition: 2
* windows2012-64-add-on-devel: 1
* windows2012-32-add-on-devel: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1455534&startday=2018-04-23&endday=2018-04-23&tree=trunk

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

merged by pmoore last night:
https://github.com/mozilla-releng/OpenCloudConfig/commit/f9296e2195f5d367c32581c54fb3116c9928860c

rollout of new gecko-(1,2,3)-b-win2012 amis in progress:
https://tools.taskcluster.net/groups/dz9lxKKsRHafzfQx_Zl0uA

monitoring...

Comment 13

[Intermittent Failures Robot]

130 failures in 780 pushes (0.167 failures/push) were associated with this bug in the last 7 days. 

This is the #3 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* autoland: 56
* mozilla-central: 50
* mozilla-beta: 13
* mozilla-inbound: 10
* try: 1

Platform breakdown:
* windows2012-64: 66
* windows2012-32: 45
* windows2012-64-noopt: 9
* windows2012-32-noopt: 4
* windows2012-32-devedition: 3
* windows2012-64-devedition: 1
* windows2012-64-add-on-devel: 1
* windows2012-32-add-on-devel: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1455534&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 14

[Intermittent Failures Robot]

1 failures in 551 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1455534&startday=2018-04-30&endday=2018-05-06&tree=trunk