Perma pfu abort: No such file or directory: /home/worker/mozilla-beta/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d
Categories
(Release Engineering :: Release Automation: Other, defect)
Tracking
(firefox69 fixed)
Tracking | Status | |
---|---|---|
firefox69 | --- | fixed |
People
(Reporter: NarcisB, Assigned: sfraser)
References
Details
Attachments
(1 file)
Log snippet:
- CLONE_CMD='/usr/bin/hg clone https://hg.mozilla.org/releases/mozilla-beta mozilla-beta'
- /usr/bin/hg clone https://hg.mozilla.org/releases/mozilla-beta mozilla-beta
applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/releases/mozilla-beta/dae867a5e8fed9e8b7aa0ecf14b3f1cd93f703c5.packed1.hg
520042 files to transfer, 2.58 GB of data
transferred 2.58 GB in 93.0 seconds (28.4 MB/sec)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 20 changesets with 64 changes to 58 files
new changesets 39520fe84be8:dc28babd8428
updating to branch default
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
abort: No such file or directory: /home/worker/mozilla-beta/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d
[taskcluster 2019-05-27 12:38:04.458Z] === Task Finished ===
[taskcluster 2019-05-27 12:38:05.673Z] Unsuccessful task run with exit code: 255 completed in 9393.779 seconds
Comment 1•5 years ago
|
||
It looks like we install robustcheckout
into the image, but don't use it. I wonder if that would be sufficient to address this error.
It also appears that the checkout isn't happening in a cache, which has significantly worse performance[1], in docker.
[1] Docker uses a layered filesystem for storing images. Writes to the container outside of volumes (which back caches) cause overhead of copying data to a new layer.
Assignee | ||
Comment 2•5 years ago
|
||
(In reply to Tom Prince [:tomprince] from comment #1)
It looks like we install
robustcheckout
into the image, but don't use it. I wonder if that would be sufficient to address this error.
It doesn't appear to be sufficient, sadly. From inside an interactive task:
# hg robustcheckout --sharebase /tmp/hg -b default https://hg.mozilla.org/releases/mozilla-beta mozilla-beta2
(using Mercurial 4.8.1)
ensuring https://hg.mozilla.org/releases/mozilla-beta@default is available at mozilla-beta2
(sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/releases/mozilla-beta/6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.pac
ked1.hg
520050 files to transfer, 2.58 GB of data
transferred 2.58 GB in 60.7 seconds (43.5 MB/sec)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files
new changesets 35528f5b95dd:c246934cd0fa
searching for changes
no changes found
(pulling to obtain default)
(remote resolved default to c246934cd0fa26c9ed4cee40b07d4c5c7c32bc4d; result is not deterministic)
(revision already present locally; not pulling)
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
abort: No such file or directory: /tmp/hg/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d
PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["c5d.4xlarge"], "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozi
lla.org", "shouldAlert": false, "subtests": [], "value": 83.6685619354248}, {"extraOptions": ["c5d.4xlarge"], "lowerIsBetter": true, "name": "update", "se
rverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 10.156893968582153}, {"extraOptions": ["c5d.4xlarge"], "lowerIsBetter": true, "
name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 95.23151183128357}, {"extraOptions": ["c5d.4xlarge"], "low
erIsBetter": true, "name": "overall_clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 95.23151183128357}, {"extraOptio
ns": ["c5d.4xlarge"], "lowerIsBetter": true, "name": "overall_clone_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "v
alue": 95.23151183128357}]}
I've tried one various platforms and inside/outside of containers, and it doesn't happen, unless the clone bundle is https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/releases/mozilla-beta/6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.packed1.hg
- from outside taskcluster the clone bundle provided is https://hg.cdn.mozilla.net/releases/mozilla-beta/6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.zstd.hg
, which works.
sfraser@belfry ~/throwaway $ hg init mb1 & cd mb1
sfraser@belfry ~/throwaway/mb1 $ hg debugapplystreamclonebundle ../6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.packed1.hg
520050 files to transfer, 2.58 GB of data
transferred 2.58 GB in 142.7 seconds (18.5 MB/sec)
sfraser@belfry ~/throwaway/mb1 $ hg update
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
abort: No such file or directory: /Users/sfraser/throwaway/mb1/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d
sfraser@belfry ~/throwaway/mb1 $ cd ../
sfraser@belfry ~/throwaway $ hg init mb2
sfraser@belfry ~/throwaway $ cd mb2/
sfraser@belfry ~/throwaway/mb2 $ hg unbundle ../6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.zstd.hg
adding changesets
adding manifests
adding file changes
added 533435 changesets with 3222866 changes to 518350 files (+134 heads)
new changesets 8ba995b74e18:6ec3cb391f03 (533435 drafts)
(run 'hg heads' to see heads)
sfraser@belfry ~/throwaway/mb2 $ hg update
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
277143 files updated, 0 files merged, 0 files removed, 0 files unresolved
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 3•5 years ago
|
||
Connor, is this something vcs related?
(we should probably swap to robustcheckout anyway, it just doesn't seem related)
Assignee | ||
Comment 4•5 years ago
|
||
From discussion on IRC with :sheehan this appears to have been a bad bundle, which is being reverted.
Comment 5•5 years ago
|
||
I'm running hg verify
against mozilla-beta on hgssh1 (the master write server for hgmo) and it appears "data/browser/branding/official/bgstub_2x.jpg.d" is missing from the fncache. This would have resulted in the file being missing in the bundle. It's unclear to me why this would be missing in one bundle but not another - perhaps the bundle process is corrupting the cache in some way. I'll investigate this further today.
In the meantime, I'm rolling back to yesterdays bundle, and rebuilding the fncache on hgssh1.
Assignee | ||
Comment 6•5 years ago
|
||
Comment hidden (Intermittent Failures Robot) |
Pushed by sfraser@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4656c6e4964a Use robustcheckout for periodic file udpates r=tomprince
Pushed by nerli@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/cf13d6e64f97 Backed out changeset 4656c6e4964a for linting failure in periodic_file_updates.sh CLOSED TREE https://hg.mozilla.org/integration/autoland/rev/9baa0e441779 Use robustcheckout for periodic file udpates r=tomprince
Comment 10•5 years ago
|
||
bugherder |
Description
•