Closed Bug 1554733 Opened 5 years ago Closed 5 years ago

Perma pfu abort: No such file or directory: /home/worker/mozilla-beta/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(firefox69 fixed)

RESOLVED FIXED
Tracking Status
firefox69 --- fixed

People

(Reporter: NarcisB, Assigned: sfraser)

References

Details

Attachments

(1 file)

Fail started with:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=617d77613a30d7feab1621cd36cf6d29d8e08544&selectedJob=248513812

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248513812&repo=mozilla-beta&lineNumber=43056

Log snippet:

Flags: needinfo?(sfraser)

It looks like we install robustcheckout into the image, but don't use it. I wonder if that would be sufficient to address this error.

It also appears that the checkout isn't happening in a cache, which has significantly worse performance[1], in docker.


[1] Docker uses a layered filesystem for storing images. Writes to the container outside of volumes (which back caches) cause overhead of copying data to a new layer.

(In reply to Tom Prince [:tomprince] from comment #1)

It looks like we install robustcheckout into the image, but don't use it. I wonder if that would be sufficient to address this error.

It doesn't appear to be sufficient, sadly. From inside an interactive task:

# hg robustcheckout --sharebase /tmp/hg -b default https://hg.mozilla.org/releases/mozilla-beta mozilla-beta2
(using Mercurial 4.8.1)
ensuring https://hg.mozilla.org/releases/mozilla-beta@default is available at mozilla-beta2
(sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/releases/mozilla-beta/6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.pac
ked1.hg
520050 files to transfer, 2.58 GB of data
transferred 2.58 GB in 60.7 seconds (43.5 MB/sec)                                                                                                         
finished applying clone bundle
searching for changes
adding changesets                                                                                                                                         
adding manifests
adding file changes
added 3 changesets with 3 changes to 3 files
new changesets 35528f5b95dd:c246934cd0fa
searching for changes
no changes found
(pulling to obtain default)
(remote resolved default to c246934cd0fa26c9ed4cee40b07d4c5c7c32bc4d; result is not deterministic)
(revision already present locally; not pulling)
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
abort: No such file or directory: /tmp/hg/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d
PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["c5d.4xlarge"], "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozi
lla.org", "shouldAlert": false, "subtests": [], "value": 83.6685619354248}, {"extraOptions": ["c5d.4xlarge"], "lowerIsBetter": true, "name": "update", "se
rverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 10.156893968582153}, {"extraOptions": ["c5d.4xlarge"], "lowerIsBetter": true, "
name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 95.23151183128357}, {"extraOptions": ["c5d.4xlarge"], "low
erIsBetter": true, "name": "overall_clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 95.23151183128357}, {"extraOptio
ns": ["c5d.4xlarge"], "lowerIsBetter": true, "name": "overall_clone_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "v
alue": 95.23151183128357}]}

I've tried one various platforms and inside/outside of containers, and it doesn't happen, unless the clone bundle is https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/releases/mozilla-beta/6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.packed1.hg - from outside taskcluster the clone bundle provided is https://hg.cdn.mozilla.net/releases/mozilla-beta/6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.zstd.hg, which works.

sfraser@belfry ~/throwaway $ hg init mb1 & cd mb1
sfraser@belfry ~/throwaway/mb1 $ hg debugapplystreamclonebundle ../6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.packed1.hg 
520050 files to transfer, 2.58 GB of data
transferred 2.58 GB in 142.7 seconds (18.5 MB/sec)                                                                                                              
sfraser@belfry ~/throwaway/mb1 $ hg update
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
abort: No such file or directory: /Users/sfraser/throwaway/mb1/.hg/store/data/browser/branding/official/bgstub__2x.jpg.d
sfraser@belfry ~/throwaway/mb1 $ cd ../
sfraser@belfry ~/throwaway $ hg init mb2
sfraser@belfry ~/throwaway $ cd mb2/
sfraser@belfry ~/throwaway/mb2 $ hg unbundle ../6ec3cb391f03a3d02c47e3cb4525fe9eb6abd242.zstd.hg
adding changesets
adding manifests                                                                                                                                          
adding file changes                                                                                                                                       
added 533435 changesets with 3222866 changes to 518350 files (+134 heads)                                                                                 
new changesets 8ba995b74e18:6ec3cb391f03 (533435 drafts)
(run 'hg heads' to see heads)
sfraser@belfry ~/throwaway/mb2 $ hg update
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
277143 files updated, 0 files merged, 0 files removed, 0 files unresolved        
Flags: needinfo?(sfraser)
Assignee: nobody → sfraser

Connor, is this something vcs related?

(we should probably swap to robustcheckout anyway, it just doesn't seem related)

Flags: needinfo?(sheehan)

From discussion on IRC with :sheehan this appears to have been a bad bundle, which is being reverted.

Flags: needinfo?(sheehan)

I'm running hg verify against mozilla-beta on hgssh1 (the master write server for hgmo) and it appears "data/browser/branding/official/bgstub_2x.jpg.d" is missing from the fncache. This would have resulted in the file being missing in the bundle. It's unclear to me why this would be missing in one bundle but not another - perhaps the bundle process is corrupting the cache in some way. I'll investigate this further today.

In the meantime, I'm rolling back to yesterdays bundle, and rebuilding the fncache on hgssh1.

Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4656c6e4964a
Use robustcheckout for periodic file udpates r=tomprince
Pushed by nerli@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cf13d6e64f97
Backed out changeset 4656c6e4964a for linting failure in periodic_file_updates.sh CLOSED TREE
https://hg.mozilla.org/integration/autoland/rev/9baa0e441779
Use robustcheckout for periodic file udpates r=tomprince
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Duplicate of this bug: 1466053
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: