Closed
Bug 1351149
Opened 7 years ago
Closed 7 years ago
unable to build dev env: rsync: connection unexpectedly closed
Categories
(MozReview Graveyard :: Testing / Development Environment, defect)
MozReview Graveyard
Testing / Development Environment
Tracking
(Not tracked)
RESOLVED
WORKSFORME
People
(Reporter: glob, Unassigned)
References
Details
likely fallout from bug 1349407. i should have tested it with a full rebuild. hgweb> rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] hgweb> rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] hgrb> rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] hgrb> rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] rbweb> rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rbweb> rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.6] ERROR hgweb> rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] ERROR hgweb> rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] ERROR rbweb> rsync: connection unexpectedly closed (0 bytes received so far) [receiver] ERROR rbweb> rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.6] ERROR hgrb> rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] ERROR hgrb> rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
yup - backing out https://hg.mozilla.org/hgcustom/version-control-tools/rev/c77d8797d0ad fixes it.
Blocks: 1349407
docker-machine version 0.10.0, Docker version 17.03.1-ce, boot2docker 17.03.0-ce on osx 10.12.3
Comment 3•7 years ago
|
||
I haven't attempted to reproduce this yet. However, I noticed today that my `rsync` processes between TCP endpoints are now just as slow as volume/mount-based rsyncs which bug 1349407 purportedly fixed by using TCP endpoints. I have no clue WTF is going on. Docker is my bane. I guess that means we can back out the switch from volume-based rsyncs since it didn't work out in the end.
Comment 5•7 years ago
|
||
As part of hacking on the bmoweb image yesterday, I think I found the source of slowness: fchownat() is super slow on overlay2. If you do an internet search for "fchownat overlay2" many of the results mention Docker (probably because Docker is one of the major users of overlay2). I suspect the underlying issue is "touching" file's metadata, thus triggering COW at the filesystem level or something. We could potentially speed up rsync by telling it to avoid syncing certain metadata, such as uid, gid, and mtime values. I think processes that access the v-c-t rsync are running as root, so the uid/gid shouldn't matter.
Comment 6•7 years ago
|
||
Replacing `rsync -a` with `rsync -rlp` sped up `d0cker build-hgmo` from 165s to 39s. Good grief.
from bug 1349407: Pushed by gszorc@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/63f19cfacc5c docker: stop using `rsync -a` because owner/group updates are slow on overlayfs i had no issues with a full dev env rebuild after with this change.
Comment 8•7 years ago
|
||
I'm going to call this WFM then.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
You need to log in
before you can comment on or make changes to this bug.
Description
•