AWS instance fails to clone project repo 'cypress'

RESOLVED FIXED

Status

Developer Services
General
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: hwine, Assigned: hwine)

Tracking

Details

(Whiteboard: [vcs-sync])

Attachments

(1 attachment)

(Assignee)

Description

2 years ago
The cypress repo can no longer be cloned on the AWS instance processing vcs-sync for gecko-projects in developer services.

The cypress repo was just reset in bug 1222301 and much additional code added to it since. It is now the largest hg repo being converted.

Traceback reports for the failed clone are all similar to:
    00:12:41     INFO -    File "/opt/vcs2vcs/vcssync2/build/venv/lib64/python2.7/site-packages/mercurial/httppeer.py", line 23, in zgenerator
    00:12:41     INFO -      raise IOError(None, _('connection ended unexpectedly'))
    00:12:41     INFO -  IOError: [Errno None] connection ended unexpectedly
    00:12:41    ERROR -  abort: connection ended unexpectedly
    00:12:41    ERROR -  Automation Error: hg not responding
    00:12:41    ERROR - Return code: 255
(Assignee)

Comment 1

2 years ago
After some research[1], this appears to be due to a combination of:
  a) older mercurial client version used in vcs-sync (2.6.3) (per :gps)
  b) size of repository-to-be-cloned (cypress is merge of
     mozilla-central and comm-central)
  c) size of ec2 instance used by dev-services instance
     (releng used larger and did not have this problem)

This combination leads to the decompressing of the bundle taking so long that the TCP connection times out.

The quickest way to address the issue is to add the "--uncompressed" option to the "hg clone" command for the repository. Manual testing shows this works.


[1] https://hglabhq.com/blog/2014/6/20/working-with-mercurial-over-unreliable-connections was most helpful
(Assignee)

Comment 2

2 years ago
Created attachment 8699805 [details] [diff] [review]
add clone options

Add an optional config parameter to specify additional clone options.

Tested by running in production for gecko-projects :/ but works without issue for both projects with new config value and projects without config value.

Not (yet) tested for the other conversions. The impact will be easy to test after this is landed on default, before merge to production branch.
Assignee: nobody → hwine
Status: NEW → ASSIGNED
(Assignee)

Comment 3

2 years ago
https://hg.mozilla.org/build/mozharness/rev/007f3de6e029557710c587608ac5fc727f0414a3
bug 1233611 - allow tuning of clone parameters; r=me
(Assignee)

Comment 4

2 years ago
Comment on attachment 8699805 [details] [diff] [review]
add clone options

On production branch after testing no impact on l10n conversions.
  https://hg.mozilla.org/build/mozharness/rev/9901bc9e8ee4
Attachment #8699805 - Flags: review+
Attachment #8699805 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
See Also: → bug 1253068
You need to log in before you can comment on or make changes to this bug.