654938 - StagingRepositorySetupFactory should retry on clone failures

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Rail Aliiev [:rail]]

Attachment: use retry.py

It usually fails due to hg.m.o timeouts or connection closures.

Comment 1

[Rail Aliiev [:rail]]

Attachment: use retry.py

Tested in staging.

Normal clone output:
==========================
Calling <function run_with_timeout at 0xb7c84fb4> with args: (['ssh', '-l', 'stage-ffxbld', '-oIdentityFile=~cltbld/.ssh/ffxbld_dsa', 'hg.mozilla.org', 'clone', 'nn-NO', 'l10n-central/nn-NO'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['ssh', '-l', 'stage-ffxbld', '-oIdentityFile=~cltbld/.ssh/ffxbld_dsa', 'hg.mozilla.org', 'clone', 'nn-NO', 'l10n-central/nn-NO']
Process stdio:
Please wait.  Cloning /l10n-central/nn-NO to /users/stage-ffxbld/nn-NO
Clone complete.
Fixing permissions, don't interrupt.

Process stderr:
==========================

Output with failure:
==========================
Calling <function run_with_timeout at 0xb7c6cfb4> with args: (['ssh', '-l', 'stage-ffxbld', '-oIdentityFile=~cltbld/.ssh/ffxbld_dsa', 'hg.mozilla.org', 'clone', 'nl', 'l10n-central/nl'], 300, None, None, False, True), kwargs: {}, attempt #4
Executing: ['ssh', '-l', 'stage-ffxbld', '-oIdentityFile=~cltbld/.ssh/ffxbld_dsa', 'hg.mozilla.org', 'clone', 'nl', 'l10n-central/nl']
Failed, sleeping 30 seconds before retrying
Calling <function run_with_timeout at 0xb7c6cfb4> with args: (['ssh', '-l', 'stage-ffxbld', '-oIdentityFile=~cltbld/.ssh/ffxbld_dsa', 'hg.mozilla.org', 'clone', 'nl', 'l10n-central/nl'], 300, None, None, False, True), kwargs: {}, attempt #5
Executing: ['ssh', '-l', 'stage-ffxbld', '-oIdentityFile=~cltbld/.ssh/ffxbld_dsa', 'hg.mozilla.org', 'clone', 'nl', 'l10n-central/nl']
Process stdio:
Please wait.  Cloning /l10n-central/nl to /users/stage-ffxbld/nl

Process stderr:
Killed by signal 15.

Process stdio:
Please wait.  Cloning /l10n-central/nl to /users/stage-ffxbld/nl
Clone complete.
Fixing permissions, don't interrupt.

Process stderr:
==========================

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 530258 [details] [diff] [review]
use retry.py

Testing another, more robust version.

Comment 3

[Rail Aliiev [:rail]]

Attachment: more robust repo_setup

* Add "?rnd=$randomInt" to the end of the user repo URLs. Sometimes we get 404 even thought the repo exists.

Generates URLs like the following:
http://hg.mozilla.org/users/stage-ffxbld/pt-PT?rnd=528539

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 530274 [details] [diff] [review]
more robust repo_setup

http://hg.mozilla.org/build/buildbotcustom/rev/21a8316e55b3

Comment 5

[Rail Aliiev [:rail]]

retry.py has its own timeout, cloning mozilla repo times out.

Comment 6

[Rail Aliiev [:rail]]

Attachment: sync timeouts

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 530333 [details] [diff] [review]
sync timeouts

http://hg.mozilla.org/build/buildbotcustom/rev/f8824518fe6b

Comment 8

[Rail Aliiev [:rail]]

Passed.