991205 - legacy vcs-sync doesn't AUTOFIX git hangs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Hal Wine [:hwine] use NI!]

Since we started mirroring repositories from Taiwan, we occasionally get git processes hung. For example:

On 2014-04-02 07:27 , [e-mail] wrote:
> report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
> Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit git --git-dir /opt/vcs2vcs/repos/external-sprd-aosp-platform-external-safe-iop/.git fetch --force
> 0 S vcs2vcs   9675  6455  0  80   0 - 26523 wait   14:08 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit git --git-dir /opt/vcs2vcs/repos/external-sprd-aosp-platform-external-safe-iop/.git fetch --force
> Unknown delay: wait S+ /usr/bin/time -f 2014-04-02T14:08:34+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job11/update.log git --git-dir /opt/vcs2vcs/repos/external-sprd-aosp-platform-external-safe-iop/.git fetch --force
> 0 S vcs2vcs   9678  9675  0  80   0 -  1015 wait   14:08 pts/9    00:00:00 /usr/bin/time -f 2014-04-02T14:08:34+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job11/update.log git --git-dir /opt/vcs2vcs/repos/external-sprd-aosp-platform-external-safe-iop/.git fetch --force
> Unknown delay: pipe_w S+ git --git-dir /opt/vcs2vcs/repos/external-sprd-aosp-platform-external-safe-iop/.git fetch --force
> 0 S vcs2vcs   9679  9678  0  80   0 - 26709 pipe_w 14:08 pts/9    00:00:00 git --git-dir /opt/vcs2vcs/repos/external-sprd-aosp-platform-external-safe-iop/.git fetch --force
> Unknown delay: poll_s S+ git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/platform/external/safe-iop
> 0 S vcs2vcs   9680  9679  0  80   0 - 19714 poll_s 14:08 pts/9    00:00:00 git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/platform/external/safe-iop

The existing code only attempts to AUTOFIX sk_wai conditions

Comment 1

[Hal Wine [:hwine] use NI!]

Attachment: poll.diff

Works on production, produces email output:

On 2014-04-02 09:47 , [e-mail] wrote:
> report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
> Attempting unstick of hung process 9680
> 0 S vcs2vcs   9680  9679  0  80   0 - 19714 poll_s 14:08 pts/9    00:00:03 git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/platform/external/safe-iop
>
> (if fix fails, another email will come within 5 minutes)
> (just add this email to https://bugzil.la/829025)

Comment 2

[Hal Wine [:hwine] use NI!]

Comment on attachment 8400789 [details] [diff] [review]
poll.diff

http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/rev/80701aa55d32

deployed to all production boxes

Comment 3

[Hal Wine [:hwine] use NI!]

This appears to be working fine. Note that when a particular git server is experiencing issues, it is "normal" for a string of AUTOFIX emails to appear (at :07, :27, :47), as long as they are for different repositories on that server.