Closed Bug 1230586 (bumper_hang_from_git) Opened 8 years ago Closed 8 years ago

[tracker] socket hang on https pull from git.mozilla.org

Categories

(Developer Services :: Git, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: hwine, Unassigned)

References

Details

(Whiteboard: [b2g_bumper])

Starting a tracker. This is first occurrence afaik. We a similar issue on hgweb heads for a while.

Will wait to see data on how often this is happening.
b2g_bumper hung from buildbot-master66.bb.releng.usw2.mozilla.com for approximately 1 hour. kill -hup cleared. (There is a nagios alert in #buildduty for b2g_bumper staleness is how this was noticed.)

Fri Dec  4 08:41:57 PST 2015
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps flwww -s 3516
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   500  3516  3511  20   0 106096  1276 wait   Ss   ?          0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
0   500  5428  3516  20   0 209228 12964 pipe_w S    ?          0:00  \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/v2.2r -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/v2.2r.py --import-git-ref-cache --push-loop --export-git-ref-cache
0   500  5616  5428  20   0  89996  8912 wait   S    ?          0:00      \_ python /usr/local/bin/gittool.py -r origin/v2.2r https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/v2.2r/build/manifests
0   500  5624  5616  20   0  16484  1604 pipe_w S    ?          0:00          \_ git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/*
0   500  5625  5624  20   0  91176  6500 pipe_w S    ?          0:00              \_ git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git
0   500  5639  5625  20   0  18792  2132 pipe_w S    ?          0:00                  \_ git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps -flwwws 3516
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S cltbld    3516  3511  0  80   0 - 26524 wait   07:40 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
0 S cltbld    5428  3516  0  80   0 - 52307 pipe_w 07:42 ?        00:00:00 python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/v2.2r -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/v2.2r.py --import-git-ref-cache --push-loop --export-git-ref-cache
0 S cltbld    5616  5428  0  80   0 - 22499 wait   07:43 ?        00:00:00 python /usr/local/bin/gittool.py -r origin/v2.2r https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/v2.2r/build/manifests
0 S cltbld    5624  5616  0  80   0 -  4121 pipe_w 07:43 ?        00:00:00 git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/*
0 S cltbld    5625  5624  0  80   0 - 22794 pipe_w 07:43 ?        00:00:00 git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git
0 S cltbld    5639  5625  0  80   0 -  4698 pipe_w 07:43 ?        00:00:00 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ kill -hup 5639
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps -flwwws 3516
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S cltbld    3516  3511  0  80   0 - 26524 wait   07:40 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
0 S cltbld    5428  3516  0  80   0 - 52307 poll_s 07:42 ?        00:00:00 python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/v2.2r -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/v2.2r.py --import-git-ref-cache --push-loop --export-git-ref-cache
Alias: bumper_hang_from_git
Whiteboard: [b2g_bumper]
This has happened before, see bug 1040062. However, we now have web heads serving http. At time of other report, there was only a single host supporting git.mozilla.org.

Troubleshooting instructions used by releng are at:
 https://wiki.mozilla.org/ReleaseEngineering/Applications/Bumper#Troubleshooting
Please update there if additional data is requested.
See Also: → 1040062
root      1308  0.0  0.0 117324   780 ?        Ss   Mar14   3:30 crond
root     20204  0.0  0.0 136088  1304 ?        S    11:50   0:00  \_ CROND
cltbld   20207  0.0  0.0 106096  1276 ?        Ss   11:50   0:00  |   \_ /bin/bash /usr/local/bin/run_b2g_bumper.sh
cltbld   22042  0.0  0.3 209284 13072 ?        S    11:53   0:00  |       \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/master -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/master.py --import-git-ref-cache --push-loop --export-git-ref-cache
cltbld   22052  0.0  0.2  89988  8904 ?        S    11:53   0:00  |           \_ python /usr/local/bin/gittool.py -r origin/master https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/master/build/manifests
cltbld   22055  0.0  0.0  16484  1608 ?        S    11:53   0:00  |               \_ git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/*
cltbld   22056  0.0  0.1  91036  6504 ?        S    11:53   0:00  |                   \_ git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git
cltbld   22058  0.0  0.0  18816  2156 ?        S    11:53   0:00  |                       \_ git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/

after killing 22058....

root      1308  0.0  0.0 117324   780 ?        Ss   Mar14   3:30 crond
root     20204  0.0  0.0 136088  1304 ?        S    11:50   0:00  \_ CROND
cltbld   20207  0.0  0.0 106096  1276 ?        Ss   11:50   0:00      \_ /bin/bash /usr/local/bin/run_b2g_bumper.sh
cltbld   22042  0.0  0.3 209284 13084 ?        S    11:53   0:00          \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/master -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/master.py --import-git-ref-cache --push-loop --export-git-ref-cache
Another one:
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps flwww -s $(pgrep bumper)
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   500 22743 22739  20   0 106096  1224 wait   Ss   ?          0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
0   500 15134 15133  20   0 106096  1276 wait   Ss   ?          0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
0   500 15396 15134  20   0 209040 12676 pipe_w S    ?          0:00  \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/b2g-ota -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/b2g-ota.py --import-git-ref-cache --push-loop --export-git-ref-cache
0   500 15406 15396  20   0  89996  8912 wait   S    ?          0:00      \_ python /usr/local/bin/gittool.py -r origin/b2g-ota https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/b2g-ota/build/manifests
0   500 15409 15406  20   0  16484  1608 pipe_w S    ?          0:00          \_ git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/*
0   500 15410 15409  20   0  91036  6500 pipe_w S    ?          0:00              \_ git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git
0   500 15412 15410  20   0  18888  2160 pipe_w S    ?          0:00                  \_ git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ kill -hup 15412
[cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps flwww -s $(pgrep bumper)
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   500 15134 15133  20   0 106096  1276 wait   Ss   ?          0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
0   500 15396 15134  20   0 209040 12688 poll_s S    ?          0:00  \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/b2g-ota -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/b2g-ota.py --import-git-ref-cache --push-loop --export-git-ref-cache
service decommissioned in bug 1277297
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.