Closed Bug 1230586 (bumper_hang_from_git) Opened 10 years ago Closed 9 years ago

[tracker] socket hang on https pull from git.mozilla.org

Categories

(Developer Services :: Git, defect)

Product:
Developer Services
Component:
Git
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: hwine, Unassigned)

References

Details

(Whiteboard: [b2g_bumper])

Starting a tracker. This is first occurrence afaik. We a similar issue on hgweb heads for a while. Will wait to see data on how often this is happening.
b2g_bumper hung from buildbot-master66.bb.releng.usw2.mozilla.com for approximately 1 hour. kill -hup cleared. (There is a nagios alert in #buildduty for b2g_bumper staleness is how this was noticed.) Fri Dec 4 08:41:57 PST 2015 [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps flwww -s 3516 F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 0 500 3516 3511 20 0 106096 1276 wait Ss ? 0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh 0 500 5428 3516 20 0 209228 12964 pipe_w S ? 0:00 \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/v2.2r -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/v2.2r.py --import-git-ref-cache --push-loop --export-git-ref-cache 0 500 5616 5428 20 0 89996 8912 wait S ? 0:00 \_ python /usr/local/bin/gittool.py -r origin/v2.2r https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/v2.2r/build/manifests 0 500 5624 5616 20 0 16484 1604 pipe_w S ? 0:00 \_ git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/* 0 500 5625 5624 20 0 91176 6500 pipe_w S ? 0:00 \_ git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git 0 500 5639 5625 20 0 18792 2132 pipe_w S ? 0:00 \_ git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/ [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps -flwwws 3516 F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S cltbld 3516 3511 0 80 0 - 26524 wait 07:40 ? 00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh 0 S cltbld 5428 3516 0 80 0 - 52307 pipe_w 07:42 ? 00:00:00 python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/v2.2r -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/v2.2r.py --import-git-ref-cache --push-loop --export-git-ref-cache 0 S cltbld 5616 5428 0 80 0 - 22499 wait 07:43 ? 00:00:00 python /usr/local/bin/gittool.py -r origin/v2.2r https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/v2.2r/build/manifests 0 S cltbld 5624 5616 0 80 0 - 4121 pipe_w 07:43 ? 00:00:00 git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/* 0 S cltbld 5625 5624 0 80 0 - 22794 pipe_w 07:43 ? 00:00:00 git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git 0 S cltbld 5639 5625 0 80 0 - 4698 pipe_w 07:43 ? 00:00:00 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/ [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ kill -hup 5639 [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps -flwwws 3516 F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S cltbld 3516 3511 0 80 0 - 26524 wait 07:40 ? 00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh 0 S cltbld 5428 3516 0 80 0 - 52307 poll_s 07:42 ? 00:00:00 python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/v2.2r -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/v2.2r.py --import-git-ref-cache --push-loop --export-git-ref-cache
Alias: bumper_hang_from_git
Whiteboard: [b2g_bumper]
This has happened before, see bug 1040062. However, we now have web heads serving http. At time of other report, there was only a single host supporting git.mozilla.org. Troubleshooting instructions used by releng are at: https://wiki.mozilla.org/ReleaseEngineering/Applications/Bumper#Troubleshooting Please update there if additional data is requested.
See Also: → 1040062
root 1308 0.0 0.0 117324 780 ? Ss Mar14 3:30 crond root 20204 0.0 0.0 136088 1304 ? S 11:50 0:00 \_ CROND cltbld 20207 0.0 0.0 106096 1276 ? Ss 11:50 0:00 | \_ /bin/bash /usr/local/bin/run_b2g_bumper.sh cltbld 22042 0.0 0.3 209284 13072 ? S 11:53 0:00 | \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/master -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/master.py --import-git-ref-cache --push-loop --export-git-ref-cache cltbld 22052 0.0 0.2 89988 8904 ? S 11:53 0:00 | \_ python /usr/local/bin/gittool.py -r origin/master https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/master/build/manifests cltbld 22055 0.0 0.0 16484 1608 ? S 11:53 0:00 | \_ git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/* cltbld 22056 0.0 0.1 91036 6504 ? S 11:53 0:00 | \_ git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git cltbld 22058 0.0 0.0 18816 2156 ? S 11:53 0:00 | \_ git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/ after killing 22058.... root 1308 0.0 0.0 117324 780 ? Ss Mar14 3:30 crond root 20204 0.0 0.0 136088 1304 ? S 11:50 0:00 \_ CROND cltbld 20207 0.0 0.0 106096 1276 ? Ss 11:50 0:00 \_ /bin/bash /usr/local/bin/run_b2g_bumper.sh cltbld 22042 0.0 0.3 209284 13084 ? S 11:53 0:00 \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/master -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/master.py --import-git-ref-cache --push-loop --export-git-ref-cache
Another one: [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps flwww -s $(pgrep bumper) F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 0 500 22743 22739 20 0 106096 1224 wait Ss ? 0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh 0 500 15134 15133 20 0 106096 1276 wait Ss ? 0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh 0 500 15396 15134 20 0 209040 12676 pipe_w S ? 0:00 \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/b2g-ota -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/b2g-ota.py --import-git-ref-cache --push-loop --export-git-ref-cache 0 500 15406 15396 20 0 89996 8912 wait S ? 0:00 \_ python /usr/local/bin/gittool.py -r origin/b2g-ota https://git.mozilla.org/b2g/b2g-manifest.git /builds/b2g_bumper/b2g-ota/build/manifests 0 500 15409 15406 20 0 16484 1608 pipe_w S ? 0:00 \_ git fetch -q https://git.mozilla.org/b2g/b2g-manifest.git +refs/heads/*:refs/remotes/origin/* 0 500 15410 15409 20 0 91036 6500 pipe_w S ? 0:00 \_ git-remote-https https://git.mozilla.org/b2g/b2g-manifest.git https://git.mozilla.org/b2g/b2g-manifest.git 0 500 15412 15410 20 0 18888 2160 pipe_w S ? 0:00 \_ git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.mozilla.org/b2g/b2g-manifest.git/ [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ kill -hup 15412 [cltbld@buildbot-master66.bb.releng.usw2.mozilla.com b2g_bumper]$ ps flwww -s $(pgrep bumper) F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 0 500 15134 15133 20 0 106096 1276 wait Ss ? 0:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh 0 500 15396 15134 20 0 209040 12688 poll_s S ? 0:00 \_ python /builds/b2g_bumper/mozharness/scripts/b2g_bumper.py --base-work-dir /builds/b2g_bumper/b2g-ota -c /builds/b2g_bumper/mozharness/configs/b2g_bumper/b2g-ota.py --import-git-ref-cache --push-loop --export-git-ref-cache
service decommissioned in bug 1277297
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.