993632 - gaia github->git.m.o sync is broken

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Jordan Lund (:jlund)]

checking on bm66

/builds/b2g_bumper/b2g_bumper.log and /builds/gaia_bumper/gaia_bumper.log both have recent output picking up revisions and I can not see any error output.

aki is landing a patch that may help debugging status of the bumper though.

Comment 2

[Aki Sasaki (not active)]

Attachment: log rotate

Comment 3

[Jordan Lund (:jlund)]

looking at procs I am getting varied results:

[[e-mail] b2g_bumper]$ ps -elfy | grep bumper
S cltbld    8429 23396  0  80   0   844 25825 pipe_w 12:54 pts/0    00:00:00 grep bumper

[[e-mail] b2g_bumper]$ ps -elfy | grep bumper
S cltbld    8464  8462  0  80   0  1244  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8465  8463  0  80   0  1204  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_gaia_bumper.sh
S cltbld    8488  8465  1  80   0 10140 24252 pipe_w 12:55 ?        00:00:00 python mozharness/scripts/bump_gaia_json.py -c gaia_bumper/gaia_json.py
S cltbld    8494  8464  0  80   0   572  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8496  8494  1  80   0 11796 27161 pipe_w 12:55 ?        00:00:00 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/master.py
S cltbld    8497  8464  0  80   0   576  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8499  8497  1  80   0 11792 27159 pipe_w 12:55 ?        00:00:00 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/v1.4.py
S cltbld    8500  8464  0  80   0   576  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8502  8500  2  80   0 11788 27159 pipe_w 12:55 ?        00:00:00 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/v1.3.py
S cltbld    8503  8464  0  80   0   576  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8504  8503  1  80   0 11736 27096 pipe_w 12:55 ?        00:00:00 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/v1.3t.py
S cltbld    8506  8504  0  80   0  8380 20243 wait   12:55 ?        00:00:00 python /usr/local/bin/hgtool.py --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-b2g28_v1_3t.hg https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t /builds/b2g_bumper/v1.3t/build/mozilla-b2g28_v1_3t
S cltbld    8510  8502  0  80   0  8380 20243 wait   12:55 ?        00:00:00 python /usr/local/bin/hgtool.py --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-b2g28_v1_3.hg https://hg.mozilla.org/releases/mozilla-b2g28_v1_3 /builds/b2g_bumper/v1.3/build/mozilla-b2g28_v1_3
S cltbld    8511  8496  1  80   0  8384 20245 wait   12:55 ?        00:00:00 python /usr/local/bin/hgtool.py --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/b2g-inbound.hg https://hg.mozilla.org/integration/b2g-inbound /builds/b2g_bumper/master/build/b2g-inbound
S cltbld    8512  8499  1  80   0  8380 20243 wait   12:55 ?        00:00:00 python /usr/local/bin/hgtool.py --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-aurora.hg https://hg.mozilla.org/releases/mozilla-aurora /builds/b2g_bumper/v1.4/build/mozilla-aurora
S cltbld    8537 23396  0  80   0   848 25826 pipe_w 12:55 pts/0    00:00:00 grep bumper


[[e-mail] b2g_bumper]$ ps -elfy | grep bumper
S cltbld    8464  8462  0  80   0  1244  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8465  8463  0  80   0  1204  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_gaia_bumper.sh
S cltbld    8488  8465  0  80   0 11504 26122 pipe_w 12:55 ?        00:00:00 python mozharness/scripts/bump_gaia_json.py -c gaia_bumper/gaia_json.py
S cltbld    8494  8464  0  80   0   572  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
R cltbld    8496  8494  5  80   0 17508 332314 -     12:55 ?        00:00:03 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/master.py
S cltbld    8497  8464  0  80   0   576  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8499  8497  5  80   0 17672 332055 futex_ 12:55 ?       00:00:03 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/v1.4.py
S cltbld    8500  8464  0  80   0   576  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8502  8500  5  80   0 17556 332049 futex_ 12:55 ?       00:00:03 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/v1.3.py
S cltbld    8503  8464  0  80   0   576  2322 wait   12:55 ?        00:00:00 /bin/bash /usr/local/bin/run_b2g_bumper.sh
S cltbld    8504  8503  1  80   0 13784 281879 pipe_w 12:55 ?       00:00:01 python ../mozharness/scripts/b2g_bumper.py -c b2g_bumper/v1.3t.py
R cltbld    9797  8504  6  80   0 82420 65937 -      12:55 ?        00:00:02 /tools/python27/bin/python2.7 /usr/local/bin/hg --config web.cacerts=/etc/pki/tls/certs/ca-bundle.crt commit -u B2G Bumper Bot <[e-mail]> -m Bumping manifests a=b2g-bump
S cltbld   11149 23396  0  80   0   864 25826 pipe_w 12:56 pts/0    00:00:00 grep bumper

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8403520 [details] [diff] [review]
log rotate

so I see bump_gaia_json.py uses this: http://mxr.mozilla.org/build/source/mozharness/configs/gaia_bumper/gaia_json.py#6

but looks like that's not the config used for b2g_bumper.py

so lgtm more backups!

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 8403520 [details] [diff] [review]
log rotate

http://hg.mozilla.org/build/mozharness/rev/379a327473c1

These will show up in /builds/b2g_bumper/*/build/upload/logs

Comment 6

[Hal Wine [:hwine] use NI!]

For some reason, we can no longer connect to github from the conversion box:

[[e-mail]3 tmp]$ nc -vz github.com 9418
nc: connect to github.com port 9418 (tcp) failed: Connection timed out
[[e-mail]3 tmp]$

Comment 7

[Aki Sasaki (not active)]

"github is working on changing routing preferences on their end to route our traffic to ntt instead of L3 in hopes that will fix the problem with connectivity to them"

"Mozilla has escalated the issue to above.net and we're attempting to resolve the issue ourselves by modifying our routing policies."

Comment 8

[Arzhel Younsi [:XioNoX]]

We applied a temporary fix to solve comment 6 (and everything in .dmz.scl3) the time this issue is fixed.

Comment 9

[Aki Sasaki (not active)]

https://bugzilla.mozilla.org/show_bug.cgi?id=993766#c22 and followup comments.