1036819 - build/* repos currently not able to push tags to github

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

We have been getting emails like this for three hours for failures of pushing tags to github repos. Suspected (but not confirmed) cause is described in bug 1020613.

Temporary fix is to disable pushing of tags for build/* repos.


On 10 Jul 2014, at 08:57, [e-mail] wrote:

Unable to push these repos:
build-buildbot-configs: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-buildbot-configs to build-buildbot-configs-github!

build-tools: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-tools to build-tools-github!



Summary is non-zero:

error - Unable to push build-buildbot-configs.
error - Unable to push build-tools.

06:52:57     INFO - Copy/paste: git push -f [e-mail]:mozilla/build-buildbot-configs.git +refs/tags/FIREFOX_24_0b3_RELEASE:refs/tags/FIREFOX_24_0b3_RELEASE +refs/tags/FIREFOX_24_0b3_BUILD1:refs/tags/FIREFOX_24_0b3_BUILD1 +refs/tags/THUNDERBIRD_24_0b1_RELEASE:refs/tags/THUNDERBIRD_24_0b1_RELEASE +refs/tags/THUNDERBIRD_24_0b1_BUILD4:refs/tags/THUNDERBIRD_24_0b1_BUILD4 +refs/tags/THUNDERBIRD_24_0b1_BUILD3:refs/tags/THUNDERBIRD_24_0b1_BUILD3 +refs/tags/FIREFOX_23_0_1_BUILD1:refs/tags/FIREFOX_23_0_1_BUILD1 +refs/tags/THUNDERBIRD_24_0b1_BUILD2:refs/tags/THUNDERBIRD_24_0b1_BUILD2 +refs/tags/THUNDERBIRD_24_0b1_BUILD1:refs/tags/THUNDERBIRD_24_0b1_BUILD1 +refs/tags/FIREFOX_24_0b2_RELEASE:refs/tags/FIREFOX_24_0b2_RELEASE +refs/tags/FIREFOX_24_0b2_BUILD1:refs/tags/FIREFOX_24_0b2_BUILD1
06:52:57     INFO - Using partial env: {'GIT_SSH': '/opt/vcs2vcs/vcs_sync/mozharness/external_tools/git-ssh-wrapper.sh',
06:52:57     INFO -  'GIT_SSH_KEY': '~/.ssh/releng-github-id_rsa'}
06:52:57     INFO - Calling ['git', 'push', '-f', '[e-mail]:mozilla/build-buildbot-configs.git', '+refs/tags/FIREFOX_24_0b3_RELEASE:refs/tags/FIREFOX_24_0b3_RELEASE', '+refs/tags/FIREFOX_24_0b3_BUILD1:refs/tags/FIREFOX_24_0b3_BUILD1', '+refs/tags/THUNDERBIRD_24_0b1_RELEASE:refs/tags/THUNDERBIRD_24_0b1_RELEASE', '+refs/tags/THUNDERBIRD_24_0b1_BUILD4:refs/tags/THUNDERBIRD_24_0b1_BUILD4', '+refs/tags/THUNDERBIRD_24_0b1_BUILD3:refs/tags/THUNDERBIRD_24_0b1_BUILD3', '+refs/tags/FIREFOX_23_0_1_BUILD1:refs/tags/FIREFOX_23_0_1_BUILD1', '+refs/tags/THUNDERBIRD_24_0b1_BUILD2:refs/tags/THUNDERBIRD_24_0b1_BUILD2', '+refs/tags/THUNDERBIRD_24_0b1_BUILD1:refs/tags/THUNDERBIRD_24_0b1_BUILD1', '+refs/tags/FIREFOX_24_0b2_RELEASE:refs/tags/FIREFOX_24_0b2_RELEASE', '+refs/tags/FIREFOX_24_0b2_BUILD1:refs/tags/FIREFOX_24_0b2_BUILD1'] with output_timeout 1800
06:54:58     INFO -  Received disconnect from 192.30.252.131: 11: Bye Bye
06:54:58    ERROR -  fatal: The remote end hung up unexpectedly
06:54:58    ERROR - Return code: 128
06:54:58    ERROR - build-buildbot-configs: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-buildbot-configs to build-buildbot-configs-github!
06:54:58    ERROR - Unable to push build-buildbot-configs.
06:54:58     INFO - Writing to file /opt/vcs2vcs/vcs_sync/build/upload/repo_update.json
06:54:58     INFO - Contents:
06:54:58     INFO -  {
06:54:58     INFO -      "last_pull_datetime": "2014-07-10 06:51 UTC", 
06:54:58     INFO -      "last_pull_timestamp": 1404975105, 
--
06:54:59     INFO - Copy/paste: git push -f [e-mail]:mozilla/build-tools.git +refs/notes/commits:refs/notes/commits +refs/heads/master:refs/heads/master +refs/tags/FIREFOX_31_0b8_RELEASE_RUNTIME:refs/tags/FIREFOX_31_0b8_RELEASE_RUNTIME +refs/tags/FIREFOX_31_0b8_BUILD1_RUNTIME:refs/tags/FIREFOX_31_0b8_BUILD1_RUNTIME +refs/tags/FIREFOX_31_0b8_RELEASE:refs/tags/FIREFOX_31_0b8_RELEASE +refs/tags/FIREFOX_31_0b8_BUILD1:refs/tags/FIREFOX_31_0b8_BUILD1 +refs/tags/FENNEC_31_0b8_RELEASE:refs/tags/FENNEC_31_0b8_RELEASE +refs/tags/FENNEC_31_0b8_BUILD1:refs/tags/FENNEC_31_0b8_BUILD1 +refs/tags/THUNDERBIRD_31_0b2_RELEASE_RUNTIME:refs/tags/THUNDERBIRD_31_0b2_RELEASE_RUNTIME +refs/tags/THUNDERBIRD_31_0b2_BUILD1_RUNTIME:refs/tags/THUNDERBIRD_31_0b2_BUILD1_RUNTIME
06:54:59     INFO - Using partial env: {'GIT_SSH': '/opt/vcs2vcs/vcs_sync/mozharness/external_tools/git-ssh-wrapper.sh',
06:54:59     INFO -  'GIT_SSH_KEY': '~/.ssh/releng-github-id_rsa'}
06:54:59     INFO - Calling ['git', 'push', '-f', '[e-mail]:mozilla/build-tools.git', '+refs/notes/commits:refs/notes/commits', '+refs/heads/master:refs/heads/master', '+refs/tags/FIREFOX_31_0b8_RELEASE_RUNTIME:refs/tags/FIREFOX_31_0b8_RELEASE_RUNTIME', '+refs/tags/FIREFOX_31_0b8_BUILD1_RUNTIME:refs/tags/FIREFOX_31_0b8_BUILD1_RUNTIME', '+refs/tags/FIREFOX_31_0b8_RELEASE:refs/tags/FIREFOX_31_0b8_RELEASE', '+refs/tags/FIREFOX_31_0b8_BUILD1:refs/tags/FIREFOX_31_0b8_BUILD1', '+refs/tags/FENNEC_31_0b8_RELEASE:refs/tags/FENNEC_31_0b8_RELEASE', '+refs/tags/FENNEC_31_0b8_BUILD1:refs/tags/FENNEC_31_0b8_BUILD1', '+refs/tags/THUNDERBIRD_31_0b2_RELEASE_RUNTIME:refs/tags/THUNDERBIRD_31_0b2_RELEASE_RUNTIME', '+refs/tags/THUNDERBIRD_31_0b2_BUILD1_RUNTIME:refs/tags/THUNDERBIRD_31_0b2_BUILD1_RUNTIME'] with output_timeout 1800
06:57:00     INFO -  Received disconnect from 192.30.252.128: 11: Bye Bye
06:57:00    ERROR -  fatal: The remote end hung up unexpectedly
06:57:00    ERROR - Return code: 128
06:57:00    ERROR - build-tools: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-tools to build-tools-github!
06:57:00    ERROR - Unable to push build-tools.
06:57:00     INFO - Writing to file /opt/vcs2vcs/vcs_sync/build/upload/repo_update.json
06:57:00     INFO - Contents:
06:57:00     INFO -  {
06:57:00     INFO -      "last_pull_datetime": "2014-07-10 06:51 UTC", 
06:57:00     INFO -      "last_pull_timestamp": 1404975105, 
--
06:57:00     INFO -              "push_datetime": "2014-07-09 23:06 UTC", 
06:57:00     INFO -              "push_timestamp": 1404947179
06:57:00     INFO -          }
06:57:00     INFO -      }
06:57:00     INFO -  }
06:57:00    FATAL - Unable to push these repos:
06:57:00    FATAL - build-buildbot-configs: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-buildbot-configs to build-buildbot-configs-github!
06:57:00    FATAL - 
06:57:00    FATAL - build-tools: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-tools to build-tools-github!
06:57:00    FATAL - 
06:57:00    FATAL - Running post_fatal callback...
06:57:00     INFO - Job took 318 seconds.
06:57:00     INFO - Getting output from command: ['egrep', '-C5', '^[0-9:]+ +(ERROR|CRITICAL|FATAL) -', '/opt/vcs2vcs/vcs_sync/logs/build-repos_info.log']
06:57:00     INFO - Copy/paste: egrep -C5 "^[0-9:]+ +(ERROR|CRITICAL|FATAL) -" /opt/vcs2vcs/vcs_sync/logs/build-repos_info.log

Comment 1

[Pete Moore [:pmoore][:pete]]

Attachment: bug1036819_mozharness.patch

Sorry for delay, I had a meeting with simone about puppet stuff. If you're not around, I'll get simone to review instead, no worries.

Thanks!
Pete

Comment 2

[Simone Bruno [:simone]]

Comment on attachment 8453663 [details] [diff] [review]
bug1036819_mozharness.patch

Review of attachment 8453663 [details] [diff] [review]:
-----------------------------------------------------------------

Rubber-stamped!

Comment 3

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8453663 [details] [diff] [review]
bug1036819_mozharness.patch

Checked in, and merged to production...

Comment 4

[Pete Moore [:pmoore][:pete]]

This issue is now resolved.

On 10 Jul 2014, at 13:17, [e-mail] wrote:

Summary is non-zero:

info - Previously unsuccessful push of build-buildbot-configs is now successful!
info - Previously unsuccessful push of build-buildbotcustom is now successful!
info - Previously unsuccessful push of build-mozharness is now successful!