600s timeout when accessing git.mozilla.org from buildbot era hosts



Developer Services
3 years ago
2 years ago


(Reporter: hwine, Unassigned)





3 years ago
Started happening Thursday evening, and was originally thought to be part of bug 1184422, but now appears separate.

Typical log entry:

    22:44:34 INFO - repo has been initialized in /builds/slave/b2g_m-in_l64-b2g-haz_dep-00000/build
    22:54:34 INFO - Automation Error: mozprocess timed out after 600 seconds running ['./config.sh', '-q', 'emulator-jb', '/builds/slave/b2g_m-in_l64-b2g-haz_dep-00000/build/tmp_manifest/emulator-jb.xml']
    22:54:34 ERROR - timed out after 600 seconds of no output
    22:54:34 ERROR - Return code: -9
checked for corrupted repos on git.m.o under /b2g/ and /external/ with git fsck. no fatal errors (as seen in the libpng corruption issue); only errors seen were 'unterminated header' errors which appear to be a spurious warning that will go away in a newer version of git.
We've been doing our best to trigger more builds and haven't been able to reproduce. At this point, I'm going to reopen trees and see how things go.
Severity: normal → critical

Comment 3

3 years ago
As a wild guess, I did make a modification to the external/caf/kernel/msm.git repo. It did not change the "error in tag" messages such as:
  error in tag e34c586c99f12db9ebf61598a960a27f4e0359cd: unterminated header

In looking at the master on the vcs-sync host (which is what updates the git.m.o copy), I noticed the following about the repo:
  - git fsck did not report any errors or warnings
  - many branches are rejected during push as they contain changes that can not be applied with fast forward.

Research in the past has show the presence of messages like that only impact the specified branch - all other branches update normally. (This would not affect the client pull, as the server only had changes that could be applied via the fast forward technique.) The whole issue of whether or not we should prevent upstream history rewrites from propagating to our mirror needs to be revisited -- but that is a separate issue.

Since we're looking for anomalies, I did re-push kernel/msm.git by:
    - modify repo config locally on git1 server: git config receive.denyNonFastForwards false
    - push from vcs-sync copy: git push -f
    - edit local config to remove the override of default.
At this point:
    - repo instance on git.mozilla.org still reports unterminated headers
    - push from vcs-sync does not report rejected branch pushes (as expected)
service decommissioned in bug 1277297
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.