1250588 - git.m.o not serving repos via https causing: b2g_bumper.stamp is CRITICAL: FILE_AGE CRITICAL

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Unsure how important it is at this point, but bumperbot has been alerting in #buildduty for an hour or two now.

Comment 1

[Hal Wine [:hwine] use NI!]

bumper unhappy
    [10:22] < nagios-rele> | Tue 10:22:29 PST [4017] buildbot-master66.bb.releng.usw2.mozilla.com:File Age -
    /builds/b2g_bumper/b2g_bumper.stamp is CRITICAL: FILE_AGE CRITICAL: /builds/b2g_bumper/b2g_bumper.stamp is 6956
    seconds old and 0 bytes (http://m.mozilla.org/File+Age+-+/builds/b2g_bumper/b2g_bumper.stamp)

Following https://wiki.mozilla.org/ReleaseEngineering/Applications/Bumper#Troubleshooting yields master.log complaining about no-such-repo on git.mozilla.org via https: for repos that exist via ssh: connection:
    10:27:42  WARNING - https://git.mozilla.org/b2g/device-sony-common:refs/heads/sony-aosp-l - got output: fatal: 
              remote error: FATAL: R any b2g/device-sony-common gitweb DENIED by fallthru
    10:35:42  WARNING - https://git.mozilla.org/b2g/device-sony-scorpion_windy:refs/heads/sony-aosp-l - got output: 
              fatal: remote error: FATAL: R any b2g/device-sony-scorpion_windy gitweb DENIED by fallthru
    10:37:12  WARNING - https://git.mozilla.org/b2g/device-sony-common:refs/heads/sony-aosp-l - got output: fatal: 
              remote error: FATAL: R any b2g/device-sony-common gitweb DENIED by fallthru
    10:37:43  WARNING - https://git.mozilla.org/b2g/device-sony-scorpion_windy:refs/heads/sony-aosp-l - got output: 
              fatal: remote error: FATAL: R any b2g/device-sony-scorpion_windy gitweb DENIED by fallthru

Moving to dev services

Comment 2

[Kendall Libby [:fubar] (he/him)]

Indeed those repos do not exist, nor have they ever. Hal says it's a bad manifest.

Comment 3

[Hal Wine [:hwine] use NI!]

Looks like several repos were added before being mirrored:
  https://github.com/mozilla-b2g/b2g-manifest/commit/7fafd8003c6157e23f1145f7c58c4af0ebf699aa

Those changes should be backed out until the new repositories are mirrored.

Comment 4

[Hal Wine [:hwine] use NI!]

Downtiming this nagios alert for 7 days to avoid noise while this is sorted out.

Comment 5

[:gerard-majax]

Sorry, I thought that being part of tier 3 would make us not allowed to ask any new mirroring.

Comment 6

[Hal Wine [:hwine] use NI!]

Well, that would imply a complete shutdown of all automated builds then. We're not there, so you need to request them. We may not be able to respond immediately, which is why I downtimed for 7 days, but that level of support is mostly scripted.

If you don't want the repos mirrored, then short term the change needs to be backed out. If you could indicate your choice in this bug, that will help us keep all the pieces coordinated.

Comment 7

[:gerard-majax]

Thanks! I filed the mirroring bug this morning: bug 1250864

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I just saw the first bumperbot bump since Tuesday go by, after that mirroring bug got deployed.. I think the nagios alert is still downtimed, but this bug can probably be resolved.