966261 - B2G emulator build failure on try | OSError: [Errno 20] Not a directory: '.repo'

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Andrew Halberstadt [:ahal]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=33876383&tree=Try

This seems to only be affecting try (that I've noticed). Looks unrelated to any gecko changeset.

Comment 1

[Armen [:armenzg]]

I don't see failures on:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=emulator
https://tbpl.mozilla.org/?tree=Mozilla-Central&jobname=emulator

The last non-failing try job was past 2am last night:
https://tbpl.mozilla.org/php/getParsedLog.php?id=33874395&tree=Try&full=1

06:44:41     INFO -  gpg: keyring `/home/cltbld/.repoconfig/gnupg/secring.gpg' created
06:44:41     INFO -  gpg: keyring `/home/cltbld/.repoconfig/gnupg/pubring.gpg' created
06:44:41     INFO -  gpg: /home/cltbld/.repoconfig/gnupg/trustdb.gpg: trustdb created
06:44:41     INFO -  gpg: key 920F5C65: public key "Repo Maintainer <[e-mail]>" imported
06:44:41     INFO -  gpg: key 692B382C: public key "Conley Owens <[e-mail]>" imported
06:44:41     INFO -  gpg: Total number processed: 2
06:44:41     INFO -  gpg:               imported: 2  (RSA: 1)
06:44:45     INFO -  fatal: refs/remotes/origin/v1.12.9^0: not a valid SHA1
06:44:45     INFO -  fatal: repo init failed; run without --quiet to see why
06:44:45     INFO -  Traceback (most recent call last):
06:44:45     INFO -    File "/builds/slave/b2g_try_emu_dep-00000000000000/build/repo", line 758, in <module>
06:44:45     INFO -      main(sys.argv[1:])
06:44:45     INFO -    File "/builds/slave/b2g_try_emu_dep-00000000000000/build/repo", line 732, in main
06:44:45     INFO -      os.rmdir(repodir)
06:44:45     INFO -  OSError: [Errno 20] Not a directory: '.repo'
06:44:45    ERROR - Return code: 1
06:44:45    FATAL - Halting on failure while running ['/builds/slave/b2g_try_emu_dep-00000000000000/build/repo', 'init', '--repo-url', 'https://git.mozilla.org/external/google/gerrit/git-repo.git', '--no-repo-verify', '--repo-branch', 'v1.12.9', '-q', '-u', '/builds/slave/b2g_try_emu_dep-00000000000000/build/tmp_manifest', '-m', 'generic.xml', '-b', 'master']
06:44:45    FATAL - Running post_fatal callback...
06:44:45    FATAL - Exiting 1

Comment 2

[Armen [:armenzg]]

I see this in one of the failing slaves:
[[e-mail] build]$ ls -la | grep repo
-rwxrwxr-x  1 cltbld cltbld 23618 Jan 31 06:41 repo
lrwxrwxrwx  1 cltbld cltbld    23 Jan 31 06:44 .repo -> /builds/git-shared/repo

Comment 3

[Armen [:armenzg]]

Attachment: without quiet

This is the failing command:
 /builds/slave/b2g_try_emu_dep-00000000000000/build/repo init --repo-url https://git.mozilla.org/external/google/gerrit/git-repo.git --no-repo-verify --repo-branch v1.12.9 -q -u /builds/slave/b2g_try_emu_dep-00000000000000/build/tmp_manifest -m generic.xml -b master

We have this code:
721     if cmd == 'init':
722       if my_git:
723         _SetDefaultsTo(my_git)
724       try:
725         _Init(args)
726       except CloneFailure:
727         for root, dirs, files in os.walk(repodir, topdown=False):
728           for name in files:
729             os.remove(os.path.join(root, name))
730           for name in dirs:
731             os.rmdir(os.path.join(root, name))
732         os.rmdir(repodir)
733         sys.exit(1)
734       repo_main, rel_repo_dir = _FindRepo()

Comment 4

[Armen [:armenzg]]

mwu: any idea on what is going on?

Comment 5

[Armen [:armenzg]]

I think, this is the problem:
fatal: branch 'v1.12.9' has not been signed

Why are we not hitting this on other branches?

Comment 6

[Michael Wu [:mwu]]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #4)
> mwu: any idea on what is going on?

No idea.

Comment 7

[Armen [:armenzg]]

We backed the change out: http://hg.mozilla.org/build/mozharness/rev/e7dc863f3d76

jhford, any ideas on what could have been wrong with using --repo-tag?
It seems that it was *only* affecting try builds.

Comment 8

[Armen [:armenzg]]

I've re-triggered https://tbpl.mozilla.org/?jobname=emulator&rev=735a648bca0d to see if the backout would affect the other trees or not.

I've also re-triggered jobs on the try server to see if it starts working as it did in our local testing:
https://tbpl.mozilla.org/?tree=Try&jobname=emulator&rev=28d9445b205e

We should know in less than 15 minutes.

Comment 9

[Armen [:armenzg]]

So far so good.

Comment 10

[Armen [:armenzg]]

Please re-open if it happens again.