965608 - lock reporepo to a tag - b2g builds failing with AttributeError: 'list' object has no attribute 'values' | caught OS error 2: No such file or directory while running ['./gonk-misc/add-revision.py', '-o', 'sources.xml', '--force', '.repo/manifest.xml']

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=33766735&tree=B2g-Inbound



15:18:16     INFO - Fetching projects:  99% (116/117)  remote: Counting objects: 11, done.[K
15:18:16     INFO -  remote: Compressing objects:  14% (1/7)   [K
15:18:16     INFO - remote: Compressing objects:  28% (2/7)   [K
15:18:16     INFO - remote: Compressing objects:  42% (3/7)   [K
15:18:16     INFO - remote: Compressing objects:  57% (4/7)   [K
15:18:16     INFO - remote: Compressing objects:  71% (5/7)   [K
15:18:16     INFO - remote: Compressing objects:  85% (6/7)   [K
15:18:16     INFO - remote: Compressing objects: 100% (7/7)   [K
15:18:16     INFO - remote: Compressing objects: 100% (7/7), done.[K
15:18:16     INFO -  remote: Total 7 (delta 4), reused 0 (delta 0)[K
15:18:17     INFO -  Unpacking objects:  14% (1/7)   
15:18:17     INFO - Unpacking objects:  28% (2/7)   
15:18:17     INFO - Unpacking objects:  42% (3/7)   
15:18:17     INFO - Unpacking objects:  57% (4/7)   
15:18:17     INFO - Unpacking objects:  71% (5/7)   
15:18:17     INFO - Unpacking objects:  85% (6/7)   
15:18:17     INFO - Unpacking objects: 100% (7/7)   
15:18:17     INFO - Unpacking objects: 100% (7/7), done.
15:18:17     INFO -  From https://git.mozilla.org/external/google/gerrit/git-repo
15:18:17     INFO -     f1f1137..c9877c7  master     -> origin/master
15:18:17     INFO -     12fd10c..c9877c7  stable     -> origin/stable
15:18:17     INFO -   * [new tag]         v1.12.8    -> v1.12.8
15:18:17     INFO -  
15:18:17     INFO - Fetching projects: 100% (117/117)  
15:18:17     INFO - Fetching projects: 100% (117/117), done.
15:18:17     INFO -  info: A new version of repo is available
15:18:18     INFO -  project .repo/repo/
15:18:18     INFO -  Updating 12fd10c..c9877c7
15:18:18     INFO -  Fast-forward
15:18:18     INFO -   command.py               |   26 +++---
15:18:18     INFO -   docs/manifest-format.txt |   12 +--
15:18:18     INFO -   git_command.py           |    2 +-
15:18:18     INFO -   git_config.py            |    4 +-
15:18:18     INFO -   git_refs.py              |    2 +-
15:18:18     INFO -   hooks/commit-msg         |    4 +-
15:18:18     INFO -   hooks/pre-auto-gc        |    2 +-
15:18:18     INFO -   manifest_xml.py          |   70 +++++++++++-----
15:18:18     INFO -   project.py               |  204 +++++++++++++++++++++++++++++++++++-----------
15:18:18     INFO -   repo                     |   32 ++++----
15:18:18     INFO -   subcmds/branches.py      |    2 +-
15:18:18     INFO -   subcmds/init.py          |   21 +++++
15:18:18     INFO -   subcmds/rebase.py        |    3 +
15:18:18     INFO -   subcmds/sync.py          |   67 +++++++++++----
15:18:18     INFO -   subcmds/upload.py        |   15 +++-
15:18:18     INFO -   15 files changed, 341 insertions(+), 125 deletions(-)
15:18:18     INFO -  info: Restarting repo with latest version
15:18:18     INFO -  Traceback (most recent call last):
15:18:18     INFO -    File "/builds/slave/b2g_b2g-in_emu-jb-d_dep-000000/build/.repo/repo/main.py", line 418, in <module>
15:18:18     INFO -      _Main(sys.argv[1:])
15:18:18     INFO -    File "/builds/slave/b2g_b2g-in_emu-jb-d_dep-000000/build/.repo/repo/main.py", line 394, in _Main
15:18:18     INFO -      result = repo._Run(argv) or 0
15:18:18     INFO -    File "/builds/slave/b2g_b2g-in_emu-jb-d_dep-000000/build/.repo/repo/main.py", line 142, in _Run
15:18:18     INFO -      result = cmd.Execute(copts, cargs)
15:18:18     INFO -    File "/builds/git-shared/repo/repo/subcmds/sync.py", line 622, in Execute
15:18:18     INFO -      _PostRepoUpgrade(self.manifest, quiet=opt.quiet)
15:18:18     INFO -    File "/builds/git-shared/repo/repo/subcmds/sync.py", line 705, in _PostRepoUpgrade
15:18:18     INFO -      for project in manifest.projects.values():
15:18:18     INFO -  AttributeError: 'list' object has no attribute 'values'
15:18:18     INFO - Return code: 0
15:18:18     INFO - Running command: ['./gonk-misc/add-revision.py', '-o', 'sources.xml', '--force', '.repo/manifest.xml'] in /builds/slave/b2g_b2g-in_emu-jb-d_dep-000000/build
15:18:18     INFO - Copy/paste: ./gonk-misc/add-revision.py -o sources.xml --force .repo/manifest.xml
15:18:18    FATAL - caught OS error 2: No such file or directory while running ['./gonk-misc/add-revision.py', '-o', 'sources.xml', '--force', '.repo/manifest.xml']
15:18:18    FATAL - Running post_fatal callback...
15:18:18    FATAL - Exiting -1
program finished with exit code 255

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

All trees closed, by the way.

Comment 2

[Aki Sasaki (not active)]

Pretty sure this is due to https://gerrit.googlesource.com/git-repo/+/v1.12.9 :

"the value of Manifest.projects has changed from being the dictionary
to the values of the dictionary.  Here we handle this change
correctly on a PostRepoUpgrade."

Looks like 1.12.8 is busted, 1.12.9 is fixed.  But we may be stuck on 1.12.8 until we help it along somehow =\

I'm leaning towards using --repo-branch to the repo call, and locking it to 1.12.7 or 1.12.9 or something so upstream can't break the world like this.  Not sure if we also have to do cleanup on every machine as well.

Comment 3

[Aki Sasaki (not active)]

Attachment: repo_repo

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8367711 [details] [diff] [review]
repo_repo

r+ to avoid rando-bustage in future. You could bump to 1.12.9 now that we know that's building OK.

Comment 5

[Aki Sasaki (not active)]

Catlee: this seems to conflict with http://hg.mozilla.org/users/asasaki_mozilla.com/ash-mozharness/rev/a1b72f6e9ced on ash -- is that going to land soon?  Or should I test my patch out and land?

Comment 6

[Aki Sasaki (not active)]

Went green on Ash.
I'll land this tomorrow.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 8367711 [details] [diff] [review]
repo_repo

https://hg.mozilla.org/build/mozharness/rev/15f2c2b6e2c6

Comment 8

[Aki Sasaki (not active)]

In production.

Comment 9

[Armen [:armenzg]]

This *might* be causing bug 966261

Comment 10

[Armen [:armenzg]]

Backed out for now:
http://hg.mozilla.org/build/mozharness/rev/15dcf07b51b0
http://hg.mozilla.org/build/mozharness/rev/e7dc863f3d76

catlee and I cannot wrap our heads around the fact that it does not affect anything else besides try builds.

Comment 11

[Aki Sasaki (not active)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #10)
> Backed out for now:
> http://hg.mozilla.org/build/mozharness/rev/15dcf07b51b0
> http://hg.mozilla.org/build/mozharness/rev/e7dc863f3d76
> 
> catlee and I cannot wrap our heads around the fact that it does not affect
> anything else besides try builds.

Weird.  Maybe it's a clobber thing?  But then nightlies would be affected.

Comment 12

[Aki Sasaki (not active)]

1. fix upstream (get repo to allow for a[n unsigned] tag
2. if we can't do that, we can pre-clone the repo repo to hopefully avoid this bug.

Comment 13

[Aki Sasaki (not active)]

Catlee ended up making many repo improvements in bug 1007259... now managed with gittool.