1429815 - `mach watch` fails with "ValueError: Item already in manifest: chrome/content/*"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Brian Grinstead [:bgrins]]

On an m-c checkout, running `./mach watch` I get the following error:

    Error running mach:

        ['watch']

    The error occurred in code that was called by the mach command. This is either
    a bug in the called code itself or in the way that mach is calling it.

    You should consider filing a bug for this issue.

    If filing a bug, please include the full output of mach, including this error
    message.

    The details of the failure are as follows:

    ValueError: Item already in manifest: chrome/content/*

      File "/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 112, in watch
        return daemon.watch()
      File "/mozilla-central/python/mozbuild/mozbuild/faster_daemon.py", line 278, in watch
        for change in self.output_changes(verbose=verbose):
      File "/mozilla-central/python/mozbuild/mozbuild/faster_daemon.py", line 254, in output_changes
        for change in self.input_changes(verbose=verbose):
      File "/mozilla-central/python/mozbuild/mozbuild/faster_daemon.py", line 203, in input_changes
        input_to_outputs = self.file_copier.input_to_outputs_tree()
      File "/mozilla-central/python/mozbuild/mozbuild/util.py", line 962, in __get__
        setattr(instance, name, self.func(instance))
      File "/mozilla-central/python/mozbuild/mozbuild/faster_daemon.py", line 93, in file_copier
        'faster', 'unified_install_dist_bin'))
      File "/mozilla-central/python/mozbuild/mozpack/manifests.py", line 121, in __init__
        self._load_from_fileobj(fh)
      File "/mozilla-central/python/mozbuild/mozpack/manifests.py", line 158, in _load_from_fileobj
        self.add_pattern_link(base, pattern, dest)
      File "/mozilla-central/python/mozbuild/mozpack/manifests.py", line 297, in add_pattern_link
        (self.PATTERN_LINK, base, pattern, dest))
      File "/mozilla-central/python/mozbuild/mozpack/manifests.py", line 332, in _add_entry
        raise ValueError('Item already in manifest: %s' % dest)

Comment 1

[Brian Grinstead [:bgrins]]

Nick, any idea what's going on here? I don't see `chrome/content/*` in the source: https://searchfox.org/mozilla-central/search?q=chrome%2Fcontent%2F*&path=, but I do see some hits in the objdir: https://dxr.mozilla.org/mozilla-central/search?q=chrome%2Fcontent%2F*%22&redirect=false

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Brian Grinstead [:bgrins] from comment #1)
> Nick, any idea what's going on here? I don't see `chrome/content/*` in the
> source:
> https://searchfox.org/mozilla-central/search?q=chrome%2Fcontent%2F*&path=,
> but I do see some hits in the objdir:
> https://dxr.mozilla.org/mozilla-central/
> search?q=chrome%2Fcontent%2F*%22&redirect=false

I dug into this; here's what is happening.  There's a bug in InstallManifest::add_entries_from, some of the manifest entries bake their destination into both the manifest key and the manifest value, and add_entries_from with a non-empty "base" parameter to preprend to the destination only updates the manifest key and not the value.  Patch inbound.

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1429815 - Fix InstallManifest::add_entries_from with non-trivial base.

There's a bug in InstallManifest::add_entries_from: some of the
manifest entries bake their destination into both the manifest key and
the manifest value, and add_entries_from with a non-empty "base"
parameter to prepend to the destination only updates the manifest key
and not the value.

This bug causes |mach watch| to fail to _read_ the unified manifest
that aggregates all of the build manifests relevant to |mach watch|
that |mach build-backend -b FasterMake| successfully _wrote_, because
the manifest keys are validated against the manifest values written to
disk.

I see no way to address this other than to manually reach into the
manifest values and patch the internal destination parameter, which
this patch does.

Review commit: https://reviewboard.mozilla.org/r/214208/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/214208/

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8944026 [details]
Bug 1429815 - Fix InstallManifest::add_entries_from with non-trivial base.

https://reviewboard.mozilla.org/r/214208/#review222430

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/31a0af4e29bb
Fix InstallManifest::add_entries_from with non-trivial base. r=gps

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/31a0af4e29bb