1476355 - Support vendoring a subdirectory of a repository

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Kartikaya Gupta (email:[e-mail])]

For webrender vendoring, we take the webrender/, webrender_api/, and wrench/ folders from the repo at https://github.com/servo/webrender and put them into gfx/ in the mozilla-central tree, alongside other mozilla code.

Based on my reading of the moz.yaml format, this setup isn't really supported. Either we could put a moz.yaml file at gfx/moz.yaml which would vendor those three folders, or we could put a moz.yaml file into each of the gfx/webrender/, gfx/webrender_api/, and gfx/wrench/ folders.

In the former case, I don't know what I'd put for the "keep" line [1] because we'd want to keep "everything except {webrender,webrender_api,wrench}" in gfx/. We could manually list all these things, but then we'd have to update it every time we add a new folder which is not really desirable.

In the latter case (having three moz.yaml files) we'd need to vendor from a subdirectory in the repo. i.e. the gfx/webrender/moz.yaml file would want to vendor just the contents of the webrender/ folder from the repo, but not into a nested gfx/webrender/webrender/ folder. The semantics of the "include" [2] aren't explicitly stated but my assumption is that it would keep the folder structure of the files being vendored (not doing so would be problematic in other ways).

So it seems like neither of these approaches would work for our current setup.

[1] https://searchfox.org/mozilla-central/rev/6f86cc3479f80ace97f62634e2c82a483d1ede40/python/mozbuild/mozbuild/moz_yaml.py#114
[2] https://searchfox.org/mozilla-central/rev/6f86cc3479f80ace97f62634e2c82a483d1ede40/python/mozbuild/mozbuild/moz_yaml.py#133

Comment 1

[Gregory Szorc [:gps]]

needinfo glob to triage.

Comment 2

[:glob ✱]

i prefer the later case, and you're right in your reading that this would result in unnecessary directories.

the simplest option may be to add an optional `strip` field to `vendoring`..

  strip: num
  # optional
  # Strip the smallest prefix containing `num` leading slashes from each file name
  # found in the origin.  This is akin to the "-p" switch to "patch".
  # eg. the file `webrenderer/src/lib.rs` with strip=1 will create `src/lib.rs`
  #     within the target directory.

kats - thoughts?

Comment 3

[Kartikaya Gupta (email:[e-mail])]

That seems reasonable. It would be worth explicitly describing the semantics of mixing multiple include entries with this option. e.g. if you have "foo/bar" and "baz/" included with strip: 1, presumably it would copy the contents of baz/ along with bar and lump them all into the same target directory. What happens if there is a baz/bar file as well? Also, would strip apply to exclude and keep entries as well?

Comment 4

[:glob ✱]

thanks - great points.

strip would apply to include and exclude only as those are the entries that apply to the origin repo (keep applies to the local repo).

i agree with you about the handling of multiple includes..

the following moz.yaml fragment:
  include:
    - foo/bar
    - bar
  strip: 1

applied to:
  foo/bar/readme.txt
  foo/bar/lib.rs
  bar/another.js
  something_else.py

would result in:
  bar/readme.txt
  bar/lib.rs
  another.js


where things get interesting are when you strip more directories that exist in the origin.
to be the most obvious thing to do is..

the following moz.yaml fragment:
  include:
    - foo/bar
    - bar
  strip: 2

applied to:
  foo/bar/readme.txt
  foo/bar/lib.rs
  bar/another.js
  something_else.py

would result in:
  readme.txt
  lib.rs
  another.js

Comment 5

[Kartikaya Gupta (email:[e-mail])]

And name collisions? i.e.

the following moz.yaml fragment:
  include:
    - foo/
    - baz/
  strip: 1

applied to:
  foo/bar.txt
  baz/bar.txt

would result in:
  bar.txt    # but which one?

Comment 6

[:glob ✱]

we should throw an error in that case.
moz.yaml would need be to updated to exclude the version of bar.txt we don't want.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Btw in bug 1507524 we're planning to start copying the whole webrender repo in rather than selected subfolders, so this won't be a problem for us anymore.