1792224 - Support vendoring individual files from an upstream repository

Status: RESOLVED FIXED

(Developer Infrastructure :: Mach Vendor & Updatebot, enhancement)

Description

[Tom Ritter [:tjr] (OOTO until 7/21)]

Sometimes we don't want to clone an entire upstream repo just to get 5 files from it.

This is a leaky abstraction. We already have include and exclude that would allow us to accomplish this. It's just that cloning llvm or netbsd just to get 5 files is inefficient. Related, we have this kind of complicated code that's designed to detect spurious updates - which is when we update from upstream but nothing changes, because the upstream changes were e.g. in files we excluded.

We could get rid of the spurious update code if we expressed everything in terms of 'individual-files' - that is, only the files in tree get checked for updates, and if there is none then there is no update. But if we have a hundred files in-tree that isn't efficient either.

Ultimately, it seems like this is probably the best trade-off...

Comment 1

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Add support for individual-files in the moz.yaml r?jewilde

Depends on D157952

Comment 2

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Add tests for individual-files yaml parsing r?jewilde

Depends on D158044

Comment 3

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Do some refactoring in preparation for real work r?jewilde

If there is a vendoring section missing in the moz.yaml,
we will error out. Detect this right away and give a better error.

Perform some cleanup since we know we have a vendoring section.

Depends on D158045

Comment 4

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Do not allow branch in vendoring tasks r?jewilde

I noticed that 'branch' is only used in Updatebot in the
Commit Alert task (we made it for Angle.) It doesn't do
anything anywhere else. So assert this in the validation checks

Depends on D158046

Comment 5

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Move the Updatebot section below vendoring to make it more logical r?jewilde

Depends on D158047

Comment 6

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Make the tracking field validation simpler r?jewilde

Depends on D158048

Comment 7

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Do not allow tag tracking with individual-files r?jewilde

Depends on D158049

Comment 8

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1792224: Implement vendoring individual files r?jewilde

Depends on D158050

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b9b8631a5eaf
Add support for individual-files in the moz.yaml r=jewilde
https://hg.mozilla.org/integration/autoland/rev/87392c6e2c7b
Add tests for individual-files yaml parsing r=jewilde
https://hg.mozilla.org/integration/autoland/rev/36b217d098a4
Do some refactoring in preparation for real work r=jewilde
https://hg.mozilla.org/integration/autoland/rev/2bcdcab75a23
Do not allow branch in vendoring tasks r=jewilde
https://hg.mozilla.org/integration/autoland/rev/e1a8a5c347f8
Move the Updatebot section below vendoring to make it more logical r=jewilde
https://hg.mozilla.org/integration/autoland/rev/14af6d382874
Make the tracking field validation simpler r=jewilde
https://hg.mozilla.org/integration/autoland/rev/d7884101b2c9
Do not allow tag tracking with individual-files r=jewilde
https://hg.mozilla.org/integration/autoland/rev/c3943b948dc9
Implement vendoring individual files r=jewilde

Comment 10

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/b9b8631a5eaf
https://hg.mozilla.org/mozilla-central/rev/87392c6e2c7b
https://hg.mozilla.org/mozilla-central/rev/36b217d098a4
https://hg.mozilla.org/mozilla-central/rev/2bcdcab75a23
https://hg.mozilla.org/mozilla-central/rev/e1a8a5c347f8
https://hg.mozilla.org/mozilla-central/rev/14af6d382874
https://hg.mozilla.org/mozilla-central/rev/d7884101b2c9
https://hg.mozilla.org/mozilla-central/rev/c3943b948dc9