1637845 - Update moz_yaml to support significantly more expressive instructions

Status: RESOLVED FIXED

(Developer Infrastructure :: Mach Vendor & Updatebot, enhancement)

Description

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

I would like to refactor the moz.yaml schema to support significantly more expressive updating steps. libdav1d and libaom are presently codified as separate python scripts and I'd like to convert these into pure-yaml updates.

When I get that working, let's look at the resulting yaml schema and see how confusing it is and if we want to land it.

This bug will probably accumulate a fairly large patchset as I work towards that goal.

The purpose of this is to make it easier to specify vendor in new libraries (by avoiding duplicating python logic) as well more easily support top-level vendor commands such as --check-for-update which would return an identifier of an update if available.

Comment 1

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Move all mach vendor files to a subdirectory r?glob

This is a straight copy, no edits.

Comment 2

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add 'license-file' to origin, remove 'revision' from vendoring, add 'vendor-directory' r?glob

Add license-file to allow one to specify a special file the
project's license is in.

Add 'vendor-directory' to allow vendoring the files into a
separate directory (e.g. under third_party.) Remove a contradictory
comment.

Remove 'revision' from 'vendoring'. This doesn't make sense to me:
'vendoring' is about how to update the library. In a future iteration
we may wish to restrict updates to a branch; but specifying a revision
is specifically about not updating. It appears the intent of revision
was to identify what revision was currently in-tree; but that information
should live in the 'origin' section.

Depends on D75693

Comment 3

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add source-hosting to the schema and vendoring section r?glob

This will tell us how to interact with the upstream repository.

Depends on D75694

Comment 4

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add 'file-updates' to the moz.yaml schema r?glob

This section is for expressing actions that occur after extracting
the files, before vendoring is completed. While we support running
scripts (or at least, we will...) this section can be used for simple
actions that don't need scripts.

Also, add the dav1d excluded files.

Depends on D75695

Comment 5

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Extend the file-updates schema to support a copy-file action r?glob

Additionally, raise voluptuous.Invalid errors so the message provided in
the Exception is shown to the user on error.

Depends on D75696

Comment 6

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Rename VerifyError to MozYamlVerifyError r?glob

This is more descriptive for where this error is coming from.

Depends on D75697

Comment 7

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Re-implement mach vendor r?glob

This copies the steps of dav1d but tries to make them generic
for future libraries.

Depends on D75698

Comment 8

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

glob: Adding a ni so you see this. I'm not done with this bug; I need to delete the old implementations and port aom over, but I wanted to post it up for early feedback.

Comment 9

[:glob ✱]

(In reply to Tom Ritter [:tjr] (OOTO until 6/1?) from comment #8)

glob: Adding a ni so you see this. I'm not done with this bug; I need to delete the old implementations and port aom over, but I wanted to post it up for early feedback.

Thanks, I've had a quick skim and it looks sane. Will leave some random comments on Phabricator.

What are your thoughts on running black on this code with the default settings? It's highly likely this will be required for python code in m-c as some point, and it would immediately resolve the formatting issues.

Comment 10

[:glob ✱]

Can you also mark the revisions as WIP until they are ready for final review?

Comment 11

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Apply 'black' to the vendor subdirectory r?glob

Depends on D75693

Comment 12

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add a new function 'copy_contents' to mozfile r?glob

When you have a directory src and you want to move all the contents
to dst, this is surprisingly annoying. shutil.move doesn't operate
on globs. shutil.copytree does exactly what we need except it
requires the destination to not exist. In Python 3.7 it gained the
ability to not care about the destination existing but... we need
to handle the pre-Python 3.7 case.

Depends on D75698

Comment 13

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add/change support for running scripts to the moz.yaml schema r?glob

Here we unify the 'run_after' section with 'file-updates' just naming
them 'update-actions'. This will allow a simpler schema and a clearer
picture of the order of actions that are taken.

Depends on D75699

Comment 14

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add a delete-path action to the update-actions r?glob

Depends on D76428

Comment 15

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add support for github and googlesource as hosting repositories r?glob

Depends on D76429

Comment 16

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add a moz.yaml for libaom and remove version information from the README r?glob

Depends on D76430

Comment 17

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Allow ./mach command --help to print out the help for both sub-commands and command arguments r?glob

It is possible to have both a default command (with positional arguments) and
sub-commands (with arguments) in mach. If the subcommand exists, it
is dispatched to; if it doesn't the default command is called the positional
argument filled in.

However, when you run ./mach command --help, it will detect the subcommands
and only print out their help section. If the default command has arguments,
they were not printed out. Now they are.

Small papercuts in this patch are that the Default Command Arguments are
printed after the subcommands and that subcommand help without default
arguments have an extra newline after them. Both of these seem small
enough that the refactoring necessary to abate them is undesirable.

Depends on D76432

Comment 18

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Remove the old implementations of dav1d and aom vendoring and clean up the mach command r?glob

Depends on D76505

Comment 19

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Implement --check-for-update, controlling the output carefully for scripted use r?glob

Depends on D76506

Comment 20

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Clean up mozfile r?glob

Move all imports to the top, instead of inside functions,
then run it through black.

Depends on D76083

Comment 21

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Attachment: Bug 1637845 - Add debug logging to the Update Actions r?glob

Depends on D76614

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c72208359b4
Move all mach vendor files to a subdirectory r=glob
https://hg.mozilla.org/integration/autoland/rev/5410c7af6596
Apply 'black' to the vendor subdirectory r=glob
https://hg.mozilla.org/integration/autoland/rev/5c7bae6fcd36
Add 'license-file' to origin, remove 'revision' from vendoring, add 'vendor-directory' r=glob
https://hg.mozilla.org/integration/autoland/rev/3ad87515c7fd
Add source-hosting to the schema and vendoring section r=glob
https://hg.mozilla.org/integration/autoland/rev/287275c7275e
Add 'file-updates' to the moz.yaml schema r=glob
https://hg.mozilla.org/integration/autoland/rev/a55dd5ad2285
Extend the file-updates schema to support a copy-file action r)glob
https://hg.mozilla.org/integration/autoland/rev/3a08c2a3b302
Rename VerifyError to MozYamlVerifyError r=glob
https://hg.mozilla.org/integration/autoland/rev/b459114e8008
Add a new function 'copy_contents' to mozfile r=glob
https://hg.mozilla.org/integration/autoland/rev/47744e5c0829
Clean up mozfile r=gbrown
https://hg.mozilla.org/integration/autoland/rev/f495b58337dc
Re-implement mach vendor r=glob
https://hg.mozilla.org/integration/autoland/rev/98ae9c7d8dfb
Add/change support for running scripts to the moz.yaml schema r=glob
https://hg.mozilla.org/integration/autoland/rev/d22bc1280d8f
Add a delete-path action to the update-actions r=glob
https://hg.mozilla.org/integration/autoland/rev/5877126012cd
Add support for github and googlesource as hosting repositories r=glob
https://hg.mozilla.org/integration/autoland/rev/f100ac23b748
Add a moz.yaml for libaom and remove version information from the README r=glob
https://hg.mozilla.org/integration/autoland/rev/a15f6c29bbeb
Allow ./mach command --help to print out the help for both sub-commands and command arguments r=ahal
https://hg.mozilla.org/integration/autoland/rev/84ed48eb7d0c
Remove the old implementations of dav1d and aom vendoring and clean up the mach command r=glob
https://hg.mozilla.org/integration/autoland/rev/70bc5f4b56d5
Implement --check-for-update, controlling the output carefully for scripted use r=glob
https://hg.mozilla.org/integration/autoland/rev/6500c049e562
Add debug logging to the Update Actions r=glob CLOSED TREE

Comment 23

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/9c72208359b4
https://hg.mozilla.org/mozilla-central/rev/5410c7af6596
https://hg.mozilla.org/mozilla-central/rev/5c7bae6fcd36
https://hg.mozilla.org/mozilla-central/rev/3ad87515c7fd
https://hg.mozilla.org/mozilla-central/rev/287275c7275e
https://hg.mozilla.org/mozilla-central/rev/a55dd5ad2285
https://hg.mozilla.org/mozilla-central/rev/3a08c2a3b302
https://hg.mozilla.org/mozilla-central/rev/b459114e8008
https://hg.mozilla.org/mozilla-central/rev/47744e5c0829
https://hg.mozilla.org/mozilla-central/rev/f495b58337dc
https://hg.mozilla.org/mozilla-central/rev/98ae9c7d8dfb
https://hg.mozilla.org/mozilla-central/rev/d22bc1280d8f
https://hg.mozilla.org/mozilla-central/rev/5877126012cd
https://hg.mozilla.org/mozilla-central/rev/f100ac23b748
https://hg.mozilla.org/mozilla-central/rev/a15f6c29bbeb
https://hg.mozilla.org/mozilla-central/rev/84ed48eb7d0c
https://hg.mozilla.org/mozilla-central/rev/70bc5f4b56d5
https://hg.mozilla.org/mozilla-central/rev/6500c049e562

Comment 24

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

Re-assigning bugs to the new Mach Vendor and Updatebot Component. Guid for this change: ade7b229-ec85-41f8-bafe-9d4451bfeed1

Comment 25

[Tom Ritter [:tjr] (OOTO Oct 18-26)]

A mass updatebot change inadvertently added a security group to bugs. Removing it. Guid for this change: 09d63db4-11a1-4820-a398-d718f112f885