1253436 - Write a machine readable file containing moz.build derived binaries metadata

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

In bug 1253110 we'll be producing archives with binaries files. Instead of hard-coding a list of binaries (like we do with artifact builds today), let's derive the list of binaries from moz.build.

The first step is to produce a machine readable file containing binaries metadata.

Comment 1

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1253436 - Remove unused imports; r=glandium

Review commit: https://reviewboard.mozilla.org/r/37993/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37993/

Comment 2

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium

Now when we print instances in a debugger, we'll see something like:

<StaticLibrary: other-licenses/snappy/libother-licenses_snappy.a>
<StaticLibrary: toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a>
<SharedLibrary: toolkit/library/XUL>
<StaticLibrary: toolkit/library/libxul_s.a>
<HostProgram: config/nsinstall_real>
<Program: memory/replace/logalloc/replay/logalloc-replay>
<SimpleProgram: mfbt/tests/TestArrayUtils>

... instead of the the class name and memory address.

Review commit: https://reviewboard.mozilla.org/r/37995/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37995/

Comment 3

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium

This will make it easier for binaries only archive generation to consult
the list of binaries that are relevant to packaging.

This does overlap somewhat with compile databases. Perhaps someday the
logic could converge.

While I was here, I cleaned up writing of the all-tests.json file to
avoid an intermediate string variable.

Review commit: https://reviewboard.mozilla.org/r/37997/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37997/

Comment 4

[Gregory Szorc [:gps]]

It just occurred to me that we could potentially use this code for updating the dist/bin install manifest. We could also potentially write out make rules to create binaries in their final location under dist/bin. That is scope bloat, however.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8726448 [details]
MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37997/diff/1-2/

Comment 6

[Mike Hommey [:glandium]]

Comment on attachment 8726446 [details]
MozReview Request: Bug 1253436 - Remove unused imports; r=glandium

https://reviewboard.mozilla.org/r/37993/#review35273

Comment 7

[Mike Hommey [:glandium]]

Comment on attachment 8726447 [details]
MozReview Request: Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium

https://reviewboard.mozilla.org/r/37995/#review35275

Comment 8

[Mike Hommey [:glandium]]

Comment on attachment 8726448 [details]
MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium

https://reviewboard.mozilla.org/r/37997/#review35277

::: python/mozbuild/mozbuild/backend/common.py:204
(Diff revision 2)
> +    def to_dict(self):
> +        SHARED_ATTRS = ('basename', 'lib_name', 'import_name', 'soname',
> +                        'install_target', 'relobjdir')
> +        shared = []
> +        for l in self.shared_libraries:
> +            shared.append({k: getattr(l, k) for k in SHARED_ATTRS})
> +
> +        PROGRAM_ATTRS = ('program', 'KIND', 'install_target', 'relobjdir')
> +        programs = []
> +        for p in self.programs:
> +            programs.append({k.lower(): getattr(p, k) for k in PROGRAM_ATTRS})
> +
> +        return {
> +            'shared_libraries': shared,
> +            'programs': programs,
> +        }

I'd rather use a JSONEncoder subclass, that could be repurposed later.

Comment 9

[(not currently active) Ted Mielczarek]

I'd love to figure out a more general solution here to capture the parsed moz.build data for use in scripts/mach commands/etc instead of continuing to accumulate one-off json files. That's not to say we shouldn't land this patch, since it's useful, but it would be great to figure out a better long-term plan.

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8726446 [details]
MozReview Request: Bug 1253436 - Remove unused imports; r=glandium

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37993/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8726447 [details]
MozReview Request: Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37995/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8726448 [details]
MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37997/diff/2-3/

Comment 13

[Mike Hommey [:glandium]]

Comment on attachment 8726448 [details]
MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium

https://reviewboard.mozilla.org/r/37997/#review36461

> In the spirit of perfect is the enemy of done, I think this should be deferred to a follow-up bug.

Please file that followup bug, otherwise, it's never going to happen.

::: python/mozbuild/mozbuild/frontend/data.py:379
(Diff revision 3)
> +    def to_dict(self):
> +        return {k: getattr(self, k) for k in self.DICT_ATTRS}

You might as well move this method to the base class. It will fail on all classes with no DICT_ATTRS, which is fine.

Comment 14

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c4a868c76959d3a2e13af6ca5adb0d629dcb2b34
Bug 1253436 - Remove unused imports; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/84849ad026c9ba1bbf71c93172b0a03440e51bec
Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/4167dfdf10457360c9c94ee6e55b03ef1b92c16d
Bug 1253436 - Write out a machine readable file with binaries metadata; r=glandium

Comment 15

[Phil Ringnalda (:philor)]

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/04a9056213bd for https://treeherder.mozilla.org/logviewer.html#?job_id=23744325&repo=mozilla-inbound.

Comment 16

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f1b77ed0c0c4a5d40d4ba854db9ae506d906b070
Bug 1253436 - Remove unused imports; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/e1533121f2e3adbe89e4a799c714d065a44ee5c3
Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/bd4a4166af52256da8e99d90668bb40a82801694
Bug 1253436 - Write out a machine readable file with binaries metadata; r=glandium

Comment 17

[Gregory Szorc [:gps]]

Fixed the case of 'kind' vs 'KIND' to unbust it.

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f1b77ed0c0c4
https://hg.mozilla.org/mozilla-central/rev/e1533121f2e3
https://hg.mozilla.org/mozilla-central/rev/bd4a4166af52