1442306 - create json file during build process that can be consumed by buildhub

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Kim Moir [:kmoir] ET]

I had a meeting with bwong, catlee, leplatrem, peterbe today regarding providing a json file as part of the build process that can be consumed by buildhub.  The current schema is here 

https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/initialization.yml#L71-L198

The request is that a json file be created as part of the build process for nightly, beta, release, rc, devedition builds.  

Other notes
* locale vs. locales in that bug.
* Generally we something rather than perfect all-things-possible in the schema.
* Releng will populate the json file with data from later tasks in the release graph

Comment 1

[(not currently active) Ted Mielczarek]

For reference, we already create several informational files during packaging, most relevant to this bug is probably the buildinfo file:
https://dxr.mozilla.org/mozilla-central/rev/b996cabc7ef54bbe050d647494bf00d668ec52e6/toolkit/mozapps/installer/packager.mk#122

which is a JSON file generated by informulate.py:
https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/informulate.py

For an example, the Firefox 52 Win32 en-US build file looks like:
https://ftp.mozilla.org/pub/firefox/candidates/52.0-candidates/build2/win32/en-US/firefox-52.0.json

...which is already pretty close to that schema.

Comment 2

[(not currently active) Ted Mielczarek]

If I were bikeshedding that schema, I would probably want to include the compiler versions, and the version of rustc as well. about:buildconfig has the former but not the latter.

Comment 3

[Peter Bengtsson [:peterbe]]

Thank you Kim. That's a good start. 
And Thank you Ted for getting more concrete details into the discussion.

Our goal is to write a new .json file and do it slightly differently:

* We want to validate the writing with a JSON Schema. (i.e. bad build data should break things early)
* We want a different name of the .json file. Leaning towards calling it buildhub.json

It's not clear, correct me if I'm wrong Kim, but we assume my team will actually write the code. I.e. me. Is that right? Or is there someone more suited who can do this? For what it's worth, I can write a piece of Python code that makes, or raises an error, the .json file. I guess, similar to that informulate.py but slightly different. But TaskCluster stuff in mozilla-central is still something where experienced hands would be much appreciated. 

Please confirm your understanding about who should write the code.
Does it make sense to write a script that if it fails on writing the JSON it will "break the build"?


(Ted, unless you've already figured it out, the objective with all of this is to ultimately stop scraping like Socorro scrapes archive.mozilla.org. We are also interested in making this new .json file end up in our database (Buildhub) but that process is best thought about once this new .json file exists.)

Comment 4

[(not currently active) Ted Mielczarek]

I think myself or one of the other build peers could easily write the generation for this. Note that I wasn't suggesting that we should reuse the existing build info JSON, just that that'd be a good place to start because we're already doing 90% of what we need there. We also already have voluptuous in the tree (it's used in Taskcluster task graph generation), so we could define the schema with that and then it should be pretty simple to ensure we're generating JSON that fits the schema.

And yeah, this is great and this is something I've wanted for *years*!

Comment 5

[Kim Moir [:kmoir] ET]

Thank you Ted for all the information on this bug, it's very helpful!

:peterbe 

So I'm a bit confused because looking at the meeting minutes they state

What data do we have and how do we output it? 
* Kim’s team should own this part for building out meta data for nightly/beta builds
* Release metadata = releng

So my understanding from the meeting is that my team are going to write code to output the new json file once we have agreed on the schema.  The plan was to output this json as part of the build process for nightly builds.  Is this your understanding or am I misunderstanding?  Happy to discuss on irc/vidyo if this is easier

Comment 6

[Peter Bengtsson [:peterbe]]

Kim, 
The root of the confusion was as follows; after our vidyo meeting I wrote my summary and the bugzilla comment above. Then I had my 1:1 with Benson who had just previously had a 1:1 with catlee, and through Chinese whispers I was told that the work probably will land on me. 
In the midst of that, Ted chimed in. In particular comment #4. 

Ideally I don't want to write the code because the domain is not my best. Python scripting, sure, yes, I can do that. 
My ultimate goal and current top priority is that there is a new .json file Buildhub's tooling can consume (instead of hotch-potch digging through uploaded files trying to shoehorn that into Buildhubs schema). 

Ted, Kim,
Would you mind if the 3 of us form a little team to iron out the who's going to do the work? And actually do it :) I'd be happy to "project manage" this.

Comment 7

[Kim Moir [:kmoir] ET]

Thanks for the explanation Peter, it all makes sense now.  

Sure, it would be good to figure out who is going to do the work as a team.

Ted, are you able to mentor Peter on this work or would you like to defer to another build peer?  If not, I can ask the rest of the team in the tomorrow's meeting and see who able to help.

Comment 8

[(not currently active) Ted Mielczarek]

I'd be happy to mentor Peter on this, but it might just be easier to get a build peer to do the work since most of the hassle will probably be dealing with build system nonsense.

Comment 9

[Peter Bengtsson [:peterbe]]

Some meeting notes here: https://docs.google.com/document/d/1HiHmVazeGjnif2vQfsmx0O38hn_4wBn5XdNPKPudvz8/edit#

Ted, 
The JSON Schema is as per comment 0 here in this bug. 
Permalink https://github.com/mozilla-services/buildhub/blob/53518a873b45daf497347ac71770859ec6ce2b8a/jobs/buildhub/initialization.yml#L71-L198
The scraping is currently done here: https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/inventory_to_records.py 
It's a more (async) modern version of https://github.com/mozilla-services/socorro/blob/master/socorro/cron/jobs/ftpscraper.py but unlike Socorro makes one entry per locale.

Mathieu, CC'ed on this bug, knows a lot more details about how the "scraping" currently work if you have more detailed questions about how the buildhub does/did things. 

Again, thank you Ted!

Comment 10

[(not currently active) Ted Mielczarek]

Peter and Kim and I chatted about this, a few salient points:
* I'm going to write the build system bits to generate the JSON file. It should not be much work for someone who knows the build system.
* We're going to call the file "buildhub.json" to make its purpose explicit.
* The JSON schema includes three keys that the build system can provide (build, target, source), and one that it cannot (download). The latter will need to be added by the release pipeline as Kim mentioned in comment 0, so we should file a bug for that work. I'd like to have a schema in-tree that we validate the data against, but if we're not going to fill in the `download` bit we'll need to modify the schema in some way so we can validate just the subset we provide.

Comment 11

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> For an example, the Firefox 52 Win32 en-US build file looks like:
> https://ftp.mozilla.org/pub/firefox/candidates/52.0-candidates/build2/win32/
> en-US/firefox-52.0.json
> 
> ...which is already pretty close to that schema.

(In reply to Peter Bengtsson [:peterbe] from comment #9)
> The scraping is currently done here:
> https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/
> inventory_to_records.py 

Looking at this code, it attempts to scrape info from this very JSON file, falling back to the older .txt files if it's not available, which makes sense:
https://github.com/mozilla-services/buildhub/blob/72b919a0bb0ebed1258d3446401dbdbbebc4d5cf/jobs/buildhub/inventory_to_records.py#L152

Comment 12

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=246887a74bdbad7473c0ba486bf33cd292867a38

Comment 13

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=13e00b16c6ea0e3d1d69d473eb4a0e3c60f7420b

Comment 14

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #12)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=246887a74bdbad7473c0ba486bf33cd292867a38

I realized immediately after pushing that I forgot to add buildhub.json to UPLOAD_FILES so it wasn't going to wind up in the build artifacts... The second try push corrects this.

These patches produce a sensible looking buildhub.json that conforms to the schema when run on my machine (with a few extra things set in the environment, since the patches as written assume a CI environment);
https://gist.github.com/luser/f1f2bd684f083da46ef6cc8ad4fd38b7

Comment 15

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b158b73ce4ffc541ee92f02f0bb8c1940d3cfba1

Comment 16

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8239af15a3c9d57d26e5076e1f827ca070b064f0

Comment 17

[(not currently active) Ted Mielczarek]

Attachment: bug 1442306 - simplify make-buildinfo-file by letting it assume Mozilla CI.

The `make-buildinfo-file` target is called as part of `make package` to
generate a json file and a text file with some info about the build. Because
a few of those things like the Build ID and source repo / changeset are not
guaranteed to be set at configure time it has to pass them down from the make
environment.

In CI all of these things are passed in the environment, populated at task
graph generation time, so doing this work only in automation builds lets
us simplify things a bit.

Review commit: https://reviewboard.mozilla.org/r/240990/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240990/

Comment 18

[(not currently active) Ted Mielczarek]

Attachment: bug 1442306 - move build info.txt generation into informulate.py.

This script is already doing most of the work, so make it write the text
file instead of using shell commands.

Review commit: https://reviewboard.mozilla.org/r/240992/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240992/

Comment 19

[(not currently active) Ted Mielczarek]

Attachment: bug 1442306 - move make-buildinfo-file and make-mozinfo-file targets inline into make-package target.

Currently we have a mess of targets in packager.mk which doesn't really help
much. This patch removes the make-buildinfo-file and make-mozinfo-file targets
and just puts their commands inline in the make-package target.

Review commit: https://reviewboard.mozilla.org/r/240994/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240994/

Comment 20

[(not currently active) Ted Mielczarek]

Attachment: bug 1442306 - set MOZ_SOURCE_CHANGESET and MH_BRANCH consistently in the environment for mozharness tasks.

We currently set GECKO_HEAD_REV to the revision being built, but the build
system already looks for MOZ_SOURCE_CHANGESET so set that as well. Additionally,
set MH_BRANCH for mozharness tasks on generic-worker to match docker-worker.

Review commit: https://reviewboard.mozilla.org/r/240996/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240996/

Comment 21

[(not currently active) Ted Mielczarek]

Attachment: bug 1442306 - generate a buildhub.json file during packaging.

buildhub is a service that stores a list of nightly and release builds and
can be queried to find specific builds. Currently it ingests data by scraping
info from ftp.mozilla.org. This patch makes it so we generate a buildhub.json
during packaging with the data that buildhub wants.

There are a few pieces of data that we can't accurately provide from the
build system such as the URL to the build, so we provide some stub data
there with the expectation that a release engineering process will fill
them in later.

Review commit: https://reviewboard.mozilla.org/r/240998/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240998/

Comment 22

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8a81f06ca93dc2bcc029e86af2183c29ba70ecc

Comment 23

[(not currently active) Ted Mielczarek]

I tried to avoid too much yak shaving with these patches (there's so many tempting things in packager.mk!). I did consolidate a few things so that one Python script is writing out several files with similar content, and I removed some unnecessary makefile targets.

One thing I didn't do in this patch series is import the JSON schema from buildhub and validate the data as we produce it. That should not be hard, I will just need to vendor a JSON schema library for Python.

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8972376 [details]
bug 1442306 - simplify make-buildinfo-file by letting it assume Mozilla CI.

https://reviewboard.mozilla.org/r/240990/#review247564

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8972377 [details]
bug 1442306 - move build info.txt generation into informulate.py.

https://reviewboard.mozilla.org/r/240992/#review247568

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8972378 [details]
bug 1442306 - move make-buildinfo-file and make-mozinfo-file targets inline into make-package target.

https://reviewboard.mozilla.org/r/240994/#review247570

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8972379 [details]
bug 1442306 - set MOZ_SOURCE_CHANGESET and MH_BRANCH consistently in the environment for mozharness tasks.

https://reviewboard.mozilla.org/r/240996/#review247572

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8972380 [details]
bug 1442306 - generate a buildhub.json file during packaging.

https://reviewboard.mozilla.org/r/240998/#review247576

::: toolkit/mozapps/installer/upload-files.mk:393
(Diff revision 1)
>    $(call QUOTED_WILDCARD,$(DIST)/$(PKG_PATH)$(MOZHARNESS_PACKAGE)) \
>    $(call QUOTED_WILDCARD,$(DIST)/$(PKG_PATH)$(SYMBOL_ARCHIVE_BASENAME).zip) \
>    $(call QUOTED_WILDCARD,$(DIST)/$(PKG_PATH)$(GENERATED_SOURCE_FILE_PACKAGE)) \
>    $(call QUOTED_WILDCARD,$(MOZ_SOURCESTAMP_FILE)) \
>    $(call QUOTED_WILDCARD,$(MOZ_BUILDINFO_FILE)) \
> +  $(MOZ_BUILDHUB_JSON) \

It's a bit weird that this isn't using QUOTED_WILDCARD like everything else. I can't recall why we use QUOTED_WILDCARD. I wouldn't be surprised if it were a cargo cult. If it works, it works and I'm fine being inconsistent.

Comment 29

[(not currently active) Ted Mielczarek]

Comment on attachment 8972380 [details]
bug 1442306 - generate a buildhub.json file during packaging.

https://reviewboard.mozilla.org/r/240998/#review247576

> It's a bit weird that this isn't using QUOTED_WILDCARD like everything else. I can't recall why we use QUOTED_WILDCARD. I wouldn't be surprised if it were a cargo cult. If it works, it works and I'm fine being inconsistent.

I'm going to put this back only because I triggered l10n repack jobs on my try push and they're broken without it here, because they're not generating a buildhub.json file so it doesn't exist to upload. I will fix them, but that's probably going to be enough work that I want to get these patches and do it in another bug.

IIRC this is effectively just working as `$(wildcard)`, in that it lets us list everything we could possibly upload but only upload files that are actually generated. We couldn't use `$(wildcard)` at the time because we still supported pretty package names which had spaces, which GNU make can't handle properly. (We've since dropped support for pretty package names, so maybe we could just make these `$(wildcard)` at this point, but I'd like to just remove this whole list at some point in the future anyway.)

Comment 30

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/db2d6e2667d268ff2be84818fbec1fe67a1a7db3
bug 1442306 - simplify make-buildinfo-file by letting it assume Mozilla CI. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/3a2c60a752cc58035f7982225e3efdf374c76ff1
bug 1442306 - move build info.txt generation into informulate.py. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/604aa7d37098eb9ee89d7a6c670dd72c31d35116
bug 1442306 - move make-buildinfo-file and make-mozinfo-file targets inline into make-package target. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/63abddfab15f0effa13c6019b0889dc58cdb33b6
bug 1442306 - set MOZ_SOURCE_CHANGESET and MH_BRANCH consistently in the environment for mozharness tasks. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/7507516552fe76d0d00d32917d88a0056e731fb5
bug 1442306 - generate a buildhub.json file during packaging. r=gps

Comment 31

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/db2d6e2667d2
https://hg.mozilla.org/mozilla-central/rev/3a2c60a752cc
https://hg.mozilla.org/mozilla-central/rev/604aa7d37098
https://hg.mozilla.org/mozilla-central/rev/63abddfab15f
https://hg.mozilla.org/mozilla-central/rev/7507516552fe