1464235 - Upload tup database as an artifact

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Michael Shal [:mshal]]

We should upload the tup database (the file $topsrcdir/.tup/db) as an artifact after a tup build so we can analyze it for bug 1464231. Note the database is large (~1GB), so we may need to compress it, and/or only upload it under certain conditions (eg: perhaps a separate nightly tup build to get a snapshot of the database once per day rather than per push).

Comment 1

[Chris Manchester (limited bugmail, email directly)]

I think we'll want to add this as a flag to do on-demand on a try push to start, and worry about a periodic tup-db-nightly build later.

We had some discussion off-bug about how to get the tup build to upload since it doesn't run automation/build at all. After looking into this a bit today it looks like we should be able to copy the tup db to the UPLOAD_DIR set in the build's environment as a one-off.

There isn't a really great place to do this right now, but we could just put it in this method for now: https://searchfox.org/mozilla-central/rev/292d295d6b084b43b70de26a42e68513bb7b36a3/python/mozbuild/mozbuild/backend/tup.py#268 after invoking the build (if it succeeds).

Comment 2

[Sofia Carrillo]

Made some changes and compressed the db but...

Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file (tup_db.zip): 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2

So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which is now somewhat of a misnomer):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64

Will that be ok for now?

Comment 3

[Michael Shal [:mshal]]

(In reply to Sofia Carrillo from comment #2)
> Made some changes and compressed the db but...
> 
> Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file
> (tup_db.zip): 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2
> 
> So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which
> is now somewhat of a misnomer):
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64
> 
> Will that be ok for now?

Yep, that should be fine. The only other files in the .tup directory after the tup process exits are some empty lock files, so zipping up the whole thing won't be much larger than just the database itself. This approach might be better anyway in case we add some debugging info to tup, which would likely also be written out in the same directory. So if/when that happens, we'll already be able to grab them from this artifact.

Comment 4

[Sofia Carrillo]

Attachment: Bug 1464235 - option to upload tup db

Review commit: https://reviewboard.mozilla.org/r/250052/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/250052/

Comment 5

[Sofia Carrillo]

Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/250052/diff/1-2/

Comment 6

[Sofia Carrillo]

Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

Sorry for the spam, difference between the two mozreview requests was just whitespace.

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

https://reviewboard.mozilla.org/r/250052/#review256600

Looks great!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c77f7fe4c21b
option to upload tup db r=mshal

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c77f7fe4c21b