Upload tup database as an artifact

RESOLVED FIXED in Firefox 62

Status

enhancement
RESOLVED FIXED
Last year
7 months ago

People

(Reporter: mshal, Assigned: scarrillo)

Tracking

(Blocks 1 bug)

3 Branch
mozilla62
Dependency tree / graph

Firefox Tracking Flags

(firefox62 fixed)

Details

Attachments

(1 attachment)

We should upload the tup database (the file $topsrcdir/.tup/db) as an artifact after a tup build so we can analyze it for bug 1464231. Note the database is large (~1GB), so we may need to compress it, and/or only upload it under certain conditions (eg: perhaps a separate nightly tup build to get a snapshot of the database once per day rather than per push).
Assignee: nobody → scarrillo
I think we'll want to add this as a flag to do on-demand on a try push to start, and worry about a periodic tup-db-nightly build later.

We had some discussion off-bug about how to get the tup build to upload since it doesn't run automation/build at all. After looking into this a bit today it looks like we should be able to copy the tup db to the UPLOAD_DIR set in the build's environment as a one-off.

There isn't a really great place to do this right now, but we could just put it in this method for now: https://searchfox.org/mozilla-central/rev/292d295d6b084b43b70de26a42e68513bb7b36a3/python/mozbuild/mozbuild/backend/tup.py#268 after invoking the build (if it succeeds).
Made some changes and compressed the db but...

Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file (tup_db.zip): 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2

So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which is now somewhat of a misnomer):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64

Will that be ok for now?
(In reply to Sofia Carrillo from comment #2)
> Made some changes and compressed the db but...
> 
> Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file
> (tup_db.zip): 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2
> 
> So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which
> is now somewhat of a misnomer):
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64
> 
> Will that be ok for now?

Yep, that should be fine. The only other files in the .tup directory after the tup process exits are some empty lock files, so zipping up the whole thing won't be much larger than just the database itself. This approach might be better anyway in case we add some debugging info to tup, which would likely also be written out in the same directory. So if/when that happens, we'll already be able to grab them from this artifact.
Attachment #8984325 - Flags: review?(mshal)
Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

Sorry for the spam, difference between the two mozreview requests was just whitespace.
Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

https://reviewboard.mozilla.org/r/250052/#review256600

Looks great!
Attachment #8984325 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/c77f7fe4c21b
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.