Closed Bug 1464235 Opened 2 years ago Closed 2 years ago

Upload tup database as an artifact

Categories

(Firefox Build System :: General, enhancement)

3 Branch
enhancement
Not set
normal

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: mshal, Assigned: scarrillo)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

We should upload the tup database (the file $topsrcdir/.tup/db) as an artifact after a tup build so we can analyze it for bug 1464231. Note the database is large (~1GB), so we may need to compress it, and/or only upload it under certain conditions (eg: perhaps a separate nightly tup build to get a snapshot of the database once per day rather than per push).
Assignee: nobody → scarrillo
I think we'll want to add this as a flag to do on-demand on a try push to start, and worry about a periodic tup-db-nightly build later.

We had some discussion off-bug about how to get the tup build to upload since it doesn't run automation/build at all. After looking into this a bit today it looks like we should be able to copy the tup db to the UPLOAD_DIR set in the build's environment as a one-off.

There isn't a really great place to do this right now, but we could just put it in this method for now: https://searchfox.org/mozilla-central/rev/292d295d6b084b43b70de26a42e68513bb7b36a3/python/mozbuild/mozbuild/backend/tup.py#268 after invoking the build (if it succeeds).
Made some changes and compressed the db but...

Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file (tup_db.zip): 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2

So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which is now somewhat of a misnomer):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64

Will that be ok for now?
(In reply to Sofia Carrillo from comment #2)
> Made some changes and compressed the db but...
> 
> Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file
> (tup_db.zip): 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2
> 
> So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which
> is now somewhat of a misnomer):
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64
> 
> Will that be ok for now?

Yep, that should be fine. The only other files in the .tup directory after the tup process exits are some empty lock files, so zipping up the whole thing won't be much larger than just the database itself. This approach might be better anyway in case we add some debugging info to tup, which would likely also be written out in the same directory. So if/when that happens, we'll already be able to grab them from this artifact.
Attachment #8984325 - Flags: review?(mshal)
Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

Sorry for the spam, difference between the two mozreview requests was just whitespace.
Comment on attachment 8984325 [details]
Bug 1464235 - option to upload tup db

https://reviewboard.mozilla.org/r/250052/#review256600

Looks great!
Attachment #8984325 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/c77f7fe4c21b
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Depends on: 1468688
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.