Closed Bug 1464235 Opened 2 years ago Closed 2 years ago
Upload tup database as an artifact
We should upload the tup database (the file $topsrcdir/.tup/db) as an artifact after a tup build so we can analyze it for bug 1464231. Note the database is large (~1GB), so we may need to compress it, and/or only upload it under certain conditions (eg: perhaps a separate nightly tup build to get a snapshot of the database once per day rather than per push).
I think we'll want to add this as a flag to do on-demand on a try push to start, and worry about a periodic tup-db-nightly build later. We had some discussion off-bug about how to get the tup build to upload since it doesn't run automation/build at all. After looking into this a bit today it looks like we should be able to copy the tup db to the UPLOAD_DIR set in the build's environment as a one-off. There isn't a really great place to do this right now, but we could just put it in this method for now: https://searchfox.org/mozilla-central/rev/292d295d6b084b43b70de26a42e68513bb7b36a3/python/mozbuild/mozbuild/backend/tup.py#268 after invoking the build (if it succeeds).
Made some changes and compressed the db but... Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file (tup_db.zip): https://treeherder.mozilla.org/#/jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2 So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which is now somewhat of a misnomer): https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64 Will that be ok for now?
(In reply to Sofia Carrillo from comment #2) > Made some changes and compressed the db but... > > Trying to compress $topsrcdir/.tup/db resulted in an empty compressed file > (tup_db.zip): > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=17284c9a4c7eec7e70b47e00351549c8685a04d2 > > So then I just compressed the entirety of $topsrcdir/.tup (tup_db.zip, which > is now somewhat of a misnomer): > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=7e44540402631e85a4f8418b0dfc41adacd64e64 > > Will that be ok for now? Yep, that should be fine. The only other files in the .tup directory after the tup process exits are some empty lock files, so zipping up the whole thing won't be much larger than just the database itself. This approach might be better anyway in case we add some debugging info to tup, which would likely also be written out in the same directory. So if/when that happens, we'll already be able to grab them from this artifact.
Comment on attachment 8984325 [details] Bug 1464235 - option to upload tup db Sorry for the spam, difference between the two mozreview requests was just whitespace.
Comment on attachment 8984325 [details] Bug 1464235 - option to upload tup db https://reviewboard.mozilla.org/r/250052/#review256600 Looks great!
Attachment #8984325 - Flags: review?(mshal) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/c77f7fe4c21b option to upload tup db r=mshal
You need to log in before you can comment on or make changes to this bug.