676420 - please clone and prune try

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Aki Sasaki (not active)]

Try is slow, and at >4000 heads.
We need to prune those heads or re-clone try, but John O'Duinn needs the pushloghtml stats from existing try before we do that.

I think the path to success involves

a) moving or cloning existing try to a temporary location, until John O'Duinn can gather stats, and then nuke the site from orbit.
b) pruning the heads from hg.m.o/try, or recloning try.

Adding bug 554656 and bug 633161 as "depends on" bugs, even though they don't strictly block; more for reference.

Comment 1

[(not currently active) Ted Mielczarek]

The point of the work in bug 633161 was to be able to strip old heads without having to completely reset the repo. All indications were that it seemed to work, I think we should continue with that instead of blowing away the entire repo, since it can be done without any impact on pushing to try.

Comment 2

[Dumitru Gherman [:dumitru]]

I can follow the path to success described by Aki by
a) cloning

Comment 3

[Dumitru Gherman [:dumitru]]

Sorry for the last comment, I hit enter to save changes accidentally.
According to  https://wiki.mozilla.org/ReleaseEngineering:ResetTryServer a TreeClosure is needed. CC'ing arr to confirm and let me know when can we do this.

Also, is a) necessary or can I jump to b) directly?

Comment 4

[Aki Sasaki (not active)]

a) is necessary to get stats from Try; otherwise we have to block on John O'Duinn's schedule, which will push this out several days.

bug 633161 has a script (https://bug633161.bugzilla.mozilla.org/attachment.cgi?id=531989) that Ted says will work on an open Try tree.
https://bugzilla.mozilla.org/show_bug.cgi?id=633161#c22 has details on how to use it.

I'll negotiate with the sheriff for a soft downtime just in case.

Comment 5

[Shyam Mani [:fox2mike]]

a) is done, we have a backup of the db.

Ted's script will run into issues with try + pushlog hook enabled..he ran into that while testing, so we're going to have to reclone try, I guess.

Comment 6

[Shyam Mani [:fox2mike]]

I think this was supposed to happen on Monday, Aug 8th at 0600 PDT. Not sure if notices were sent. CC'ing catlee who's supposed to be on buildduty on Monday.

Comment 7

[Aki Sasaki (not active)]

Notice sent.
Not sure if catlee is aware of this yet; if it's approaching 6am PDT Monday and he's not around, feel free to call my cell (in phonebook) =\

Comment 8

[Shyam Mani [:fox2mike]]

Sweet. Following steps from the wiki :

[hg@dm-svn01 mozilla]$ time hg clone -U mozilla-central try

real    7m24.144s
user    0m3.694s
sys     0m7.957s

[hg@dm-svn01 mozilla]$ cp /repo/hg/mozilla/mozilla-central/.hg/pushlog2.db /repo/hg/mozilla/try/.hg/pushlog2.db 
[hg@dm-svn01 mozilla]$ chown hg:scm_level_1 /repo/hg/mozilla/try/.hg/pushlog2.db 
[hg@dm-svn01 mozilla]$ chmod g+w /repo/hg/mozilla/try/.hg/pushlog2.db
[hg@dm-svn01 mozilla]$ cd /repo/hg/mozilla/try/.hg/ 
[hg@dm-svn01 .hg]$ sqlite3 pushlog2.db 
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> delete from changesets;
sqlite> delete from pushlog; 
sqlite> .exit

[root@dm-svn01 mozilla]# chown -R hg:scm_level_1 try
[root@dm-svn01 mozilla]#

And the copy on the varnish server has been rebuilt as well. 

This should be good. I have a copy of the pushlogdb and can put it up someplace when John needs it.