Last Comment Bug 629268 - Reset try server
: Reset try server
Product: Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All All
-- major (vote)
: ---
Assigned To: Aravind Gottipati [:aravind]
: matthew zeier [:mrz]
Depends on:
Blocks: 626751
  Show dependency treegraph
Reported: 2011-01-26 18:50 PST by Dave Townsend [:mossop]
Modified: 2015-03-12 08:17 PDT (History)
5 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Description User image Dave Townsend [:mossop] 2011-01-26 18:50:55 PST
This changeset failed every build due to timeouts while cloning:

This changeset looks to be about to do the same:

Same patch I know but should be no way for it to cause clone timeouts. I've also noticed lately that pushing to try is extremely slow.

I wonder if the try repo is just too big now and needs to be recloned from m-c again? I note that m-c has some 61k changesets in it, try has about 93k in comparison.
Comment 1 User image Nick Thomas [:nthomas] 2011-01-26 19:03:14 PST
IT, please check on the health of dm-vcview04. We're approaching 100% failure to clone try now, with the first problem showing up about 4:30 today.
Comment 2 User image Nick Thomas [:nthomas] 2011-01-26 19:09:22 PST
Non-try repos seem fine. Try failures are independent of slave hardware.

While resetting try has probably come round again I don't know the pre-requisites for doing so and we can't do it straight away. Besides, it would be strange if a relatively normal day's worth of pushes to try suddenly sent us over the edge.
Comment 3 User image Aravind Gottipati [:aravind] 2011-01-26 19:13:34 PST
Looks like there are about 12 try pulls in process right now.  They seem to be going okay.  This is a known issue that the try server can only handle so many try pulls at the same time.
Comment 4 User image Aravind Gottipati [:aravind] 2011-01-26 21:31:26 PST
21:21 < nthomas> saw three clones happen in 20mins on linux64 slaves
21:21 < nthomas> finishing a few mins ago
21:24 < nthomas> yeah, so morph the bug into 'reset try repo again' ?
Comment 5 User image Aravind Gottipati [:aravind] 2011-01-26 21:31:58 PST
@john:  Can you co-ordinate with folks and let us know when we can do this?
Comment 6 User image Nick Thomas [:nthomas] 2011-01-27 00:14:30 PST
Haven't seen any failures since the accidental restart of Apache when bkero was adjusting the config earlier. And we're now in the window where we'd be getting timeouts an hour after this pretty lot of pushes
Comment 7 User image John O'Duinn [:joduinn] (please use "needinfo?" flag) 2011-01-27 12:57:01 PST
(In reply to comment #5)
> @john:  Can you co-ordinate with folks and let us know when we can do this?

Happy to. 
1) How much of a downtime do you need for this?
2) this would only impact try repo, and other branches could remain open throughout?
Comment 8 User image John O'Duinn [:joduinn] (please use "needinfo?" flag) 2011-01-31 11:49:27 PST
Would be great if this could be done at same time as bug#614786. (both require tree closures).
Comment 9 User image Aravind Gottipati [:aravind] 2011-02-03 08:41:23 PST

Note You need to log in before you can comment on or make changes to this bug.