836626 - Cannot clone try

Status: RESOLVED WONTFIX

(Developer Services :: General, task)

Description

[(no longer active)]

$ time hg clone http://hg.mozilla.org/try
destination directory: try
requesting all changes
abort: HTTP Error 413: Request Entity Too Large

real	0m17.116s
user	0m0.148s
sys	0m0.044s

Comment 1

[Ben Kero [:bkero]]

What version of hg client are you using to clone?

Comment 2

[(no longer active)]

(In reply to comment #1)
> What version of hg client are you using to clone?

2.3.1 on mac, and whatever we have installed on people.

Comment 3

[Corey Shields [:cshields]]

verified to be reproducible with older hg (1.5) as well, so this is not an HTTP header size issue.

requests to clone try end up producing a 400 Bad Request error.  Something is dorked up with try.

Comment 4

[Ben Kero [:bkero]]

We're suspecting this is being caused by the try repository having too many heads. This repository is occasionally stripped of old heads, which should be our next step in debugging this. I have seen no errors on any of the web heads.

Alternatively we could simply reset try to the tip of mozilla-central.

Comment 5

[Ben Kero [:bkero]]

[[e-mail]3 try]# hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
133468 files, 190587 changesets, 859592 total revisions

The repository does not exhibit any corruption.

Comment 6

[Justin Wood (:Callek)]

(In reply to :Ehsan Akhgari (Away 2/7-2/15) from comment #0)
> $ time hg clone http://hg.mozilla.org/try

So I question why you are cloning try, especially over http. I don't recall that ever being a supported use-case in my entire memory of try.
[ For reference, supported use case is you clone from a m-c-like tree, write a patch and push said tree+patch to try with -f to force the new head if needed ]

That said, we have now verified:
* The usual "pushing to try" use case works, and is still snappy. This is unlike it had been in the past many times, when we used to  reset try for slowness in pushing. We verified this by having philor, cshields and myself all push to try successfully.
* Clones still work (although abysmally slow) if you clone over ssh. While we never support cloning try like this, if you want to clone specific head for some reason, please try "-r <revision>" which should work faster, and may work over http.
* All other hg repos are verified to be working normally.


Other info gathered during investigation, noting for the record:
* The "close old heads" script we have (in Bug 734225) takes forever and a day, and last few times we tried it, caused other, more impactful, problems.
* Last (known) reset was Bug 778062, in July. And no issues between then and now!
* Currently 17,319 heads on /try
* you can read http://bz.selenic.com/show_bug.cgi?id=3557 as to why it won't clone (tl;dr - too many heads)
* Releng has the doc on 'what to do if we fully reset try' at https://wiki.mozilla.org/ReleaseEngineering/How_To/Reset_the_Try_Server [hat tip to aki for finding it faster than I could]

Thanks to aki, bkero, cshields, joduinn, philor for the help.

My personal recomendation is WONTFIX given the "why would you want to clone from try like this, nevermind over http, anyway!" but I'm open to learning a legit reason for your needs.

Comment 7

[(no longer active)]

(In reply to comment #6)
> (In reply to :Ehsan Akhgari (Away 2/7-2/15) from comment #0)
> > $ time hg clone http://hg.mozilla.org/try
> 
> So I question why you are cloning try, especially over http. I don't recall
> that ever being a supported use-case in my entire memory of try.
> [ For reference, supported use case is you clone from a m-c-like tree, write a
> patch and push said tree+patch to try with -f to force the new head if needed ]

Not sure what you mean.  I've had try clones many many different times.  Sometimes you need to find an older try job that you have pushed and the only way to do that is by looking for changesets that you've committed to the try repo.

> That said, we have now verified:
> * The usual "pushing to try" use case works, and is still snappy. This is
> unlike it had been in the past many times, when we used to  reset try for
> slowness in pushing. We verified this by having philor, cshields and myself all
> push to try successfully.

Yeah, I have no problems with pushing to try.

> * Clones still work (although abysmally slow) if you clone over ssh. While we
> never support cloning try like this, if you want to clone specific head for
> some reason, please try "-r <revision>" which should work faster, and may work
> over http.

I initiated a clone from ssh, waited for ~half an hour and cancelled it and gave up.  I don't think cloning over ssh is realistic.  Plus, this is the first time I'm hearing anybody suggesting that cloning trees is not "supported".  :-)

Cloning from a revision doesn't help when you're looking *for* that revision.  Also, IIRC hg clone -r is generally quite slower than hg clone, because it needs to figure out which changesets you need and pull them one by one, or something crazy like that.

> * All other hg repos are verified to be working normally.
> 
> Other info gathered during investigation, noting for the record:
> * The "close old heads" script we have (in Bug 734225) takes forever and a day,
> and last few times we tried it, caused other, more impactful, problems.
> * Last (known) reset was Bug 778062, in July. And no issues between then and
> now!
> * Currently 17,319 heads on /try
> * you can read http://bz.selenic.com/show_bug.cgi?id=3557 as to why it won't
> clone (tl;dr - too many heads)

Hmm, no, that bug is about the hgweb front-end not working.  hgweb doesn't enter the picture when cloning over HTTP if I'm not mistaken.

Did we actually verify that this is caused by having too many heads?  Comment 4 presents that as a guess and comment 6 seems to assume that it is the case.  I _think_ that when cloning (without -r), hg doesn't need to enumerate the heads.  Also, I doubt that it can enumerate the try heads in 16-17 seconds, which is how long it takes for the clone to fail from here.

> * Releng has the doc on 'what to do if we fully reset try' at
> https://wiki.mozilla.org/ReleaseEngineering/How_To/Reset_the_Try_Server [hat
> tip to aki for finding it faster than I could]
> 
> Thanks to aki, bkero, cshields, joduinn, philor for the help.
> 
> My personal recomendation is WONTFIX given the "why would you want to clone
> from try like this, nevermind over http, anyway!" but I'm open to learning a
> legit reason for your needs.

Well, you have my reason.  I guess I won't object very strongly to WONTFIXing this, and just end up pushing the same thing multiple times instead, which is crappy towards infra load, but whatever.

Comment 8

[Kendall Libby [:fubar] (he/him)]

try was reset recently (~2 weeks ago?) and cloning via http is working again:

$ time hg clone http://hg.mozilla.org/try
destination directory: try
requesting all changes
adding changesets
adding manifests
adding file changes
added 142250 changesets with 761774 changes to 130482 files (+1404 heads)
updating to branch default
76934 files updated, 0 files merged, 0 files removed, 0 files unresolved

real	12m38.561s
user	6m25.543s
sys	0m50.587s


also, http is handled by hgweb.

Comment 9

[Shyam Mani [:fox2mike]]

Ehsan, seems like this works now?

Comment 10

[(no longer active)]

I don't know, I haven't tried.

Comment 11

[Shyam Mani [:fox2mike]]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> I don't know, I haven't tried.

Could you please try and let us know? :) (that was the hidden message in the previous needinfo :p)

Comment 12

[(no longer active)]

(In reply to comment #11)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> > I don't know, I haven't tried.
> 
> Could you please try and let us know? :) (that was the hidden message in the
> previous needinfo :p)

Well comment 4 suggests that this was due to the number of heads on try, which is at least plausible.  Comment 6 suggests that this bug is not even worth fixing (and I'm not sure why the bug was left open since then.)  And now try has been reset, so cloning try now gives you a completely different repo than cloning try before would, so I don't see what we're investigating here.  Seems like the most that can be done now is to verify that I can clone this new repo, but doing that won't buy us anything...  Or am I missing something?

Comment 13

[Shyam Mani [:fox2mike]]

Comment #6 wasn't by a member of my team. I was going through the list of open bugs and since you'd said you had issues and yes, since try was reset..we've been able to clone it fine.

Looks like you don't care about this anymore :)

Comment 14

[(no longer active)]

Adjusting the resolution to correctly match what happened here.  ;-)