813839 - hg pushlog date range query much slower on mozilla-beta than on mozilla-central

Status: RESOLVED INVALID

(Developer Services :: General, task)

Description

[mcdavis941 (sporadically reading bugmail)]

Running a pushlog date range query (spanning a month) through the web interface is taking about 15 to 20 minutes against mozilla-beta, while the same query against mozilla-central finishes in under a minute.  It would be useful if that could be more responsive.  

(I started out this morning trying to get m-b's entire pushlog from June through now, to save out locally for future reference, and didn't manage to get all of it until the end of the day.  Breaking it down into queries spanning a month at a time worked in the end, although I had to retry some of those several times before they completed.)

For example, this query:

http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?startdate=2012-10-01+00%3A00%3A00&enddate=2012-11-01+00%3A00%3A00

took about 20 minutes to finish, after three retries.

Comment 1

[mcdavis941 (sporadically reading bugmail)]

On IRC #build, nthomas added:

nthomas> probably from the merges dropping a lot of changes in a single push
nthomas> m-c is different because it has a steady stream of smaller merges
nthomas> a day or so's work vs 6 weeks

Comment 2

[Shyam Mani [:fox2mike]]

Ted's our pushlog man :) CC'ing.

Comment 3

[(not currently active) Ted Mielczarek]

nthomas' answer in comment 1 is almost assuredly the cause. The mozilla-{inbound,central} pushlog has each push containing much fewer changesets. The merge pushes from m-c to m-a and then on to m-b have a huge amount of changesets in each push.

The slowness here seems to be all in accessing the changeset data from the repo itself. If you use the json-pushes query below (I just did s/pushloghtml/json-pushes/ in your query from comment 0):
http://hg.mozilla.org/releases/mozilla-beta/json-pushes?startdate=2012-10-01+00%3A00%3A00&enddate=2012-11-01+00%3A00%3A00

You can also see the merge push in there (push ID "1490"), which is almost certainly what makes this slow. Adding &full=1 to the query makes it load changeset data from Mercurial, which makes it slower:
http://hg.mozilla.org/releases/mozilla-beta/json-pushes?startdate=2012-10-01+00%3A00%3A00&enddate=2012-11-01+00%3A00%3A00&full=1

Comment 4

[Shyam Mani [:fox2mike]]

So this is basically expected behavior?

Comment 5

[(not currently active) Ted Mielczarek]

Yeah, I'm basically going to say "you're querying a lot of data". I don't have any way to optimize this since it's Hg that's slow here.