1076826 - Use version 2 of the json-pushes API and '&startID=N' rather than '&fromchange=SHA' for polling the pushlog

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

According to jeads, using &fromchange=REV with json-pushes requires an additional join (compared to &startID=N), which is presumably what is contributing to the higher rates of timeout accessing json-pushes than with TBPL.

We should either:

a) fix json-pushes to not time out (jeads mentioned wanting to find out if it was due to deadlocks etc) / not need the additional join

b) switch back to using startID=N (at least short term), since this is what TBPL has been using successfully for years, and works "just fine". Note this will mean that we'll need to go back to having a manual cache clear when repos are reset, until the pushlog bug 1065771 is fixed.

Comment 1

[Ed Morley [:emorley]]

https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/pushlog-legacy/pushlog-feed.py#l102

   102             elif self.querystart == QueryType.CHANGESET:
   103                 where.append("id > (select c.pushid from changesets c where c.node = :start_node)")
   104                 params['start_node'] = hex(self.repo.lookup(self.querystart_value))
   105             elif self.querystart == QueryType.PUSHID:
   106                 where.append("id > :start_id")
   107                 params['start_id'] = self.querystart_value

Comment 2

[Ed Morley [:emorley]]

Pushlog schema:
https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hghooks/mozhghooks/pushlog.py#l40

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Ed Morley [:edmorley] from comment #0)
> According to jeads, using &fromchange=REV with json-pushes requires an
> additional join (compared to &startID=N), which is presumably what is
> contributing to the higher rates of timeout accessing json-pushes than with
> TBPL.
> 
> We should either:
> 
> a) fix json-pushes to not time out (jeads mentioned wanting to find out if
> it was due to deadlocks etc) / not need the additional join

Hal, how can we get access to the weblogs for hgweb?

Comment 4

[Hal Wine [:hwine] use NI!]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> Hal, how can we get access to the weblogs for hgweb?

For now, ask for them in #vcs, and specify (roughly) how much data you want (# weeks) We'll pull them off the 10 hosts and make them available.

Comment 5

[Ed Morley [:emorley]]

Re:
19:32 <jgriffin> Can someone help retrieve weblogs for hgweb queries in the format http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5d6ec4dddf14 ?
19:32 <jgriffin> We’re seeing lots of timeouts for those queries in Treeherder and are trying to determine the cause
19:33 <•fubar> jgriffin: how far back?
19:36 <jgriffin> fubar: not far, just a few days worth would be fine
19:36 <•fubar> jgriffin: np. m-c specifically, or any tree?
19:37 <jgriffin> fubar: any/all trees
19:37 <jgriffin> the problem we’re seeing doesn’t seem to be tree-specific
20:10 <•fubar> jgriffin: hg-pushloghtml.txt in your homedir on people
20:10 <jgriffin> fubar: tyvm!

The pushloghtml is not what TBPL/treeherder use, that's json-pushes.

The requests of interest are:

http://hg.mozilla.org/*/json-pushes?full=1&fromchange=*

Comment 6

[Ed Morley [:emorley]]

When we do time out requesting json-pushes, how quickly do we retry? Or more, by how long does it delay ingestion? (Just trying to set priority for this bug)

Comment 7

[Mauro Doglio [:mdoglio]]

A single timeout in the connection to json-pushes will delay the ingestion by about a minute. But that's probably not very relevant; if that timeout is due to the json-pushes service being under heavy load, we will likely see a series of timeouts, not just one. That could result in several minutes of ingestion delay, it's very hard to tell precisely.

Comment 8

[Mauro Doglio [:mdoglio]]

I forgot to mention, the pushlog ingestion is per repo. That means a series of timeouts in one repo will not affect the other ones.

Comment 9

[Ed Morley [:emorley]]

Thank you - if we retry after 1 minute, I think this is a P2 given it will require digging more into the logs and the lower hanging (and higher priority) P1s that are already filed.

Comment 10

[Ed Morley [:emorley]]

I've broken the stats gathering out to bug 1079788, since this bug is about mitigation steps we can take on treeherder's side (whereas I imagine we'll need pushlog changes too).

Comment 11

[Ed Morley [:emorley]]

Adjusting summary, since bug 1104374 will need us to use startID anyway.

Comment 12

[Gregory Szorc [:gps]]

Bug 1065771 is now in production. URLs like https://hg.mozilla.org/mozilla-central/json-pushes?version=2 now work. Adding the |version=2| querystring param will change the output format of the JSON slightly. A new top-level "lastpushid" property is now present. You can use that to track push offsets and to detect if/when the pushlog is reset (the value would decrease if the database is reset).

Comment 13

[Ed Morley [:emorley]]

eg:
http://hg.mozilla.org/integration/fx-team/json-pushes?startdate=2015-07-05&enddate=2015-07-06&version=2
{
 "lastpushid": 14479, 
 "pushes": {}
}

Comment 14

[Ed Morley [:emorley]]

(ignore the startdate params; it's 'lastpushid' that is the relevant part there)

Comment 15

[Cameron Dawson [:camd]]

Note from GPS:

Polling pushlog every 10-15 seconds should be fine. TaskCluster does something in that range. Although one thing they do that I would appreciate you don't do is queue and fire 20+ HTTP requests nearly simultaneously. That can overload the servers. Staggering requests is the way to go.

Comment 16

[Cameron Dawson [:camd]]

Attachment: PR

Comment 17

[Cameron Dawson [:camd]]

Hi Ed-- I know you already looked this over.  But I make a couple changes (including removing the change to the poll interval, as you suggested).  So please have a look at your convenience.

Thanks!  :)

Comment 18

[Ed Morley [:emorley]]

Comment on attachment 8651205 [details] [review]
PR

Thank you for doing this :-)

Comment 19

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/899ccf48b48696549cf5bdc68508d6a97c821917
Bug 1076826 - Use version 2 of the json-pushes API