1296091 - fx-team missing a push (876756efc390)

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/#/jobs?repo=fx-team is missing http://hg.mozilla.org/integration/fx-team/rev/876756efc390

Closed all tree except Try until we know what's affected.

Comment 1

[Ed Morley [:emorley]]

Raced with my IRC message :-)

22:12 <emorley> the pushlog is missing that push (https://hg.mozilla.org/integration/fx-team/pushlog) -> this is an hg.mozilla.org problem not Treeherder
22:13 <emorley> could you file a bug in Developer Services :: hg.mozilla.org and update the tree closure message (just so people don't think it's treeherder)

Comment 2

[Ed Morley [:emorley]]

Note that the push exists on the HTML pushlog (https://hg.mozilla.org/integration/fx-team/pushloghtml), just not the json one.

Comment 3

[Gregory Szorc [:gps]]

The push exists in the JSON.

This is likely a timing issue. From https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/notifications.html:

> Pulse messages are written after the push has been completely replicated
> to all active HTTPS replication mirrors.

> Unlike polling the pushlog, reacting to data advertised by Pulse events
> is not susceptible to race conditions when one mirror may have replicated
> data before another.

Comment 4

[Ed Morley [:emorley]]

Please don't close bugs about tree closures if the trees are still not open-able.
The bug might need to morph again, but it still can't be closed until the trees reopen.

I'd gotten confused and thought the json pushlog was available from https://hg.mozilla.org/integration/fx-team/pushlog (and that the browser was just displaying the feed due to the different Accept header).

The push is definitely missing from here fwiw:
https://hg.mozilla.org/integration/fx-team/pushlog

I'll look into this more.

Comment 5

[Gregory Szorc [:gps]]

The servers are each advertising the pushlog through the JSON endpoint:

$ ansible -i ansible/hosts hgweb-prod -a 'curl -s http://localhost/integration/fx-team/json-pushes?changeset=876756efc390&version=2'
hgweb12.dmz.scl3.mozilla.com | success | rc=0 >>
{
"lastpushid": 20332,
"pushes": {"20332": {"changesets": ["876756efc390115f1fde3b23e8369edd95eda403"], "date": 1471465400, "user": "[e-mail]"}}
}

hgweb13.dmz.scl3.mozilla.com | success | rc=0 >>
{
"lastpushid": 20332,
"pushes": {"20332": {"changesets": ["876756efc390115f1fde3b23e8369edd95eda403"], "date": 1471465400, "user": "[e-mail]"}}
}

hgweb14.dmz.scl3.mozilla.com | success | rc=0 >>
{
"lastpushid": 20332,
"pushes": {"20332": {"changesets": ["876756efc390115f1fde3b23e8369edd95eda403"], "date": 1471465400, "user": "[e-mail]"}}
}

hgweb11.dmz.scl3.mozilla.com | success | rc=0 >>
{
"lastpushid": 20332,
"pushes": {"20332": {"changesets": ["876756efc390115f1fde3b23e8369edd95eda403"], "date": 1471465400, "user": "[e-mail]"}}
}

If you see something different, there's likely a HTTP cache interfering. We shouldn't have caching enabled on the server-side of hg.mozilla.org.

Comment 6

[Ed Morley [:emorley]]

Ok, we'll look more on the Treeherder side.

Cameron, I don't suppose you could take a look, since I'm heading to bed soon? There are no errors in New Relic, no errors or useful info in the logs on th-prod-etl[12]. I'm guessing either:
(a) the push was semi-ingested but stuck as hidden like happened that previous time
(b) it hasn't been ingested at all, but for some reason memcached was updated with last push ID (so a memcached clear would help, since it would fall back to ingestion the last 10 pushes for that repo)

Sebastian, this seems localised to fx-team, so please can you can reopen the other trees in the meantime?

Thanks :-)

Comment 7

[Cameron Dawson [:camd]]

Hmm, so the push DID make it into Treeherder, but was never updated correctly with its push_timestamp and author.  So it's out of order:
https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=876756efc390

I'm looking into what may have caused this...

Comment 8

[Cameron Dawson [:camd]]

So a query across all repos shows this happened once on fx-team, once on gaia, and 5 times on try since the beginning of june.  People may not have noticed it on try, since they get a direct link to their push, rather than looking at them sorted by push_timestamp.

P.S. - There was a slew of these last May due to a Treeherder bug that was fixed:
https://github.com/mozilla/treeherder/commit/3642e03307428ef92ce665d267cbc144fdb61d09

I'm investigating how this could have happened now.

Comment 9

[Gregory Szorc [:gps]]

If the underlying cause is a race condition in hg.mozilla.org, the 5 occurrences on Try wouldn't surprise me: the replication/consistency window of that repo generally isn't as small as other repos.

FWIW, I recommend consumers wanting to know when pushes happen should be looking at pulse. That is faster and lighter weight than polling and doesn't suffer from data consistency issues on hg.mozilla.org.

Comment 10

[Cameron Dawson [:camd]]

GPS - Yeah, I think this is a race condition. Treeherder.allizom.org shows the push as it should be.  FWIW: we should be moving treeherder to using your hg pulse exchange by end of quarter.

We DO explicitly attempt to handle this type of race condition, though.  But apparently not well enough.

Comment 11

[Cameron Dawson [:camd]]

I have fixed the bad occurrance in the DB for that push in fx-team.  I think you can reopen the fx-team tree now.  I will continue to investigate the underlying issue, but this has been extremely rare thus far.

Comment 12

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I've reopened fx-team.