518829 - pushloghtml is unreliable

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Serge Gautherie (:sgautherie)]

I have noticed that pushlog can:

* Take a moment to acknowledge new pushes.
While I felt like it always was immediate "before". (Though maybe I just didn't notice.)
(Noticed this for some time now.)

* Lastest pushes can be displayed then forgotten then redisplayed again...
This is even more painful on http://tests.themasta.com/tinderboxpushlog/
where builds move from one push to another :-(
(Noticed this recently only.)

Comment 1

[Reed Loden [:reed]]

I've also seen this a lot lately when I've been landing patches on multiple Hg repos. It's very frustrating, and it greatly confuses tbpl.

Comment 2

[Serge Gautherie (:sgautherie)]

Pike and ted told me to check:
http://dm-hg01.mozilla.org/mozilla-central/pushloghtml?startdate=1+day+ago&enddate=now
and
http://dm-hg02.mozilla.org/mozilla-central/pushloghtml?startdate=1+day+ago&enddate=now
which are behind
http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=1+day+ago&enddate=now

Results of reloading both pages:
Both dm-hg0? "randomly" include/miss (my) lastest checkin.

Seems like the issue is at a "lower" level.
(Some stuck http/database connection/cache?)

Comment 3

[Serge Gautherie (:sgautherie)]

Fwiw, comm-central push log has this bug too.

Comment 4

[Aravind Gottipati [:aravind]]

I am unable to reproduce this issue.  Can I get a reproducible test case?  I have an idea on what could be going on, but I'd like to have a reproducible test case that I can use.

Comment 5

[Serge Gautherie (:sgautherie)]

Right now, I can reproduce with
http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=1+day+ago&enddate=now
though not with
http://hg.mozilla.org/mozilla-central/pushloghtml

Comment 6

[Aravind Gottipati [:aravind]]

Found that one of the backend webheads had the wrong timestamp on it.  Enabled time-sync options in vmware, so it should be good now.

Comment 7

[Serge Gautherie (:sgautherie)]

Can't reproduce anymore atm, let's V.Fixed it.