587559 - cache on hg.m.o not being cleared properly for tbpl urls

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

I've heard about or seen this several times over the last few days:

1, developer pushes to repository (mozilla-central, projects/birch for example)
2, builds start OK in RelEng-land and show up on tinderbox
3, builds don't appear in tinderbox pushlog because hg.m.o returns stale content for urls like
 http://hg.mozilla.org/projects/birch/pushloghtml?startdate=12+hours+ago&enddate=now
 http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=12+hours+ago&enddate=now
4, cron comes along at 3 & 33 past the hour and fixes up hg.m.o by dumping the cache
5, everyone gets on with life

IIRC there is a push hook which is supposed to be doing cache invalidating, please take a look if it's working OK.

Comment 1

[(no longer active)]

This causes serious problems for developers, especially when near code freeze.  Bumping the priority in hopes of a quicker fix.

Comment 2

[Phil Ringnalda (:philor)]

The quick fix, if aravind is still out, would be to just increase the frequency of the cronjob - a five minute wait isn't going to even really be noticeable with all the other delays built into the tinderboxpushlog chain (and the way that nothing has really happened in the first five minutes unless a push really quickly fails to compile), and I'd hope hgweb could temporarily cope with having to generate a new page to stick in the cache every five minutes.

Comment 3

[Aravind Gottipati [:aravind]]

(In reply to comment #2)
> The quick fix, if aravind is still out, would be to just increase the frequency
> of the cronjob - a five minute wait isn't going to even really be noticeable

I am in, watching this bug, have some ideas.. but its still only 11:33 here, I should be able to patch this up and roll something out by the end of the day.

Comment 4

[Aravind Gottipati [:aravind]]

(In reply to comment #0)
> IIRC there is a push hook which is supposed to be doing cache invalidating,
> please take a look if it's working OK.

I think I have a fix in place.  We have had a purge script that runs after code is checked into a repo, that clears all urls associated with the repo.  We started noticing timing issues with these triggered purges and active requests (and requests for stuff immediately after the purge), within seconds.  So I had a delayed purge script that would clear these out a second time.

I fixed that script to now delay a minute.  I left the original 3,33 tbpl purge scripts intact.  If all that theory is correct, we should be rid of these "stuff not showing up in tbpl" problems.