603678 - Need more data for Input staging (releasing on Thursday)

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

aakashd: if you head over to http://input.stage.mozilla.com/en-US/search/?q=&product=mobile&version=&date_start=&date_end=
[10:27am] aakashd: and then look at http://input.mozilla.com/en-US/search/?q=&product=mobile&version=&date_start=&date_end=
[10:27am] aakashd: we're not grabbing that part of the dump from what i can tell
[10:28am] aakashd: the same guys for - http://input.mozilla.com/en-US/themes/ and http://input.stage.mozilla.com/en-US/themes/ where "vista" themes are not shown on staging, but are there for production
[10:29am] justdave: and that's all stuff that's in the database?
[10:30am] justdave: is that stuff that's been added since the dump I did manually?
[10:30am] aakashd: yeah

Comment 1

[Aakash Desai [:aakashd]]

This is blocking testing and we have a release tomorrow. Uping.

Comment 2

[Dave Miller [:justdave]]

The cron job that was pushing this to the staging database server had a typo in the hostname, so it was just bailing saying it couldn't find it.  Fixed the typo and ran it manually, the database should be updated to this morning's backup now.

Comment 3

[Dave Miller [:justdave]]

meant to resolve that while I was at it. :)

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

http://input.stage.mozilla.com/en-US/themes/ is still empty.

Comment 5

[Dave Miller [:justdave]]

The database import is working correctly now, must be something in the app.  Sphinx index needs to be updated perhaps?  Is this something I need to add to the cron job to have it kick it after the import is done? (and if so, how do I do that?)

Comment 6

[Dave Miller [:justdave]]

ok, turned out to be the "cron cluster" cron job.  That job needs to be run after a database import.  It was currently set to run once per day at 5am.  I changed it to run once per hour, timed to hit within a few minutes after a database restore would have run, if one was picked up that hour.

The cron job pushing the backups from dracula to the database server runs at :25 every hour, and generally takes about a minute for the rsync.

The cron job on the database server which checks to see if a new file has arrived runs at :30 every hour, and usually takes 3 to 4 minutes to do the DB restore from it.

The cron job on the staging app server which runs the "cron cluster" job now runs at :35 every hour.  If we have issues and end up with stuff not working for an hour after the restore, I can probably move that to :40 or something, but for now it seems to be able to restore that quickly.

Comment 7

[Dave Miller [:justdave]]

And just to complete the story...  backups run at 7am every day.  They take a variable amount of time to complete, because there's 9 database servers being backed up, and they're done in sequence.  The backups are pushed from the backup server to dracula as they complete. By checking once an hour to see if there's a new backup available, we get the DB restored on stage as soon as it's available, no matter when that might be, even if the backups are running late on that day.  With the exception of the "cron cluster" job, the other two are no-ops if there's nothing new there to process.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks, Dave.  There's plenty of data now on staging -- both on mobile and desktop.

Verified FIXED.