693121 - Please update TBPL production to 92e6636245e0

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Phil Ringnalda (:philor)]

I was hoping to make it more than a week before we needed an update, but some weeks don't cooperate: please update tbpl prod to http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/72b0edbeb2a6. Thanks!

Comment 1

[Arpad Borsos [:Swatinem]]

On this update, import-buildbot-data.py -d5 -f should be called, to import the last 5 days again, overwriting the data. So we update the runs that have a got_revision different from revision, or that had no revision at all.

Comment 2

[Phil Ringnalda (:philor)]

I'd probably seem less fickle if I just filed as "please update tbpl prod to a cset to be named at the time you ping me on irc to say you're about to start." :)

http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/92e6636245e0, please and thanks.

Comment 3

[Phil Ringnalda (:philor)]

Not sure whether some part of the code is broken (though it's running fine on http://tbpl.swatinem.de/), or the reimport is broken, or we've just never done it live while lots of people are accessing tbpl, but we're pretty thoroughly broken, and I'm closing the trees.

Some of the tbpl calls that fetch run data from the db are giving 500 errors or just timing out, like https://tbpl.mozilla.org/php/getRevisionBuilds.php?branch=mozilla-central&rev=f7bf7ac18a79&_=1318275863065, while others pretend to succeed, but are returning data that's way out of date.

Comment 4

[Jason Thomas [:jason]]

Update information: 

[[e-mail]1 tbpl.mozilla.org]# ./update 92e6636245e0
+ date
Mon Oct 10 11:12:58 PDT 2011
+ echo -e 'Updating code...'
Updating code...
+ cd /data/genericrhel6/src/tbpl.mozilla.org/tbpl
+ hg pull
pulling from http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog
searching for changes
no changes found
+ hg update 92e6636245e0
merging dataimport/import-buildbot-data.py
6 files updated, 1 files merged, 0 files removed, 0 files unresolved
+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ cd /data/genericrhel6/src/tbpl.mozilla.org/tbpl
+ hg tip
changeset:   712:92e6636245e0
tag:         tip
user:        Arpad Borsos <[e-mail]>
date:        Thu Oct 06 09:56:59 2011 +0200
summary:     Bug 685053 - Give individual Talos jobs their own unique names on tbpl; r=mstange, feedback+fiddling=philor

+ checkretval
+ retval=0
+ [[ 0 -gt 0 ]]
+ /data/genericrhel6/deploy tbpl.mozilla.org

Comment 5

[Jason Thomas [:jason]]

import-buildbot-data.py completed. This was causing slowness/unresponsiveness on tbpl.mozilla.org. We should try to do these updates at a time which will minimize impact to users.

Comment 6

[Arpad Borsos [:Swatinem]]

philor: is this fixed now for good?
The reimport is slow, unfortunately. Did that cause the 500 errors probably? Overloading the server?

Comment 7

[Phil Ringnalda (:philor)]

Yeah, it's both updated and happy now, but we're going to need to seriously think about whether we need a reimport, and for how many days we want to reimport, any time we change the import code from now on. Remains to be seen whether or not it'll go faster in the middle of the night when there aren't so many users and so many pushes and so many builds and so many changes, but we can't do another one like today without scheduling a downtime and closing the trees for it.

Comment 8

[Arpad Borsos [:Swatinem]]

Why is it so slow? And why are requests blocked by the import?
Is it possible that the usage of ONE SQL transaction for the whole import is the cause here?

Comment 9

[Armen [:armenzg]]

Perhaps add a mechanism to schedule it to update automatically on Sunday evening/night?