554841 - Buildbot history db not being updated

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Excerpt from production-master:~cltbld/update.log:

Starting run at Wed Mar 24 19:50:01 PDT 2010
./mozilla-central-macosx-opt-unittest-mochitests-2 2181 1/79 0.00% complete ETA in 0 seconds
...
./mozilla-1.9.2-linux-unittest-reftest 134 7/7 7.10% complete ETA in 3253 seconds

./mozilla-central-win32-nightly 398 1/6 7.11% complete ETA in 3253 seconds
Traceback (most recent call last):
  File "/tools/buildbotcustom/buildbotcustom/bin/update_from_files.py", line 300, in <module>
    updated = updateFromFiles(session, options.master, options.name, builders, last_time, options.times)
  File "/tools/buildbotcustom/buildbotcustom/bin/update_from_files.py", line 220, in updateFromFiles
    db_build = model.Build.fromBBBuild(session, build, builder_name, master.id)
  File "/tools/buildbotcustom/buildbotcustom/status/db/model.py", line 536, in fromBBBuild
    b.updateFromBBBuild(session, build)
  File "/tools/buildbotcustom/buildbotcustom/status/db/model.py", line 471, in updateFromBBBuild
    self.properties = Property.fromBBProperties(session, build.getProperties())
  File "/tools/buildbotcustom/buildbotcustom/status/db/model.py", line 104, in fromBBProperties
    if props[prop.name] == prop.value and props.getPropertySource(prop.name) == prop.source:
  File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/properties.py", line 42, in __getitem__
    rv = self.properties[name][0]
KeyError: u'completeSnippetFilename'

which is blocking a lot of builds being dumped into the db. We do seems to get some data, but the last compile job from pm01 was mozilla-1.9.2-linux64-xulrunner at 2010-03-24 17:32:33 PDT.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Backed off the master_cleanup.py job from -t4 to -t6 to give us a little time to deal with this without losing data.

Comment 2

[Chris Cooper [:coop] (he/him)]

catlee: any insight here? Is there one build missing a key and it's preventing all db updates from happening?

Comment 3

[Chris AtLee [:catlee]]

I'm not really sure.  If it was just one build, I would have expected it to have been purged a while ago, and then we'd get back up and running.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps not purged yet because I changed that to 6 days ? Also noticed that pm02 is getting backed up reprocessing the same builds each time it runs, about 20 mins to run and more than running at once. I think because the timestamp isn't updated when we error on the bogus jobs. Didn't change the the purge time there.

Last time we hit this there was something bogus in the db ?

Comment 5

[Chris AtLee [:catlee]]

Attachment: Check property name before accessing it

So the problem stems from the fact that mysql does case insensitive matching on in the query, so when we search for 'completeSnippetFilename', we get back a property from the db called 'completesnippetFilename', which doesn't exist in the build.

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 435097 [details] [diff] [review]
Check property name before accessing it

changeset:   671:4acda6a63f0a