624289 - No 1.9.1 nightlies on Jan 9th; No 1.9.2 nightlies on 8th or 9th

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

It's probably related to this error on the pm01:scheduler_master, which first showed up here
Exception in /builds/buildbot/scheduler_master/twistd.log.1:
2011-01-08 03:33:12-0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/tools/python/lib/python2.6/threading.py", line 504, in __bootstrap
            self.__bootstrap_inner()
          File "/tools/python/lib/python2.6/threading.py", line 532, in __bootstrap_inner
            self.run()
          File "/tools/python/lib/python2.6/threading.py", line 484, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/python/threadpool.py", line 207, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/enterprise/adbapi.py", line 429, in _runInteraction
            result = interaction(trans, *args, **kw)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/schedulers/timed.py", line 249, in _check_timer
            self._maybe_start_build(t)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/schedulers/timed.py", line 277, in _maybe_start_build
            self.start_HEAD_build(t)
          File "/tools/buildbotcustom/buildbotcustom/scheduler.py", line 69, in start_HEAD_build
            ss = self.ssFunc(self, t)
          File "/tools/buildbotcustom/buildbotcustom/misc_scheduler.py", line 277, in ssFunc
            rev = lastChangeset(db, t, branch)
          File "/tools/buildbotcustom/buildbotcustom/misc_scheduler.py", line 138, in lastChangeset
            for c in changeEventGeneratorInTransaction(db, t, branches=[branch]):
          File "/tools/buildbotcustom/buildbotcustom/misc_scheduler.py", line 132, in changeEventGeneratorInTransaction
            for (changeid,) in rows:
        exceptions.TypeError: 'long' object is not iterable

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Dustin says this is probably fallout from bug 620115.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: buildbotcustom-r1.patch

Same patch as was posted on bug 620115, but posted in the right place.

I've been unable to stage this; see bug 620115 comment 25.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

This ran acceptibly locally:

2011-01-10 11:00:01-0600 [-] lastGoodRev: took 0.00 seconds to run; returned None
2011-01-10 11:00:01-0600 [-] lastChangeset returned None
2011-01-10 11:00:01-0600 [-] lastNightlyRevision was None

Note that this doesn't prove much, except that there are no glaring errors, because with pysqlite execute returns a cursor, which can be iterated -- so the "for (changeid,) in rows" works for sqlite.

I see the same thing in staging, although I suspect that's using sqlite too:

2011-01-10 09:20:03-0800 [-] lastGoodRev: took 0.00 seconds to run; returned None
2011-01-10 09:20:03-0800 [-] lastChangeset returned None
2011-01-10 09:20:03-0800 [-] lastNightlyRevision was None

do we have a staging MySQL configuration I can test against?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, scheduler_master uses the same configs as sm01 but runs in a different directory.

2011-01-10 10:35:04-0800 [-] lastGoodRev: took 0.04 seconds to run; returned None
2011-01-10 10:35:04-0800 [-] lastChangeset returned 7f2b60765d01cf840e63e09157106be263e4f3a2
2011-01-10 10:35:26-0800 [-] lastNightlyRevision was 7f2b60765d01cf840e63e09157106be263e4f3a2
2011-01-10 10:35:26-0800 [-] mozilla-central nightly: Creating buildset with sourcestamp ['7f2b60765d01cf840e63e09157106be263e4f3a2', "in 'mozilla-central'"]
2011-01-10 10:35:26-0800 [-] mozilla-central nightly: propfunc returned {'buildid': '20110110103526'}
2011-01-10 10:35:26-0800 [-] mozilla-central nightly: propfunc returned {'builduid': '7f1af5eaedcb4c2db0cf7c5991af1f44'}

so this seems to work.

Comment 5

[Ben Turner (not reading bugmail, use the needinfo flag!)]

I didn't get a trunk nightly today, on OS X 64-bit. Related? New bug?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

bent: related AFAIK.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 502484 [details] [diff] [review]
buildbotcustom-r1.patch

5a9a01ed84c0

Landed on /tools/buildbotcustom/buildbotcustom on scheduler-master, but the master is not reconfigured yet.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

It's on default, not production-0.8 -- I'll wait for the reconfig to do that merge.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

This failed in run_clobber:

............F...
======================================================================
FAIL: testSlaveNoPeriodicClobberOther (__main__.TestClobber)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_clobberer.py", line 285, in testSlaveNoPeriodicClobberOther
    self.assert_('mybuilder:More than' not in data, data)
AssertionError: Checking clobber URL: http://preproduction-master.build.mozilla.org/~cltbld/index.php?master=master01&slave=slave01&builddir=mybuilder&branch=branch1&buildername=My+Builder
Server gave us {}
mybuilder:Our last clobber date:  2011-01-10 10:42:11
mybuilder:Server clobber date:    None
mybuilder:More than 3600.0 seconds have passed since our last clobber
mybuilder:Clobbering...
Skipping last-clobber


----------------------------------------------------------------------
Ran 16 tests in 16.966s

I don't know anything about clobberer, so any help debugging this would be great.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Apparently that test is flaky, so I'm ignoring the failure:
13:57 < lsblakk> sometimes it's just an url hang, not that your code broke it

Comment 12

[Mike Taylor [:bear]]

dustin - did this land last night?

if so can you update this and also update the maintenance page - thanks

Comment 13

[Chris B.]

I currently see 1/10 and 1/11 builds for all operating systems, except for Windows 64-Bit, which is always a couple days behind anyway.

The 1/10 builds are labeled as 4.0b9pre.
The 1/11 builds are now labeled as 4.0b10pre.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Added to Maintenance.

The scheduler_master was restarted ~20:55 PST yesterday (2011-01-10).

Comment 15

[Mike Taylor [:bear]]

(In reply to comment #13)
> I currently see 1/10 and 1/11 builds for all operating systems, except for
> Windows 64-Bit, which is always a couple days behind anyway.
> 
> The 1/10 builds are labeled as 4.0b9pre.
> The 1/11 builds are now labeled as 4.0b10pre.

the fix for the nightly issue was applied just before nightlies would trigger
for the 11th.

4.0b9 was tagged the evening of the 10th so that's why your seeing the change
for nightlies on the 11th